hist3368-week8-working-with-tabular-data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "mUtTNBx4tOHM"
   },
   "source": [
    "# Hist 3368: Working With Tabular Data in Pandas\n",
    "\n",
    "***by Jo Guldi***\n",
    "\n",
    "*Please note: This notebook requires at least **30GB** of memory. You may have to start a new HPC session.*\n",
    "\n",
    "Until now, in this class we have worked with lists of words. We have cleaned them and counted and compared them.\n",
    "\n",
    "For the rest of the class, we will be working with data in tables. Tables allow us to keep track of the date when each word is from. If we have time data, we can compare wordcounts over time, compare wordcounts for different speakers, and so on.\n",
    "\n",
    "We will need a few special commands to navigate tabular data.\n",
    "\n",
    "In this notebook, we will learn to navigate tables:\n",
    "\n",
    "   * how to call a column\n",
    "   * how to move through a column, row by row, using a for loop\n",
    "   * how to subset or 'filter' data by a column, for example, finding all the speeches of one speaker:\n",
    "       * how to filter using square brackets -- **[ ]** \n",
    "       * the use of the operators **.isin()**, **==**, and **!=**.\n",
    "   * how to find the largest counts in a dataset using **.nlargest()**\n",
    "\n",
    "We will clean tabular data, with strategies we've seen before:\n",
    "   * stripping punctuation\n",
    "   * stopwording\n",
    "   * lemmatizing\n",
    "   * splitting into words (i.e. tokenization)\n",
    "\n",
    "We will also learn some basics of counting with tables:\n",
    "\n",
    "   * how to count the words in a subset of data.\n",
    "\n",
    "#### Learning Research Strategies\n",
    "\n",
    "We will practice navigating around the tabular data for Congress, asking the kind of questions a researcher might want to know, such as:\n",
    "\n",
    "   * given a set of years, who were the top speakers in Congress?\n",
    "   * given a speaker, what was his or her longest speech?\n",
    "   * given a certain set of words, who were the speakers who used those words the most?\n",
    "   \n",
    "The research questions profiled here are fairly simple, but if combined with strategies such as a *controlled vocabulary* they can result in a good deal of important information about which speakers were engaged with a particular topic -- for instance, the environment, crime, or women's health.  \n",
    "\n",
    "These research strategies can also help the researcher to navigate to the longest speeches where a speaker invokes those topics, or the speeches where the speaker invokes the highest number of words related to a particular topic.  Those research strategies should form the basis for guided reading.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/group/history/hist_3368-jguldi\n"
     ]
    }
   ],
   "source": [
    "cd /scratch/group/history/hist_3368-jguldi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This might take a minute. Loading takes time -- please be patient.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "congress = pd.read_csv(\"congress1967-2010.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Those who do not enjoy the privilege of the fl...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>16</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. President. on the basis of an agreement re...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>Mr. MANSFIELD</td>\n",
       "      <td>35</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>The Members of the Senate have heard the remar...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>40</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>The Chair lays before the Senate the following...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>151</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>Secretary of State.</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>Mrs. AGNES BAGGETT</td>\n",
       "      <td>3</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  Unnamed: 0.1  \\\n",
       "0           0             0   \n",
       "1           1             1   \n",
       "2           2             2   \n",
       "3           3             3   \n",
       "4           4             4   \n",
       "\n",
       "                                              speech        date  \\\n",
       "0  Those who do not enjoy the privilege of the fl...  1967-01-10   \n",
       "1  Mr. President. on the basis of an agreement re...  1967-01-10   \n",
       "2  The Members of the Senate have heard the remar...  1967-01-10   \n",
       "3  The Chair lays before the Senate the following...  1967-01-10   \n",
       "4                                Secretary of State.  1967-01-10   \n",
       "\n",
       "              speaker  word_count  year  month  month_year  \n",
       "0  The VICE PRESIDENT          16  1967      1  1967-01-01  \n",
       "1       Mr. MANSFIELD          35  1967      1  1967-01-01  \n",
       "2  The VICE PRESIDENT          40  1967      1  1967-01-01  \n",
       "3  The VICE PRESIDENT         151  1967      1  1967-01-01  \n",
       "4  Mrs. AGNES BAGGETT           3  1967      1  1967-01-01  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data you are looking at is 'tabular' -- meaning that it's in a table.  \n",
    "\n",
    "The format used by the pandas software package, which is running our table, is called a \"dataframe.\"  A dataframe is a mtwo-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns).  \"Heterogenous\" means that the dataframe can have some columns that hold strings, and other columns that hold numbers or dates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have met pandas data with an index before when we met the pandas Series.  A Series is a one-dimensional labeled array -- meaning that it only had one column, not many.  However, everything that we learned about navigating indices wlil apply to dataframes too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.index[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.index[1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can call the pandas data with the **.loc** function.  The formula for calling data is :\n",
    "\n",
    "    dataFrame.loc[<ROWS RANGE> , <COLUMNS RANGE>] -- for calling rows or columns by name\n",
    "    dataFrame.iloc[<ROWS RANGE> , <COLUMNS RANGE>] -- for calling rows or columns by number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are rows #1005-1008:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>1005</td>\n",
       "      <td>1005</td>\n",
       "      <td>Mr. President. for many years I have advocated...</td>\n",
       "      <td>1967-01-11</td>\n",
       "      <td>Mr. WILLIAMS of Delaware</td>\n",
       "      <td>184</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>1006</td>\n",
       "      <td>1006</td>\n",
       "      <td>I am delighted to have the Senator from Delawa...</td>\n",
       "      <td>1967-01-11</td>\n",
       "      <td>Mr. DIRKSEN</td>\n",
       "      <td>27</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1007</th>\n",
       "      <td>1007</td>\n",
       "      <td>1007</td>\n",
       "      <td>Mr. President. I submit a resolution to amend ...</td>\n",
       "      <td>1967-01-11</td>\n",
       "      <td>Mr. CANNON</td>\n",
       "      <td>449</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0  Unnamed: 0.1  \\\n",
       "1005        1005          1005   \n",
       "1006        1006          1006   \n",
       "1007        1007          1007   \n",
       "\n",
       "                                                 speech        date  \\\n",
       "1005  Mr. President. for many years I have advocated...  1967-01-11   \n",
       "1006  I am delighted to have the Senator from Delawa...  1967-01-11   \n",
       "1007  Mr. President. I submit a resolution to amend ...  1967-01-11   \n",
       "\n",
       "                       speaker  word_count  year  month  month_year  \n",
       "1005  Mr. WILLIAMS of Delaware         184  1967      1  1967-01-01  \n",
       "1006               Mr. DIRKSEN          27  1967      1  1967-01-01  \n",
       "1007                Mr. CANNON         449  1967      1  1967-01-01  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.iloc[1005:1008, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the speaker column. Notice the use of ':' for 'everything':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                  The VICE PRESIDENT\n",
       "1                       Mr. MANSFIELD\n",
       "2                  The VICE PRESIDENT\n",
       "3                  The VICE PRESIDENT\n",
       "4                  Mrs. AGNES BAGGETT\n",
       "                      ...            \n",
       "5992063                   Ms. GRANGER\n",
       "5992064    Ms. KILPATRICK of Michigan\n",
       "5992065                    Mr. HELLER\n",
       "5992066                   Mr. PAULSEN\n",
       "5992067          Mr. HALL of New York\n",
       "Name: speaker, Length: 5992068, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.loc[:, 'speaker']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also call columns by name using just square brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                  The VICE PRESIDENT\n",
       "1                       Mr. MANSFIELD\n",
       "2                  The VICE PRESIDENT\n",
       "3                  The VICE PRESIDENT\n",
       "4                  Mrs. AGNES BAGGETT\n",
       "                      ...            \n",
       "5992063                   Ms. GRANGER\n",
       "5992064    Ms. KILPATRICK of Michigan\n",
       "5992065                    Mr. HELLER\n",
       "5992066                   Mr. PAULSEN\n",
       "5992067          Mr. HALL of New York\n",
       "Name: speaker, Length: 5992068, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that I can also call the column with double brackets.\n",
    "\n",
    "  * The difference between the two methods of calling the column is that above, single brackets call the column as a pandas Series.  \n",
    "  * Double brackets call the column as a pandas dataframe -- such that the column is labeled with its name.  \n",
    "     * The chief difference between a dataframe and a Series is that with a dataframe you can add extra columns later if you want to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mr. MANSFIELD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Mrs. AGNES BAGGETT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5992063</th>\n",
       "      <td>Ms. GRANGER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5992064</th>\n",
       "      <td>Ms. KILPATRICK of Michigan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5992065</th>\n",
       "      <td>Mr. HELLER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5992066</th>\n",
       "      <td>Mr. PAULSEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5992067</th>\n",
       "      <td>Mr. HALL of New York</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5992068 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            speaker\n",
       "0                The VICE PRESIDENT\n",
       "1                     Mr. MANSFIELD\n",
       "2                The VICE PRESIDENT\n",
       "3                The VICE PRESIDENT\n",
       "4                Mrs. AGNES BAGGETT\n",
       "...                             ...\n",
       "5992063                 Ms. GRANGER\n",
       "5992064  Ms. KILPATRICK of Michigan\n",
       "5992065                  Mr. HELLER\n",
       "5992066                 Mr. PAULSEN\n",
       "5992067        Mr. HALL of New York\n",
       "\n",
       "[5992068 rows x 1 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress[['speaker']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also see how many rows there are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5992068"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can call data from the datatable by row, column name, and by row number.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is just the speaker and speech for row 3234:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "speaker                                            Mr. TOWER\n",
       "speech     Mr. President. on June 17. a starting gun will...\n",
       "Name: 3234, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.loc[:, ['speaker', 'speech']].iloc[3234, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is just the speech:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "speech    Mr. President. on June 17. a starting gun will...\n",
       "Name: 3234, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myspeech = congress.loc[:, ['speech']].iloc[3234, :]\n",
    "myspeech"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use some familiar tools to print out the whole speech or any portion thereof:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mr. President. on June 17. a starting gun will sound in San Marcos. Tex.. and the worlds toughest river race will be underway. The race is the Texas water safari. marking its fifth year in 1967 with a 538mile race from San Marcos. by way of the San Marcos and Guadalupe Rivers. along coastal bays and rivers. utilizing the Intracoastal Canal. to Freeport. Brave men from all over the countryand several entrants from foreign countries--will test their endurance. skill. equipment. plain physical stamina. and even luck as they brave logjams. rocks. white water. strong winds. and exhausting portages. on a journey through some of the most beautiful country in Texas. I am submitting today a concurrent resolution granting official recognition to the event. The race Is being sponsored by a nonprofit organization expressly set up for this purpose. Prizes approaching $6.500 in value are being donated. along with several fine trophies. I believe this outstanding sports event. emphasizing courage. skill. endurance. and physical fitness deserves official recognition. and I am hopeful that the Senate will agree and act with dispatch on the resolution. Mr. President. I ask that the text of the concurrent resolution be printed at this point in the RECORD. followed by a history of the Texas water safari. prepared by Mr. James Hurst of the safari.\n"
     ]
    }
   ],
   "source": [
    "for word in myspeech[:500]:\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Navigating tabular data: column by column, rows within columns\n",
    "\n",
    "In the current dataset, many words are compiled into a list that is a 'speech' in Congress.  \n",
    "\n",
    "You can call the column 'speech' with square brackets, e.g.\n",
    "\n",
    "    congress['speech']\n",
    "\n",
    "Many speeches form a column called 'speech.'  The column speech can be called and treated as a list.\n",
    "\n",
    "You can call individual speeches with an additional set of square brackets after ['speech'], e.g. \n",
    "\n",
    "    congress['speech'][0]\n",
    "    \n",
    "-- which calls the first speech in the speech column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Those who do not enjoy the privilege of the floor will please retire from the Chamber.'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speech'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Mr. President. on the basis of an agreement reached on both sides. it is suggested that the Chamber be cleared of all attaches. unless they have absolutely important business to attend to in the Chamber.'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speech'][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Members of the Senate have heard the remarks of the distinguished majority leader. All attaches and staff members who are not vitally needed for the next few minutes of the deliberations of the Senate will tetire from the Chamber.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speech'][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can work on the text -- for instance cleaning or counting -- by calling each row in a text column, one at a time, and executing a transformation, via a for-loop.\n",
    "\n",
    "Here are the last hundred characters of the last five speeches in the dataframe, in upper case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***\n",
      "here are the last ten words of a speech:\n",
      "ROLLCALL NOS. 662 AND 661. I WAS ABSENT FROM THE HOUSE. HAD I BEEN PRESENT. I WOULD HAVE VOTED \"NO.\"\n",
      "***\n",
      "here are the last ten words of a speech:\n",
      "UL TO PROTECTING THE CONSTITUTION OF THE UNITED STATES AND THE GOALS OF OUR GREAT NATION. GOD BLESS.\n",
      "***\n",
      "here are the last ten words of a speech:\n",
      "AKER. ON ROLICALL NO. 658. I WAS UNAVOIDABLY DETAINED. HAD I BEEN PRESENT. I WOULD HAVE VOTED \"YES.\"\n",
      "***\n",
      "here are the last ten words of a speech:\n",
      "LCALL NO. 658 MY FLIGHT WAS DELAYED DUE TO WEATHER AND HAD I BEEN PRESENT. I WOULD HAVE VOTED \"YES.\"\n",
      "***\n",
      "here are the last ten words of a speech:\n",
      "ME BEFORE THE HOUSE. AND DONATED MY RAISE TO LOCAL NONPROFIT ORGANIZATIONS RATHER THAN ACCEPTING IT.\n"
     ]
    }
   ],
   "source": [
    "for speech in congress['speech'][-5:]:\n",
    "    speech = speech.upper()\n",
    "    print('***')\n",
    "    print('here are the last ten words of a speech:')\n",
    "    print(speech[-100:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Counting with Tabular Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use two commands that we have seen before to count tabular data.\n",
    "\n",
    "    .count() -- produces a count of how many items are in a category.  Generally speaking this is the same as counting the number of rows.\n",
    "    .value_counts() -- produces the subtotals for every subcategory listed in a column. We have used this command previously to get the word counts for every word in a list.  We will use value_counts() to get word counts for every word in a column in pandas.\n",
    "    \n",
    "We will also use one new command to count how many unique objects there are in a category.\n",
    "\n",
    "    .unique() -- finds only the unique members of a list\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's easier to understand the difference between these commands in practice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**.count()** on its own gives you the number of rows in the dataframe as a whole.  For our data, that just means the total number of speeches. \n",
    "\n",
    "Even if .count() is applied to the column speaker, it's still measuring the total number of individual speeches -- not how many unique speakers there are.  Most speakers are responsible for more than one speech, so their name appears several times in the dataset.  The count() below counts all rows in the dataframe, regardless of how many speakers there are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5992068"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**.value_counts()** organizes the data by unique values and then creates a count of each.  We will use it for word count, as we have in the past.  \n",
    "\n",
    "Applied to the speaker column, value_counts() givesyou a list of how many speeches each speaker gave."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "The PRESIDING OFFICER      709041\n",
       "The SPEAKER pro tempore    239201\n",
       "The CHAIRMAN               137788\n",
       "The SPEAKER                 86866\n",
       "Mr. ROBERT C. BYRD          75733\n",
       "                            ...  \n",
       "Fr. JOHN MCDONNELL              1\n",
       "Mr. oQTINGER                    1\n",
       "Mr. MOYIHAN                     1\n",
       "Mr. HOI.JNGS                    1\n",
       "Miss BEULA EDMISTON             1\n",
       "Name: speaker, Length: 56350, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you want the names of every speaker?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       The VICE PRESIDENT\n",
       "1            Mr. MANSFIELD\n",
       "2       The VICE PRESIDENT\n",
       "3       The VICE PRESIDENT\n",
       "4       Mrs. AGNES BAGGETT\n",
       "5       The VICE PRESIDENT\n",
       "6    The legislative clerk\n",
       "7            Mr. MANSFIELD\n",
       "8       The VICE PRESIDENT\n",
       "9    Mr. LONG of Louisiana\n",
       "Name: speaker, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot of repetition in the 'speaker' column.  If you want to list the names of every speaker only ONCE, you need the \"unique\" values.\n",
    "\n",
    "To get the unique values in the speaker column you will use the **.unique()** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['The VICE PRESIDENT', 'Mr. MANSFIELD', 'Mrs. AGNES BAGGETT',\n",
       "       'The legislative clerk', 'Mr. LONG of Louisiana',\n",
       "       'Mr. ROBERT A. BRENKWORTH', 'Mr. PASTORE', 'Mr. RUSSELL',\n",
       "       'Mr. KUCHEL', 'Mr. CLARK'], dtype=object)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speaker'].unique()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you need to know how many unique speakers are represented in this dataframe? You can use **len()** to give you the length -- that is the number of items -- in any list.  The length of a unique set is the number of unique answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56350"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(congress['speaker'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Counting particular words per cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we noted that .count(), applied to a column, will give you the number of rows in the column.\n",
    "\n",
    "You can also use .count() to find the number of times any individual strings occurs in each cell of a column. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how many times the first ten speeches include the word 'the.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     3\n",
       "1     4\n",
       "2     7\n",
       "3    18\n",
       "4     0\n",
       "5     5\n",
       "6     9\n",
       "7     1\n",
       "8     1\n",
       "9     1\n",
       "Name: speech, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speech'][:10].str.count('the')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how many times the first ten speeches include the word 'pineapple.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    0\n",
       "5    0\n",
       "6    0\n",
       "7    0\n",
       "8    0\n",
       "9    0\n",
       "Name: speech, dtype: int64"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['speech'][:10].str.count('pineapple')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of our search for 'pineapple' says that most speeches in Congress talk about pineapples zero times.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note that we have here searched just for the string 'pineapple.' This method could create confusion in future searches unless we used regex to look for an exact word -- unless we really only care about 'pineapple,' which is unusual enough to produce good results as a free-standing string. We will not go into using regex to improve searches here, because we have previously covered this material in another notebook***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we only want the speeches that talked about pineapples the most?\n",
    "\n",
    "If we use the function **.nlargest()**, pandas will return the rows with the highest counts of the foregoing count.  \n",
    "\n",
    "The table below lists the row numbers of the speeches where pineapples are mentioned the most:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000851    65\n",
       "1084391    51\n",
       "1017189    29\n",
       "2164092    26\n",
       "1023495    24\n",
       "Name: speech, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pineapplespeeches = congress['speech'].str.count('pineapple').nlargest(5)\n",
    "pineapplespeeches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the list of speeches above is stripped down to just a **row number** and a **count**.  When we use **nlargest**, it tends to drop all the information other than the count and the row number.  Fortunately, we can use **.iloc** to navigate from row number to all the relevant information "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to print the results, using .loc and .iloc to call the speech by its index number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mr. President. I am introducing legislation today to enable Hawaiian pineapple products to compete in the U.S. market with lowcost foreign canned pineapple which can easily undersell Hawaiian pineapple. One of the finest products in all America is the sweet. juicy. delectable pineapple grown in Hawaii. Since the turn of the century. pineapple has been a mainstay in Hawaiis economy. Today it is still my States second largest agricultural industay. second only to sugar. The processed value of Hawaiian pineapple last year was $137 million. The industry employs 6.200 yearround workers who earned $42 million in annual wages and another 12.000 seasonal workers who earn a total of $10 million a year. Hawaiis pineapple industry has been very energetic and progressive. investing millions of its own dollars in research to improve pineapple quality and production. The Hawaiian pineapple industry is the most highly mechanized In the world and its fleldworkers are the highest paid in the world. The\n"
     ]
    }
   ],
   "source": [
    "for word in list(congress.loc[:, ['speech']].iloc[1000851, ]):\n",
    "        print(word[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to call the speaker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "speaker    Mr. FONG\n",
       "Name: 1000851, dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.loc[:, ['speaker']].iloc[1000851, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to call the date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date    1973-02-01\n",
       "Name: 1000851, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress.loc[:, ['date']].iloc[1000851, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to call up a series of speeches from the list above, *pineapplespeeches*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***\n",
      "speaker    here is a speech about pineapples by Mrs. MINK:\n",
      "Name: 1017189, dtype: object\n",
      "Mr. Speaker. the Hawaii pineapple industry as we know it is on the verge of extinction. Twenty years ago Hawaii had nine pineapple companies. but this number has dwindled at an accelerating pace. Today there are only four left. The demise of another within a year has already been announced. Most recently. one of the remaining three said it will discontinue its operation on the Island of Molokai by 1975 or 1976. Another of the three also announced it will terminate its activities on Molokai. leaving the island virtually without any industry of any kind. The handwriting is on the wall. and we must anticipate that in a very short time there will be no pineapple canning industry left. The only thing remaining will be fresh pineapple. which will be grown on a limited basis. In 1950. Hawaii had 72 percent of world pineapple production. Now we have less than half that figure. Simply stated. Hawaiis pineapple industry has declined while competitive pineapple industries in other parts of the wo\n",
      "date    1973-03-20\n",
      "Name: 1017189, dtype: object\n",
      "***\n",
      "speaker    here is a speech about pineapples by Mr. MATSU...\n",
      "Name: 2164092, dtype: object\n",
      "Mr. President. I well realize that the passage of this bill is considered a prerequisite to the successful conclusion of the multilateral negotiations in Geneva. The administration has predicted serious disruptive consequences if the multilateral trade negotiations are not brought to a successful conclusion. In my position as chief deputy majority whip and as a member of the Democratic Party. I have consistently supported the President. However. the special trade representatives office has acted to reduce the tariff protection for a vital Hawaiian industry. and the secretive. uncooperative position taken by oi trade representatives. greatly strains my basic goodwill toward the administration. The problem first surfaced in December of 1978. when. just by chance. I learned that the U.S. delegation to the multilateral trade negotiations received authority to offer a concession on pineapple and pineapple juice. in response to a request from the Philippines. After learning of this. I immedi\n",
      "date    1979-03-28\n",
      "Name: 2164092, dtype: object\n",
      "***\n",
      "speaker    here is a speech about pineapples by Mrs. MINK:\n",
      "Name: 1023495, dtype: object\n",
      "Mr. Speaker. many areas of our Nation are suffering economic distress as a result of increased competition for their products by foreign imports. but I doubt that any has been affected as severely as parts of the State of Hawaii. Hawaii is known as the worlds pineapple production center. and indeed the State did pioneer the industry by developing new agricultural techniques which dramatically increased the availability of this wholesome and well liked fruit. In recent years. however. Hawaiis pineapple industry has been threatened with extinctionlargely due to the adaptation of its expertise in other areas of the world with abundant supplies of lowcost labor. In some cases. the Federal Government helped to finance the start of overseas pineapple operations. In others. American pineapple companies founded foreign subsidiaries. Because of such factors. Hawaiis share of the world pineapple market has plummeted from 72 percent in 1950 to less than 33 percent today. During 195071. imports of\n",
      "date    1973-04-03\n",
      "Name: 1023495, dtype: object\n"
     ]
    }
   ],
   "source": [
    "for speechnumber in pineapplespeeches.index[-3:]:\n",
    "    for word in list(congress.loc[:, ['speech']].iloc[speechnumber, ]):\n",
    "        speaker = congress.loc[:, ['speaker']].iloc[speechnumber, ]\n",
    "        date = congress.loc[:, ['date']].iloc[speechnumber, ]\n",
    "        print('***')\n",
    "        print('here is a speech about pineapples by ' + speaker + ':')\n",
    "        print(word[:1000])\n",
    "        print(date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsetting Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the python grammar of operators to ask Python to only look at certain parts of the data -- or 'subsets' of the complete dataset.\n",
    "\n",
    "For instance, if we want *only* the data from the 1980s, we can use square brackets **[ ]** to tell python to subset a dataframe.  \n",
    "\n",
    "We use square brackets **[ ]** to tell python to subset a dataframe according to the constraints inside the brackets.\n",
    "\n",
    "The command to subset data is expressed with the grammar:\n",
    "\n",
    "    df[df['columnname'].LIMITINGOPERATOR]\n",
    "\n",
    "\n",
    "For instance, df[df['speaker']=='bob']] would tell python to find only the rows of the dataframe where 'bob' was listed as the speaker.\n",
    "\n",
    "Using square brackets to \"filter\" for particular rows is one of the major ways of navigating tabular data in pandas.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The operators for filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following 'operators' are the ones most frequently used to tell Python how to narrow down the data.  Each works a slightly different way: \n",
    "\n",
    "    .isin() -- tells Python to only look for values that are in another list\n",
    "    == -- tells Python to only look for values that are equal to another value\n",
    "    != -- tells Python to only look for values that are NOT equal to another value\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7475</th>\n",
       "      <td>7475</td>\n",
       "      <td>7475</td>\n",
       "      <td>Mr. Speaker. the January 8. 1967. \"Doanes Agri...</td>\n",
       "      <td>1967-02-01</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>201</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8657</th>\n",
       "      <td>8657</td>\n",
       "      <td>8657</td>\n",
       "      <td>Mr. Speaker. I ask unanimous consent to revise...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>12</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8659</th>\n",
       "      <td>8659</td>\n",
       "      <td>8659</td>\n",
       "      <td>Mr. Speaker. I join in the statements made by ...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>301</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8767</th>\n",
       "      <td>8767</td>\n",
       "      <td>8767</td>\n",
       "      <td>Mr. Speaker. it is my pleasure to join in the ...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>878</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12255</th>\n",
       "      <td>12255</td>\n",
       "      <td>12255</td>\n",
       "      <td>Mr. Speaker. today I have introduced a joint r...</td>\n",
       "      <td>1967-02-09</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>82</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19034</th>\n",
       "      <td>19034</td>\n",
       "      <td>19034</td>\n",
       "      <td>Mr. Speaker. it is my pleasure to join Mrs. BO...</td>\n",
       "      <td>1967-02-28</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>258</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20616</th>\n",
       "      <td>20616</td>\n",
       "      <td>20616</td>\n",
       "      <td>Mr. Speaker. I wish to associate myself with t...</td>\n",
       "      <td>1967-03-02</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>291</td>\n",
       "      <td>1967</td>\n",
       "      <td>3</td>\n",
       "      <td>1967-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24507</th>\n",
       "      <td>24507</td>\n",
       "      <td>24507</td>\n",
       "      <td>Mr. Speaker. during this year of 1967 the Fede...</td>\n",
       "      <td>1967-03-08</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>122</td>\n",
       "      <td>1967</td>\n",
       "      <td>3</td>\n",
       "      <td>1967-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25378</th>\n",
       "      <td>25378</td>\n",
       "      <td>25378</td>\n",
       "      <td>Mr. Speaker. will the gentleman yield?</td>\n",
       "      <td>1967-03-09</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>6</td>\n",
       "      <td>1967</td>\n",
       "      <td>3</td>\n",
       "      <td>1967-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25380</th>\n",
       "      <td>25380</td>\n",
       "      <td>25380</td>\n",
       "      <td>Mr. Speaker. permit me to say. first of all. t...</td>\n",
       "      <td>1967-03-09</td>\n",
       "      <td>Mr. DOLE</td>\n",
       "      <td>78</td>\n",
       "      <td>1967</td>\n",
       "      <td>3</td>\n",
       "      <td>1967-03-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0  Unnamed: 0.1  \\\n",
       "7475         7475          7475   \n",
       "8657         8657          8657   \n",
       "8659         8659          8659   \n",
       "8767         8767          8767   \n",
       "12255       12255         12255   \n",
       "19034       19034         19034   \n",
       "20616       20616         20616   \n",
       "24507       24507         24507   \n",
       "25378       25378         25378   \n",
       "25380       25380         25380   \n",
       "\n",
       "                                                  speech        date  \\\n",
       "7475   Mr. Speaker. the January 8. 1967. \"Doanes Agri...  1967-02-01   \n",
       "8657   Mr. Speaker. I ask unanimous consent to revise...  1967-02-02   \n",
       "8659   Mr. Speaker. I join in the statements made by ...  1967-02-02   \n",
       "8767   Mr. Speaker. it is my pleasure to join in the ...  1967-02-02   \n",
       "12255  Mr. Speaker. today I have introduced a joint r...  1967-02-09   \n",
       "19034  Mr. Speaker. it is my pleasure to join Mrs. BO...  1967-02-28   \n",
       "20616  Mr. Speaker. I wish to associate myself with t...  1967-03-02   \n",
       "24507  Mr. Speaker. during this year of 1967 the Fede...  1967-03-08   \n",
       "25378             Mr. Speaker. will the gentleman yield?  1967-03-09   \n",
       "25380  Mr. Speaker. permit me to say. first of all. t...  1967-03-09   \n",
       "\n",
       "        speaker  word_count  year  month  month_year  \n",
       "7475   Mr. DOLE         201  1967      2  1967-02-01  \n",
       "8657   Mr. DOLE          12  1967      2  1967-02-01  \n",
       "8659   Mr. DOLE         301  1967      2  1967-02-01  \n",
       "8767   Mr. DOLE         878  1967      2  1967-02-01  \n",
       "12255  Mr. DOLE          82  1967      2  1967-02-01  \n",
       "19034  Mr. DOLE         258  1967      2  1967-02-01  \n",
       "20616  Mr. DOLE         291  1967      3  1967-03-01  \n",
       "24507  Mr. DOLE         122  1967      3  1967-03-01  \n",
       "25378  Mr. DOLE           6  1967      3  1967-03-01  \n",
       "25380  Mr. DOLE          78  1967      3  1967-03-01  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress[congress['speaker'] == 'Mr. DOLE'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the speeches of Mr. Fong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1907</th>\n",
       "      <td>1907</td>\n",
       "      <td>1907</td>\n",
       "      <td>Mr. President. the Senate is now considering t...</td>\n",
       "      <td>1967-01-17</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>873</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7621</th>\n",
       "      <td>7621</td>\n",
       "      <td>7621</td>\n",
       "      <td>Mr. President. I introduce. for appropriate re...</td>\n",
       "      <td>1967-02-01</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>313</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8303</th>\n",
       "      <td>8303</td>\n",
       "      <td>8303</td>\n",
       "      <td>Mr. President. will the distinguished Senator ...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>9</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8305</th>\n",
       "      <td>8305</td>\n",
       "      <td>8305</td>\n",
       "      <td>Mr. President. I commend the distinguished sen...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>183</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8309</th>\n",
       "      <td>8309</td>\n",
       "      <td>8309</td>\n",
       "      <td>I thank the distinguished Senator for his very...</td>\n",
       "      <td>1967-02-02</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>1361</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9006</th>\n",
       "      <td>9006</td>\n",
       "      <td>9006</td>\n",
       "      <td>Mr. President. it is gratifying to call attent...</td>\n",
       "      <td>1967-02-03</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>417</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9757</th>\n",
       "      <td>9757</td>\n",
       "      <td>9757</td>\n",
       "      <td>Mr. President. I am particularly pleased that ...</td>\n",
       "      <td>1967-02-06</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>173</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10922</th>\n",
       "      <td>10922</td>\n",
       "      <td>10922</td>\n",
       "      <td>Mr. President. it was with deep grief and shoc...</td>\n",
       "      <td>1967-02-08</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>429</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18460</th>\n",
       "      <td>18460</td>\n",
       "      <td>18460</td>\n",
       "      <td>Mr. President. the current unrest and chaos th...</td>\n",
       "      <td>1967-02-28</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>179</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18461</th>\n",
       "      <td>18461</td>\n",
       "      <td>18461</td>\n",
       "      <td>Mr. President. during extensive hearings. cond...</td>\n",
       "      <td>1967-02-28</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>1206</td>\n",
       "      <td>1967</td>\n",
       "      <td>2</td>\n",
       "      <td>1967-02-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0  Unnamed: 0.1  \\\n",
       "1907         1907          1907   \n",
       "7621         7621          7621   \n",
       "8303         8303          8303   \n",
       "8305         8305          8305   \n",
       "8309         8309          8309   \n",
       "9006         9006          9006   \n",
       "9757         9757          9757   \n",
       "10922       10922         10922   \n",
       "18460       18460         18460   \n",
       "18461       18461         18461   \n",
       "\n",
       "                                                  speech        date  \\\n",
       "1907   Mr. President. the Senate is now considering t...  1967-01-17   \n",
       "7621   Mr. President. I introduce. for appropriate re...  1967-02-01   \n",
       "8303   Mr. President. will the distinguished Senator ...  1967-02-02   \n",
       "8305   Mr. President. I commend the distinguished sen...  1967-02-02   \n",
       "8309   I thank the distinguished Senator for his very...  1967-02-02   \n",
       "9006   Mr. President. it is gratifying to call attent...  1967-02-03   \n",
       "9757   Mr. President. I am particularly pleased that ...  1967-02-06   \n",
       "10922  Mr. President. it was with deep grief and shoc...  1967-02-08   \n",
       "18460  Mr. President. the current unrest and chaos th...  1967-02-28   \n",
       "18461  Mr. President. during extensive hearings. cond...  1967-02-28   \n",
       "\n",
       "        speaker  word_count  year  month  month_year  \n",
       "1907   Mr. FONG         873  1967      1  1967-01-01  \n",
       "7621   Mr. FONG         313  1967      2  1967-02-01  \n",
       "8303   Mr. FONG           9  1967      2  1967-02-01  \n",
       "8305   Mr. FONG         183  1967      2  1967-02-01  \n",
       "8309   Mr. FONG        1361  1967      2  1967-02-01  \n",
       "9006   Mr. FONG         417  1967      2  1967-02-01  \n",
       "9757   Mr. FONG         173  1967      2  1967-02-01  \n",
       "10922  Mr. FONG         429  1967      2  1967-02-01  \n",
       "18460  Mr. FONG         179  1967      2  1967-02-01  \n",
       "18461  Mr. FONG        1206  1967      2  1967-02-01  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress[congress['speaker'] == 'Mr. FONG'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using .isin()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the operator **.isin()** to filter our results via a certain list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are all the speeches that took place in the summer months, June-July-August, i.e. months 6-7-8:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63624</th>\n",
       "      <td>63624</td>\n",
       "      <td>63624</td>\n",
       "      <td>Mr. President. on April 19 the United Auto Wor...</td>\n",
       "      <td>1967-06-01</td>\n",
       "      <td>Mr. HARTKE</td>\n",
       "      <td>227</td>\n",
       "      <td>1967</td>\n",
       "      <td>6</td>\n",
       "      <td>1967-06-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63625</th>\n",
       "      <td>63625</td>\n",
       "      <td>63625</td>\n",
       "      <td>Without objection. it is so ordered. The resol...</td>\n",
       "      <td>1967-06-01</td>\n",
       "      <td>The PRESIDING OFFICER</td>\n",
       "      <td>17</td>\n",
       "      <td>1967</td>\n",
       "      <td>6</td>\n",
       "      <td>1967-06-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63626</th>\n",
       "      <td>63626</td>\n",
       "      <td>63626</td>\n",
       "      <td>Mr. President. I ask unanimous consent that th...</td>\n",
       "      <td>1967-06-01</td>\n",
       "      <td>Mr. McCLELLAN</td>\n",
       "      <td>47</td>\n",
       "      <td>1967</td>\n",
       "      <td>6</td>\n",
       "      <td>1967-06-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63627</th>\n",
       "      <td>63627</td>\n",
       "      <td>63627</td>\n",
       "      <td>Without objection. it is so ordered.</td>\n",
       "      <td>1967-06-01</td>\n",
       "      <td>The PRESIDING OFFICER</td>\n",
       "      <td>6</td>\n",
       "      <td>1967</td>\n",
       "      <td>6</td>\n",
       "      <td>1967-06-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63628</th>\n",
       "      <td>63628</td>\n",
       "      <td>63628</td>\n",
       "      <td>Mr. President. I ask unanimous consent that th...</td>\n",
       "      <td>1967-06-01</td>\n",
       "      <td>Mr. DIRKSEN</td>\n",
       "      <td>23</td>\n",
       "      <td>1967</td>\n",
       "      <td>6</td>\n",
       "      <td>1967-06-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5970495</th>\n",
       "      <td>5970495</td>\n",
       "      <td>5970495</td>\n",
       "      <td>Madam Speaker. I rise today to commend and con...</td>\n",
       "      <td>2010-07-30</td>\n",
       "      <td>Mr. RADANOVICH</td>\n",
       "      <td>239</td>\n",
       "      <td>2010</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-07-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5970496</th>\n",
       "      <td>5970496</td>\n",
       "      <td>5970496</td>\n",
       "      <td>Madam Speaker. I rise today to introduce the \"...</td>\n",
       "      <td>2010-07-30</td>\n",
       "      <td>Ms. LINDA T. SANCHEZ of California</td>\n",
       "      <td>307</td>\n",
       "      <td>2010</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-07-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5970497</th>\n",
       "      <td>5970497</td>\n",
       "      <td>5970497</td>\n",
       "      <td>Madam Speaker. I wish to speak today about an ...</td>\n",
       "      <td>2010-07-30</td>\n",
       "      <td>Mr. SHERMAN</td>\n",
       "      <td>277</td>\n",
       "      <td>2010</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-07-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5970498</th>\n",
       "      <td>5970498</td>\n",
       "      <td>5970498</td>\n",
       "      <td>Madam Speaker. I rise today to pay tribute and...</td>\n",
       "      <td>2010-07-30</td>\n",
       "      <td>Mr. WALDEN</td>\n",
       "      <td>677</td>\n",
       "      <td>2010</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-07-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5970499</th>\n",
       "      <td>5970499</td>\n",
       "      <td>5970499</td>\n",
       "      <td>Madam Speaker. I rise today to honor Pearl Har...</td>\n",
       "      <td>2010-07-30</td>\n",
       "      <td>Mr. SMITH of Washington</td>\n",
       "      <td>358</td>\n",
       "      <td>2010</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-07-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1525953 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0  Unnamed: 0.1  \\\n",
       "63624         63624         63624   \n",
       "63625         63625         63625   \n",
       "63626         63626         63626   \n",
       "63627         63627         63627   \n",
       "63628         63628         63628   \n",
       "...             ...           ...   \n",
       "5970495     5970495       5970495   \n",
       "5970496     5970496       5970496   \n",
       "5970497     5970497       5970497   \n",
       "5970498     5970498       5970498   \n",
       "5970499     5970499       5970499   \n",
       "\n",
       "                                                    speech        date  \\\n",
       "63624    Mr. President. on April 19 the United Auto Wor...  1967-06-01   \n",
       "63625    Without objection. it is so ordered. The resol...  1967-06-01   \n",
       "63626    Mr. President. I ask unanimous consent that th...  1967-06-01   \n",
       "63627                 Without objection. it is so ordered.  1967-06-01   \n",
       "63628    Mr. President. I ask unanimous consent that th...  1967-06-01   \n",
       "...                                                    ...         ...   \n",
       "5970495  Madam Speaker. I rise today to commend and con...  2010-07-30   \n",
       "5970496  Madam Speaker. I rise today to introduce the \"...  2010-07-30   \n",
       "5970497  Madam Speaker. I wish to speak today about an ...  2010-07-30   \n",
       "5970498  Madam Speaker. I rise today to pay tribute and...  2010-07-30   \n",
       "5970499  Madam Speaker. I rise today to honor Pearl Har...  2010-07-30   \n",
       "\n",
       "                                    speaker  word_count  year  month  \\\n",
       "63624                            Mr. HARTKE         227  1967      6   \n",
       "63625                 The PRESIDING OFFICER          17  1967      6   \n",
       "63626                         Mr. McCLELLAN          47  1967      6   \n",
       "63627                 The PRESIDING OFFICER           6  1967      6   \n",
       "63628                           Mr. DIRKSEN          23  1967      6   \n",
       "...                                     ...         ...   ...    ...   \n",
       "5970495                      Mr. RADANOVICH         239  2010      7   \n",
       "5970496  Ms. LINDA T. SANCHEZ of California         307  2010      7   \n",
       "5970497                         Mr. SHERMAN         277  2010      7   \n",
       "5970498                          Mr. WALDEN         677  2010      7   \n",
       "5970499             Mr. SMITH of Washington         358  2010      7   \n",
       "\n",
       "         month_year  \n",
       "63624    1967-06-01  \n",
       "63625    1967-06-01  \n",
       "63626    1967-06-01  \n",
       "63627    1967-06-01  \n",
       "63628    1967-06-01  \n",
       "...             ...  \n",
       "5970495  2010-07-01  \n",
       "5970496  2010-07-01  \n",
       "5970497  2010-07-01  \n",
       "5970498  2010-07-01  \n",
       "5970499  2010-07-01  \n",
       "\n",
       "[1525953 rows x 9 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress[congress['month'].isin(range(6, 8))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we make a list of all the speakers to spoke the most about pineapples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Mr. FONG', 'Mrs. MINK', 'Mr. MATSUNAGA']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pineapplespeakers = []\n",
    "\n",
    "for speechnumber in list(pineapplespeeches.index):\n",
    "        speaker = list(congress.loc[:, ['speaker']].iloc[speechnumber, ])[0]\n",
    "        if speaker not in pineapplespeakers:\n",
    "            pineapplespeakers.append(speaker)\n",
    "        \n",
    "pineapplespeakers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are all the speeches by the speakers who spoke the most about pineapples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1541</th>\n",
       "      <td>1541</td>\n",
       "      <td>1541</td>\n",
       "      <td>Mr. Speaker. it was my great privilege to list...</td>\n",
       "      <td>1967-01-16</td>\n",
       "      <td>Mr. MATSUNAGA</td>\n",
       "      <td>281</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1907</th>\n",
       "      <td>1907</td>\n",
       "      <td>1907</td>\n",
       "      <td>Mr. President. the Senate is now considering t...</td>\n",
       "      <td>1967-01-17</td>\n",
       "      <td>Mr. FONG</td>\n",
       "      <td>873</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2136</th>\n",
       "      <td>2136</td>\n",
       "      <td>2136</td>\n",
       "      <td>Mr. Speaker. the editorial entitled \"Speaker o...</td>\n",
       "      <td>1967-01-17</td>\n",
       "      <td>Mr. MATSUNAGA</td>\n",
       "      <td>347</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2276</th>\n",
       "      <td>2276</td>\n",
       "      <td>2276</td>\n",
       "      <td>Mr. Speaker. I rise to pay tribute to the memo...</td>\n",
       "      <td>1967-01-18</td>\n",
       "      <td>Mr. MATSUNAGA</td>\n",
       "      <td>18</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2296</th>\n",
       "      <td>2296</td>\n",
       "      <td>2296</td>\n",
       "      <td>Mr. Speaker. I. along with all my colleagues. ...</td>\n",
       "      <td>1967-01-18</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>200</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4742933</th>\n",
       "      <td>4742933</td>\n",
       "      <td>4742933</td>\n",
       "      <td>o1 Hawai. Mr. Chairman. I rise In support of t...</td>\n",
       "      <td>1998-03-25</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>620</td>\n",
       "      <td>1998</td>\n",
       "      <td>3</td>\n",
       "      <td>1998-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4931395</th>\n",
       "      <td>4931395</td>\n",
       "      <td>4931395</td>\n",
       "      <td>Madam Chairman. I believe strongly that all ch...</td>\n",
       "      <td>1999-10-26</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>484</td>\n",
       "      <td>1999</td>\n",
       "      <td>10</td>\n",
       "      <td>1999-10-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5003189</th>\n",
       "      <td>5003189</td>\n",
       "      <td>5003189</td>\n",
       "      <td>I would have voted \"yea.\" On the amendment to ...</td>\n",
       "      <td>2000-07-13</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>51</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>2000-07-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5145728</th>\n",
       "      <td>5145728</td>\n",
       "      <td>5145728</td>\n",
       "      <td>Mr. Speaker. today I am introducing a bill dir...</td>\n",
       "      <td>2002-02-14</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>411</td>\n",
       "      <td>2002</td>\n",
       "      <td>2</td>\n",
       "      <td>2002-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5194618</th>\n",
       "      <td>5194618</td>\n",
       "      <td>5194618</td>\n",
       "      <td>Mr. Speaker. I rise today to honor our disting...</td>\n",
       "      <td>2002-07-24</td>\n",
       "      <td>Mrs. MINK</td>\n",
       "      <td>11</td>\n",
       "      <td>2002</td>\n",
       "      <td>7</td>\n",
       "      <td>2002-07-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9910 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0  Unnamed: 0.1  \\\n",
       "1541           1541          1541   \n",
       "1907           1907          1907   \n",
       "2136           2136          2136   \n",
       "2276           2276          2276   \n",
       "2296           2296          2296   \n",
       "...             ...           ...   \n",
       "4742933     4742933       4742933   \n",
       "4931395     4931395       4931395   \n",
       "5003189     5003189       5003189   \n",
       "5145728     5145728       5145728   \n",
       "5194618     5194618       5194618   \n",
       "\n",
       "                                                    speech        date  \\\n",
       "1541     Mr. Speaker. it was my great privilege to list...  1967-01-16   \n",
       "1907     Mr. President. the Senate is now considering t...  1967-01-17   \n",
       "2136     Mr. Speaker. the editorial entitled \"Speaker o...  1967-01-17   \n",
       "2276     Mr. Speaker. I rise to pay tribute to the memo...  1967-01-18   \n",
       "2296     Mr. Speaker. I. along with all my colleagues. ...  1967-01-18   \n",
       "...                                                    ...         ...   \n",
       "4742933  o1 Hawai. Mr. Chairman. I rise In support of t...  1998-03-25   \n",
       "4931395  Madam Chairman. I believe strongly that all ch...  1999-10-26   \n",
       "5003189  I would have voted \"yea.\" On the amendment to ...  2000-07-13   \n",
       "5145728  Mr. Speaker. today I am introducing a bill dir...  2002-02-14   \n",
       "5194618  Mr. Speaker. I rise today to honor our disting...  2002-07-24   \n",
       "\n",
       "               speaker  word_count  year  month  month_year  \n",
       "1541     Mr. MATSUNAGA         281  1967      1  1967-01-01  \n",
       "1907          Mr. FONG         873  1967      1  1967-01-01  \n",
       "2136     Mr. MATSUNAGA         347  1967      1  1967-01-01  \n",
       "2276     Mr. MATSUNAGA          18  1967      1  1967-01-01  \n",
       "2296         Mrs. MINK         200  1967      1  1967-01-01  \n",
       "...                ...         ...   ...    ...         ...  \n",
       "4742933      Mrs. MINK         620  1998      3  1998-03-01  \n",
       "4931395      Mrs. MINK         484  1999     10  1999-10-01  \n",
       "5003189      Mrs. MINK          51  2000      7  2000-07-01  \n",
       "5145728      Mrs. MINK         411  2002      2  2002-02-01  \n",
       "5194618      Mrs. MINK          11  2002      7  2002-07-01  \n",
       "\n",
       "[9910 rows x 9 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress[congress['speaker'].isin(pineapplespeakers)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using .isin() to find data from the 1980s\n",
    "\n",
    "In the following line of code, we'll use **.isin()** to tell Python to look for values in the 1980s.  We tell Python to look at the 'year' column. Then we select only the years that are in a list of years from the 1980s. \n",
    "\n",
    "    eighties_data = congress[congress['year'].isin(target_years)].copy()  # filter our dataset to just this decade\n",
    "\n",
    "**.isin()**  takes as its object a list, for instance the *target_years* variable, which we will create to include every year from 1980 to 1990.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we apply .isin(), however, we need to format the data so that we can navigate for time.\n",
    "\n",
    "First, we need to make a 'year' column.\n",
    "\n",
    "Then we need to filter for years that are in our target.  Note the use of the .isin() function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call the datetime package\n",
    "    \n",
    "    .dt.year\n",
    "\n",
    "to create a new column called 'year'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Those who do not enjoy the privilege of the fl...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>16</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. President. on the basis of an agreement re...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>Mr. MANSFIELD</td>\n",
       "      <td>35</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>The Members of the Senate have heard the remar...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>40</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>The Chair lays before the Senate the following...</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>The VICE PRESIDENT</td>\n",
       "      <td>151</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>Secretary of State.</td>\n",
       "      <td>1967-01-10</td>\n",
       "      <td>Mrs. AGNES BAGGETT</td>\n",
       "      <td>3</td>\n",
       "      <td>1967</td>\n",
       "      <td>1</td>\n",
       "      <td>1967-01-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  Unnamed: 0.1  \\\n",
       "0           0             0   \n",
       "1           1             1   \n",
       "2           2             2   \n",
       "3           3             3   \n",
       "4           4             4   \n",
       "\n",
       "                                              speech        date  \\\n",
       "0  Those who do not enjoy the privilege of the fl...  1967-01-10   \n",
       "1  Mr. President. on the basis of an agreement re...  1967-01-10   \n",
       "2  The Members of the Senate have heard the remar...  1967-01-10   \n",
       "3  The Chair lays before the Senate the following...  1967-01-10   \n",
       "4                                Secretary of State.  1967-01-10   \n",
       "\n",
       "              speaker  word_count  year  month  month_year  \n",
       "0  The VICE PRESIDENT          16  1967      1  1967-01-01  \n",
       "1       Mr. MANSFIELD          35  1967      1  1967-01-01  \n",
       "2  The VICE PRESIDENT          40  1967      1  1967-01-01  \n",
       "3  The VICE PRESIDENT         151  1967      1  1967-01-01  \n",
       "4  Mrs. AGNES BAGGETT           3  1967      1  1967-01-01  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "congress['year']=pd.to_datetime(congress['date']).dt.year # make a year column\n",
    "\n",
    "congress.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using == to subset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2329890</th>\n",
       "      <td>2329890</td>\n",
       "      <td>2329890</td>\n",
       "      <td>Mr. Speaker. we in Delaware are proud of the o...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. EVANS of Delaware</td>\n",
       "      <td>122</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2329891</th>\n",
       "      <td>2329891</td>\n",
       "      <td>2329891</td>\n",
       "      <td>Mr. Speaker. it is logical for Americans to be...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. DERWINSKI</td>\n",
       "      <td>82</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2329892</th>\n",
       "      <td>2329892</td>\n",
       "      <td>2329892</td>\n",
       "      <td>The Chair has examined the Journal of the last...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>32</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2329893</th>\n",
       "      <td>2329893</td>\n",
       "      <td>2329893</td>\n",
       "      <td>Mr. Speaker. I ask unanimous consent that the ...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. WHITE</td>\n",
       "      <td>36</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2329894</th>\n",
       "      <td>2329894</td>\n",
       "      <td>2329894</td>\n",
       "      <td>Is there objection to the request of the gentl...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0  Unnamed: 0.1  \\\n",
       "2329890     2329890       2329890   \n",
       "2329891     2329891       2329891   \n",
       "2329892     2329892       2329892   \n",
       "2329893     2329893       2329893   \n",
       "2329894     2329894       2329894   \n",
       "\n",
       "                                                    speech        date  \\\n",
       "2329890  Mr. Speaker. we in Delaware are proud of the o...  1980-01-03   \n",
       "2329891  Mr. Speaker. it is logical for Americans to be...  1980-01-03   \n",
       "2329892  The Chair has examined the Journal of the last...  1980-01-03   \n",
       "2329893  Mr. Speaker. I ask unanimous consent that the ...  1980-01-03   \n",
       "2329894  Is there objection to the request of the gentl...  1980-01-03   \n",
       "\n",
       "                         speaker  word_count  year  month  month_year  \n",
       "2329890    Mr. EVANS of Delaware         122  1980      1  1980-01-01  \n",
       "2329891            Mr. DERWINSKI          82  1980      1  1980-01-01  \n",
       "2329892  The SPEAKER pro tempore          32  1980      1  1980-01-01  \n",
       "2329893                Mr. WHITE          36  1980      1  1980-01-01  \n",
       "2329894  The SPEAKER pro tempore          11  1980      1  1980-01-01  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1980 = congress[congress['year']== 1980].copy()  # filter our dataset to just this decade\n",
    "\n",
    "data1980.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using .isin() to subset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'congress' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-972279ecffc3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mtarget_years\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1980\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1989\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# List of the years 1980-1989\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0meighties_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcongress\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcongress\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'year'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtarget_years\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# filter our dataset to just this decade\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0meighties_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0meighties_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdrop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'index'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Unnamed: 0'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#minor reformatting - drop extra columns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0meighties_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'congress' is not defined"
     ]
    }
   ],
   "source": [
    "target_years = list(range(1980, 1989 + 1))  # List of the years 1980-1989\n",
    "\n",
    "eighties_data = congress[congress['year'].isin(target_years)].copy().reset_index()  # filter our dataset to just this decade\n",
    "eighties_data = eighties_data.drop(['index', 'Unnamed: 0'], 1) #minor reformatting - drop extra columns\n",
    "eighties_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the results in case we want to use them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/users/jguldi/digital-history\n"
     ]
    }
   ],
   "source": [
    "cd ~/digital-history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1980.to_csv(\"data1980.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data.to_csv(\"eighties_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsetting your data and saving copies of clean data is a good way to start work on your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to break speeches into words and remove stopwords.  We get our stopwords list from the package NLTK (natural language toolkit):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load stopwords as we have before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /users/jguldi/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\"]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('stopwords')\n",
    "\n",
    "stop = stopwords.words('english')\n",
    "stop[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll take a new and special preparation step here where we add some regex -- including the word boundary symbols you've seen before -- to make a list of stopwords that Python can search for with great ease.  Mainly you'll want to copy and paste the following line, rather than understanding it, but here are the components:\n",
    "\n",
    "    r'': 'begin regex, using the formula inside these quotation marks'\n",
    "    \\\\b: 'look for a word boundary'\n",
    "    (?:{}): 'search for the query word for each of the words in the query that follows'\n",
    "    '|'.join(stop): | means 'or', and .join() produces stopword1|stopword2|stopword3|etc... (where each stopword corresponds to 'i', 'me', 'mine,' etc.\n",
    "    \n",
    "Basically we're just formatting the stopwords list so that Python can search for the whole series efficently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords_regex = r'\\b(?:{})\\b'.format('|'.join(stop))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\\\b(?:i|me|my|myself|we|our|ours|ourselves|you|you're|you've|you'll|you'd|your|yours|yourself|yourselves|he|him|his|himself|she|she's|her|hers|herself|it|it's|its|itself|they|them|their|theirs|themselves|what|which|who|whom|this|that|that'll|these|those|am|is|are|was|were|be|been|being|have|has|had|having|do|does|did|doing|a|an|the|and|but|if|or|because|as|until|while|of|at|by|for|with|about|against|between|into|through|during|before|after|above|below|to|from|up|down|in|out|on|off|over|under|again|further|then|once|here|there|when|where|why|how|all|any|both|each|few|more|most|other|some|such|no|nor|not|only|own|same|so|than|too|very|s|t|can|will|just|don|don't|should|should've|now|d|ll|m|o|re|ve|y|ain|aren|aren't|couldn|couldn't|didn|didn't|doesn|doesn't|hadn|hadn't|hasn|hasn't|haven|haven't|isn|isn't|ma|mightn|mightn't|mustn|mustn't|needn|needn't|shan|shan't|shouldn|shouldn't|wasn|wasn't|weren|weren't|won|won't|wouldn|wouldn't)\\\\b\""
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stopwords_regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To clean our text when our text is in tabular form, we can apply many commands that are familiar.  Technically, they are being applied over each row of the pandas dataframe.  But the pandas software makes it easier for us.\n",
    "\n",
    "For each speech, we will perform some familiar tasks:\n",
    "\n",
    "  * We will **.split()** the speech into words\n",
    "  * we will use **replace** to get rid of punctuation\n",
    "  * we will use **wn.morphy()** to get the lemma of each word\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only problem with tabular data is that we have to run splitting, clearing punctuation, stopwording, and other actions on entire **columns** of lists of data rather than just lists.\n",
    "\n",
    "In theory, you might imagine writing a loop like this to deal with each cell at a time.   However, that would take FOREVER.  \n",
    "\n",
    "A more efficient approach is to work with the built-in commands that Pandas takes which work over all the cells in an entire column.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The pandas-native commands for working on columns in tabular data have familiar names:\n",
    "\n",
    "    .str.replace()\n",
    "    .str.lower()\n",
    "    .str.split()\n",
    "    \n",
    "Let's see them in action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get rid of punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data['speech'] = eighties_data['speech'].str.replace('[^\\w\\s]','')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lowercase the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data['speech'] = eighties_data['speech'].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eliminate stopwords using .replace() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This may take a minute*** -- notice the [*] in light gray to the left of the line of code. This means, 'the computer is thinking; please wait.' If your computer repeatedly crashes, you may need to allocate more memory when you next call up a session of JupyterLab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data['stopworded'] = eighties_data['speech'].str.replace(stopwords_regex, '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split each speech into a list of individual words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This may take a minute***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data['words'] = eighties_data['stopworded'].str.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2329890</td>\n",
       "      <td>mr speaker we in delaware are proud of the out...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. EVANS of Delaware</td>\n",
       "      <td>122</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker   delaware  proud   outstanding rec...</td>\n",
       "      <td>[mr, speaker, delaware, proud, outstanding, re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2329891</td>\n",
       "      <td>mr speaker it is logical for americans to be u...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. DERWINSKI</td>\n",
       "      <td>82</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker   logical  americans   upset   hold...</td>\n",
       "      <td>[mr, speaker, logical, americans, upset, holdi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2329892</td>\n",
       "      <td>the chair has examined the journal of the last...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>32</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>chair  examined  journal   last days proceedi...</td>\n",
       "      <td>[chair, examined, journal, last, days, proceed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2329893</td>\n",
       "      <td>mr speaker i ask unanimous consent that the co...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. WHITE</td>\n",
       "      <td>36</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker  ask unanimous consent   committee ...</td>\n",
       "      <td>[mr, speaker, ask, unanimous, consent, committ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2329894</td>\n",
       "      <td>is there objection to the request of the gentl...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>objection   request   gentleman  texas</td>\n",
       "      <td>[objection, request, gentleman, texas]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0.1                                             speech  \\\n",
       "0       2329890  mr speaker we in delaware are proud of the out...   \n",
       "1       2329891  mr speaker it is logical for americans to be u...   \n",
       "2       2329892  the chair has examined the journal of the last...   \n",
       "3       2329893  mr speaker i ask unanimous consent that the co...   \n",
       "4       2329894  is there objection to the request of the gentl...   \n",
       "\n",
       "         date                  speaker  word_count  year  month  month_year  \\\n",
       "0  1980-01-03    Mr. EVANS of Delaware         122  1980      1  1980-01-01   \n",
       "1  1980-01-03            Mr. DERWINSKI          82  1980      1  1980-01-01   \n",
       "2  1980-01-03  The SPEAKER pro tempore          32  1980      1  1980-01-01   \n",
       "3  1980-01-03                Mr. WHITE          36  1980      1  1980-01-01   \n",
       "4  1980-01-03  The SPEAKER pro tempore          11  1980      1  1980-01-01   \n",
       "\n",
       "                                          stopworded  \\\n",
       "0  mr speaker   delaware  proud   outstanding rec...   \n",
       "1  mr speaker   logical  americans   upset   hold...   \n",
       "2   chair  examined  journal   last days proceedi...   \n",
       "3  mr speaker  ask unanimous consent   committee ...   \n",
       "4             objection   request   gentleman  texas   \n",
       "\n",
       "                                               words  \n",
       "0  [mr, speaker, delaware, proud, outstanding, re...  \n",
       "1  [mr, speaker, logical, americans, upset, holdi...  \n",
       "2  [chair, examined, journal, last, days, proceed...  \n",
       "3  [mr, speaker, ask, unanimous, consent, committ...  \n",
       "4             [objection, request, gentleman, texas]  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eighties_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that with str.split() we have now changed the kind of data in the 'speech' column.  Formerly, we had one long string of text, to which we could apply commands such as .replace()) and .lower().  Now, we have a list of words in each row of 'speech.' This is useful for counting -- which we'll do next -- but it makes using .replace() more difficult.  \n",
    "\n",
    "***Bottom line***: when working with tabular data, use commands like .replace() before you .split() the strings of text into individual words. \n",
    "\n",
    "(NB: You can always use ' '.join(list) to weld those lists of words back together if you have to.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wordcount with Tabular Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use many of the tools we already know to count words.\n",
    "\n",
    "     value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You need lists of words in a column to count them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important observation: in the stopwording loop above, we just changed the data type in which the words are stored. \n",
    "\n",
    "Originally, our 'speech' column was just long strings of words.  In order to stopword those strings, we .split() each speech into a list of individual words -- just like the lists we've been working on so far. Those lists are easy to stopword.\n",
    "\n",
    "We could have glued the words back together into super-long strings again. But in fact, it's useful to keep the words in list form, because lists are easy to count.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How many words in any speech?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How long is the first speech, in words (not including stopwords)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(eighties_data['words'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the top words in the first speech (not including stopwords)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "railroad       3\n",
       "wilmington     3\n",
       "outstanding    3\n",
       "shops          3\n",
       "pleasure       2\n",
       "recognize      2\n",
       "mutual         2\n",
       "amtrak         2\n",
       "achieved       2\n",
       "mr             2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series.value_counts(list(eighties_data['words'][0]))[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice what happens when I set the parameter \"normalize\" for value_counts() as \"True\": (it tells Python to tell us the percentage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "railroad       0.041096\n",
       "wilmington     0.041096\n",
       "outstanding    0.041096\n",
       "shops          0.041096\n",
       "pleasure       0.027397\n",
       "recognize      0.027397\n",
       "mutual         0.027397\n",
       "amtrak         0.027397\n",
       "achieved       0.027397\n",
       "mr             0.027397\n",
       "dtype: float64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series.value_counts(list(eighties_data['words'][0]), normalize=True)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Total word count for the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the words for each speech in the dataset. We'll make a new column called 'wordcount.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "eighties_data['wordcount'] = eighties_data['words'].str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "      <th>wordcount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2329890</td>\n",
       "      <td>mr speaker we in delaware are proud of the out...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. EVANS of Delaware</td>\n",
       "      <td>122</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker   delaware  proud   outstanding rec...</td>\n",
       "      <td>[mr, speaker, delaware, proud, outstanding, re...</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2329891</td>\n",
       "      <td>mr speaker it is logical for americans to be u...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. DERWINSKI</td>\n",
       "      <td>82</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker   logical  americans   upset   hold...</td>\n",
       "      <td>[mr, speaker, logical, americans, upset, holdi...</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2329892</td>\n",
       "      <td>the chair has examined the journal of the last...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>32</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>chair  examined  journal   last days proceedi...</td>\n",
       "      <td>[chair, examined, journal, last, days, proceed...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2329893</td>\n",
       "      <td>mr speaker i ask unanimous consent that the co...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. WHITE</td>\n",
       "      <td>36</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>mr speaker  ask unanimous consent   committee ...</td>\n",
       "      <td>[mr, speaker, ask, unanimous, consent, committ...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2329894</td>\n",
       "      <td>is there objection to the request of the gentl...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>objection   request   gentleman  texas</td>\n",
       "      <td>[objection, request, gentleman, texas]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0.1                                             speech  \\\n",
       "0       2329890  mr speaker we in delaware are proud of the out...   \n",
       "1       2329891  mr speaker it is logical for americans to be u...   \n",
       "2       2329892  the chair has examined the journal of the last...   \n",
       "3       2329893  mr speaker i ask unanimous consent that the co...   \n",
       "4       2329894  is there objection to the request of the gentl...   \n",
       "\n",
       "         date                  speaker  word_count  year  month  month_year  \\\n",
       "0  1980-01-03    Mr. EVANS of Delaware         122  1980      1  1980-01-01   \n",
       "1  1980-01-03            Mr. DERWINSKI          82  1980      1  1980-01-01   \n",
       "2  1980-01-03  The SPEAKER pro tempore          32  1980      1  1980-01-01   \n",
       "3  1980-01-03                Mr. WHITE          36  1980      1  1980-01-01   \n",
       "4  1980-01-03  The SPEAKER pro tempore          11  1980      1  1980-01-01   \n",
       "\n",
       "                                          stopworded  \\\n",
       "0  mr speaker   delaware  proud   outstanding rec...   \n",
       "1  mr speaker   logical  americans   upset   hold...   \n",
       "2   chair  examined  journal   last days proceedi...   \n",
       "3  mr speaker  ask unanimous consent   committee ...   \n",
       "4             objection   request   gentleman  texas   \n",
       "\n",
       "                                               words  wordcount  \n",
       "0  [mr, speaker, delaware, proud, outstanding, re...         73  \n",
       "1  [mr, speaker, logical, americans, upset, holdi...         39  \n",
       "2  [chair, examined, journal, last, days, proceed...         19  \n",
       "3  [mr, speaker, ask, unanimous, consent, committ...         19  \n",
       "4             [objection, request, gentleman, texas]          4  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eighties_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we now have two wordcount columns -- one we made before stopwording and one we made after stopwording."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many words are there in the dataframe as a whole? We can answer that question by adding up all the individual speech wordcounts using \n",
    "\n",
    "    .sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108044320"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eighties_data['wordcount'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the longest speeches in the datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the longest speeches in the database?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "      <th>wordcount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>241386</th>\n",
       "      <td>2571276</td>\n",
       "      <td>815 16th street nw washington dc 20006 d 6 390...</td>\n",
       "      <td>1981-09-09</td>\n",
       "      <td>es. AFL-CIO</td>\n",
       "      <td>33507</td>\n",
       "      <td>1981</td>\n",
       "      <td>9</td>\n",
       "      <td>1981-09-01</td>\n",
       "      <td>815 16th street nw  dc 20006  6 390649  c jame...</td>\n",
       "      <td>[815, 16th, street, nw, washington, dc, 20006,...</td>\n",
       "      <td>29740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1217582</th>\n",
       "      <td>3547472</td>\n",
       "      <td>limits would be in effect until such time as t...</td>\n",
       "      <td>1988-06-01</td>\n",
       "      <td>The MAAC</td>\n",
       "      <td>30042</td>\n",
       "      <td>1988</td>\n",
       "      <td>6</td>\n",
       "      <td>1988-06-01</td>\n",
       "      <td>limits    effect       implements  fee schedul...</td>\n",
       "      <td>[limits, would, effect, time, secretary, imple...</td>\n",
       "      <td>17962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197843</th>\n",
       "      <td>2527733</td>\n",
       "      <td>box 269 elizabethtown pa d 6 63064 a american ...</td>\n",
       "      <td>1981-05-18</td>\n",
       "      <td>sors. P.O</td>\n",
       "      <td>21558</td>\n",
       "      <td>1981</td>\n",
       "      <td>5</td>\n",
       "      <td>1981-05-01</td>\n",
       "      <td>box 269 elizabethtown pa  6 63064   associatio...</td>\n",
       "      <td>[box, 269, elizabethtown, pa, 6, 63064, americ...</td>\n",
       "      <td>19252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286393</th>\n",
       "      <td>2616283</td>\n",
       "      <td>box 269 elizabethtown pa d 6 8461 a american a...</td>\n",
       "      <td>1981-11-24</td>\n",
       "      <td>sors. P.O</td>\n",
       "      <td>21076</td>\n",
       "      <td>1981</td>\n",
       "      <td>11</td>\n",
       "      <td>1981-11-01</td>\n",
       "      <td>box 269 elizabethtown pa  6 8461   association...</td>\n",
       "      <td>[box, 269, elizabethtown, pa, 6, 8461, america...</td>\n",
       "      <td>18817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13423</th>\n",
       "      <td>2343313</td>\n",
       "      <td>de cv balderas 36 mexico df mexico d 6 2400 e ...</td>\n",
       "      <td>1980-02-19</td>\n",
       "      <td>car. S.A</td>\n",
       "      <td>19922</td>\n",
       "      <td>1980</td>\n",
       "      <td>2</td>\n",
       "      <td>1980-02-01</td>\n",
       "      <td>de cv balderas 36 mexico df mexico  6 2400 e 2...</td>\n",
       "      <td>[de, cv, balderas, 36, mexico, df, mexico, 6, ...</td>\n",
       "      <td>17793</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0.1                                             speech  \\\n",
       "241386        2571276  815 16th street nw washington dc 20006 d 6 390...   \n",
       "1217582       3547472  limits would be in effect until such time as t...   \n",
       "197843        2527733  box 269 elizabethtown pa d 6 63064 a american ...   \n",
       "286393        2616283  box 269 elizabethtown pa d 6 8461 a american a...   \n",
       "13423         2343313  de cv balderas 36 mexico df mexico d 6 2400 e ...   \n",
       "\n",
       "               date      speaker  word_count  year  month  month_year  \\\n",
       "241386   1981-09-09  es. AFL-CIO       33507  1981      9  1981-09-01   \n",
       "1217582  1988-06-01     The MAAC       30042  1988      6  1988-06-01   \n",
       "197843   1981-05-18    sors. P.O       21558  1981      5  1981-05-01   \n",
       "286393   1981-11-24    sors. P.O       21076  1981     11  1981-11-01   \n",
       "13423    1980-02-19     car. S.A       19922  1980      2  1980-02-01   \n",
       "\n",
       "                                                stopworded  \\\n",
       "241386   815 16th street nw  dc 20006  6 390649  c jame...   \n",
       "1217582  limits    effect       implements  fee schedul...   \n",
       "197843   box 269 elizabethtown pa  6 63064   associatio...   \n",
       "286393   box 269 elizabethtown pa  6 8461   association...   \n",
       "13423    de cv balderas 36 mexico df mexico  6 2400 e 2...   \n",
       "\n",
       "                                                     words  wordcount  \n",
       "241386   [815, 16th, street, nw, washington, dc, 20006,...      29740  \n",
       "1217582  [limits, would, effect, time, secretary, imple...      17962  \n",
       "197843   [box, 269, elizabethtown, pa, 6, 63064, americ...      19252  \n",
       "286393   [box, 269, elizabethtown, pa, 6, 8461, america...      18817  \n",
       "13423    [de, cv, balderas, 36, mexico, df, mexico, 6, ...      17793  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "longest_speeches = eighties_data.nlargest(n=5, columns=['word_count']) # Get the top 5 longest speeches by word_count\n",
    "longest_speeches "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The top words for the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the top words for the longest speeches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need a list with all the words in the 'words' column in it. A simple for-loop can do that in a hurry.  Let's create a list called \"all_words\" from the content of each speech in the \"speech\" column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['815',\n",
       " '16th',\n",
       " 'street',\n",
       " 'nw',\n",
       " 'washington',\n",
       " 'dc',\n",
       " '20006',\n",
       " '6',\n",
       " '390649',\n",
       " 'c',\n",
       " 'james',\n",
       " 'hacket',\n",
       " 'american',\n",
       " 'plywood',\n",
       " 'association',\n",
       " 'po',\n",
       " 'box',\n",
       " '11700',\n",
       " 'tacoma',\n",
       " 'wash',\n",
       " 'b',\n",
       " 'american',\n",
       " 'plywood',\n",
       " 'assciation',\n",
       " 'po',\n",
       " 'box',\n",
       " '11700',\n",
       " 'tacoma',\n",
       " 'wash',\n",
       " '98411']"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "longestspeecheswords = []\n",
    "\n",
    "for speech in longest_speeches['words']:\n",
    "    for word in speech:\n",
    "        longestspeecheswords.append(word)\n",
    "\n",
    "longestspeecheswords[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "washington    3969\n",
       "dc            3915\n",
       "nw            3548\n",
       "b             3363\n",
       "street        3228\n",
       "6             2203\n",
       "e             1778\n",
       "avenue        1609\n",
       "9             1519\n",
       "suite         1297\n",
       "dtype: int64"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series.value_counts(longestspeecheswords)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now here's a line of code that does exactly the same thing in another way.\n",
    "\n",
    "It uses the function \n",
    "\n",
    "    .explode()\n",
    "\n",
    "to give each word in the list its own row. We can use **.dropna()** at the end to tell pandas to drop any rows that are empty. Check it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "241386           815\n",
       "241386          16th\n",
       "241386        street\n",
       "241386            nw\n",
       "241386    washington\n",
       "             ...    \n",
       "13423           19th\n",
       "13423         street\n",
       "13423             nw\n",
       "13423     washington\n",
       "13423             dc\n",
       "Name: words, Length: 103564, dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "longestspeecheswords = longest_speeches[\"words\"].explode().dropna()\n",
    "longestspeecheswords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "washington    3969\n",
       "dc            3915\n",
       "nw            3548\n",
       "b             3363\n",
       "street        3228\n",
       "6             2203\n",
       "e             1778\n",
       "avenue        1609\n",
       "9             1519\n",
       "suite         1297\n",
       "Name: words, dtype: int64"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "longestspeecheswords.value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the top words overall.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "within            77036\n",
       "children          76906\n",
       "agreement         76869\n",
       "upon              75960\n",
       "case              75881\n",
       "motion            75698\n",
       "international     75691\n",
       "ordered           75636\n",
       "office            73858\n",
       "human             73439\n",
       "already           73174\n",
       "local             73046\n",
       "passed            72516\n",
       "special           72003\n",
       "appropriations    71823\n",
       "still             71683\n",
       "benefits          71621\n",
       "present           71580\n",
       "political         71314\n",
       "result            71215\n",
       "Name: stopworded, dtype: int64"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topwords1980"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mr             1262448\n",
       "would           922396\n",
       "president       755471\n",
       "bill            603331\n",
       "amendment       543091\n",
       "                ...   \n",
       "rzcosn               1\n",
       "ergens               1\n",
       "tradegattwe          1\n",
       "debitatng            1\n",
       "1985thats            1\n",
       "Name: words, Length: 815067, dtype: int64"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topwords1980 = eighties_data[\"words\"].explode().dropna().value_counts()\n",
    "topwords1980"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great -- but some of those words are still pretty hollow, despite having already been stopworded!  \n",
    "\n",
    "Let's use our top words from the decade to create a new stopword list, format the list, and apply it to eighties_data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mr',\n",
       " 'would',\n",
       " 'president',\n",
       " 'bill',\n",
       " 'amendment',\n",
       " 'us',\n",
       " 'senator',\n",
       " 'time',\n",
       " 'gentleman',\n",
       " 'committee',\n",
       " 'one',\n",
       " 'speaker',\n",
       " 'states',\n",
       " 'new',\n",
       " 'people',\n",
       " 'years',\n",
       " 'chairman',\n",
       " 'senate',\n",
       " 'house',\n",
       " 'year',\n",
       " 'congress',\n",
       " 'federal',\n",
       " 'program',\n",
       " 'think',\n",
       " 'many',\n",
       " 'state',\n",
       " 'united',\n",
       " 'legislation',\n",
       " 'support',\n",
       " 'also',\n",
       " 'act',\n",
       " 'government',\n",
       " 'may',\n",
       " 'yield',\n",
       " 'today',\n",
       " 'budget',\n",
       " 'national',\n",
       " 'american',\n",
       " 'percent',\n",
       " 'make',\n",
       " 'first',\n",
       " 'country',\n",
       " 'ask',\n",
       " 'million',\n",
       " 'could',\n",
       " 'like',\n",
       " 'going',\n",
       " 'colleagues',\n",
       " 'must',\n",
       " 'resolution']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "maybestopwords = list(pd.Series.value_counts(words1980)[:200].index)\n",
    "maybestopwords[:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ideally, we would edit this list by hand. But I'm just going to use the top 200 words as stopwords for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords_regex2 = r'\\b(?:{})\\b'.format('|'.join(maybestopwords))\n",
    "eighties_data['stopworded'] = eighties_data['stopworded'].str.replace(stopwords_regex2, '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "      <th>wordcount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2329890</td>\n",
       "      <td>mr speaker we in delaware are proud of the out...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. EVANS of Delaware</td>\n",
       "      <td>122</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>delaware  proud   outstanding    achieved ...</td>\n",
       "      <td>[mr, speaker, delaware, proud, outstanding, re...</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2329891</td>\n",
       "      <td>mr speaker it is logical for americans to be u...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. DERWINSKI</td>\n",
       "      <td>82</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>logical     upset   holding   hostages  ir...</td>\n",
       "      <td>[mr, speaker, logical, americans, upset, holdi...</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2329892</td>\n",
       "      <td>the chair has examined the journal of the last...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>32</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>chair  examined  journal    days proceedings ...</td>\n",
       "      <td>[chair, examined, journal, last, days, proceed...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2329893</td>\n",
       "      <td>mr speaker i ask unanimous consent that the co...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. WHITE</td>\n",
       "      <td>36</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>banking finance  urban affairs  disc...</td>\n",
       "      <td>[mr, speaker, ask, unanimous, consent, committ...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2329894</td>\n",
       "      <td>is there objection to the request of the gentl...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>texas</td>\n",
       "      <td>[objection, request, gentleman, texas]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0.1                                             speech  \\\n",
       "0       2329890  mr speaker we in delaware are proud of the out...   \n",
       "1       2329891  mr speaker it is logical for americans to be u...   \n",
       "2       2329892  the chair has examined the journal of the last...   \n",
       "3       2329893  mr speaker i ask unanimous consent that the co...   \n",
       "4       2329894  is there objection to the request of the gentl...   \n",
       "\n",
       "         date                  speaker  word_count  year  month  month_year  \\\n",
       "0  1980-01-03    Mr. EVANS of Delaware         122  1980      1  1980-01-01   \n",
       "1  1980-01-03            Mr. DERWINSKI          82  1980      1  1980-01-01   \n",
       "2  1980-01-03  The SPEAKER pro tempore          32  1980      1  1980-01-01   \n",
       "3  1980-01-03                Mr. WHITE          36  1980      1  1980-01-01   \n",
       "4  1980-01-03  The SPEAKER pro tempore          11  1980      1  1980-01-01   \n",
       "\n",
       "                                          stopworded  \\\n",
       "0      delaware  proud   outstanding    achieved ...   \n",
       "1      logical     upset   holding   hostages  ir...   \n",
       "2   chair  examined  journal    days proceedings ...   \n",
       "3            banking finance  urban affairs  disc...   \n",
       "4                                              texas   \n",
       "\n",
       "                                               words  wordcount  \n",
       "0  [mr, speaker, delaware, proud, outstanding, re...         73  \n",
       "1  [mr, speaker, logical, americans, upset, holdi...         39  \n",
       "2  [chair, examined, journal, last, days, proceed...         19  \n",
       "3  [mr, speaker, ask, unanimous, consent, committ...         19  \n",
       "4             [objection, request, gentleman, texas]          4  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eighties_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, count (no splitting required)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "within            77036\n",
       "children          76906\n",
       "agreement         76869\n",
       "upon              75960\n",
       "case              75881\n",
       "motion            75698\n",
       "international     75691\n",
       "ordered           75636\n",
       "office            73858\n",
       "human             73439\n",
       "already           73174\n",
       "local             73046\n",
       "passed            72516\n",
       "special           72003\n",
       "appropriations    71823\n",
       "still             71683\n",
       "benefits          71621\n",
       "present           71580\n",
       "political         71314\n",
       "result            71215\n",
       "Name: stopworded, dtype: int64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topwords1980 = eighties_data[\"stopworded\"].str.split().explode().dropna().value_counts()[:20]\n",
    "topwords1980"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the top speeches that have mention the word \"democracy\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's count the number of times that the word 'democracy' appears in the 1980s.  \n",
    "\n",
    "Notice the use of .str.count():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "      <th>wordcount</th>\n",
       "      <th>democracy_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2329890</td>\n",
       "      <td>mr speaker we in delaware are proud of the out...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. EVANS of Delaware</td>\n",
       "      <td>122</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>delaware  proud   outstanding    achieved ...</td>\n",
       "      <td>[mr, speaker, delaware, proud, outstanding, re...</td>\n",
       "      <td>73</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2329891</td>\n",
       "      <td>mr speaker it is logical for americans to be u...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. DERWINSKI</td>\n",
       "      <td>82</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>logical     upset   holding   hostages  ir...</td>\n",
       "      <td>[mr, speaker, logical, americans, upset, holdi...</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2329892</td>\n",
       "      <td>the chair has examined the journal of the last...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>32</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>chair  examined  journal    days proceedings ...</td>\n",
       "      <td>[chair, examined, journal, last, days, proceed...</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2329893</td>\n",
       "      <td>mr speaker i ask unanimous consent that the co...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>Mr. WHITE</td>\n",
       "      <td>36</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>banking finance  urban affairs  disc...</td>\n",
       "      <td>[mr, speaker, ask, unanimous, consent, committ...</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2329894</td>\n",
       "      <td>is there objection to the request of the gentl...</td>\n",
       "      <td>1980-01-03</td>\n",
       "      <td>The SPEAKER pro tempore</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-01</td>\n",
       "      <td>texas</td>\n",
       "      <td>[objection, request, gentleman, texas]</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0.1                                             speech  \\\n",
       "0       2329890  mr speaker we in delaware are proud of the out...   \n",
       "1       2329891  mr speaker it is logical for americans to be u...   \n",
       "2       2329892  the chair has examined the journal of the last...   \n",
       "3       2329893  mr speaker i ask unanimous consent that the co...   \n",
       "4       2329894  is there objection to the request of the gentl...   \n",
       "\n",
       "         date                  speaker  word_count  year  month  month_year  \\\n",
       "0  1980-01-03    Mr. EVANS of Delaware         122  1980      1  1980-01-01   \n",
       "1  1980-01-03            Mr. DERWINSKI          82  1980      1  1980-01-01   \n",
       "2  1980-01-03  The SPEAKER pro tempore          32  1980      1  1980-01-01   \n",
       "3  1980-01-03                Mr. WHITE          36  1980      1  1980-01-01   \n",
       "4  1980-01-03  The SPEAKER pro tempore          11  1980      1  1980-01-01   \n",
       "\n",
       "                                          stopworded  \\\n",
       "0      delaware  proud   outstanding    achieved ...   \n",
       "1      logical     upset   holding   hostages  ir...   \n",
       "2   chair  examined  journal    days proceedings ...   \n",
       "3            banking finance  urban affairs  disc...   \n",
       "4                                              texas   \n",
       "\n",
       "                                               words  wordcount  \\\n",
       "0  [mr, speaker, delaware, proud, outstanding, re...         73   \n",
       "1  [mr, speaker, logical, americans, upset, holdi...         39   \n",
       "2  [chair, examined, journal, last, days, proceed...         19   \n",
       "3  [mr, speaker, ask, unanimous, consent, committ...         19   \n",
       "4             [objection, request, gentleman, texas]          4   \n",
       "\n",
       "   democracy_count  \n",
       "0                0  \n",
       "1                0  \n",
       "2                0  \n",
       "3                0  \n",
       "4                0  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "democracy_speeches = eighties_data.copy()\n",
    "democracy_speeches['democracy_count'] = democracy_speeches['speech'].str.count('democracy')  # Create a new column for the count of the word democracydis\n",
    "democracy_speeches.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the speeches that mentioned democracy most frequently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>speech</th>\n",
       "      <th>date</th>\n",
       "      <th>speaker</th>\n",
       "      <th>word_count</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>month_year</th>\n",
       "      <th>stopworded</th>\n",
       "      <th>words</th>\n",
       "      <th>wordcount</th>\n",
       "      <th>democracy_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1252707</th>\n",
       "      <td>3582597</td>\n",
       "      <td>mr president in listening to the speeches toda...</td>\n",
       "      <td>1988-08-10</td>\n",
       "      <td>Mr. PACKWOOD</td>\n",
       "      <td>5661</td>\n",
       "      <td>1988</td>\n",
       "      <td>8</td>\n",
       "      <td>1988-08-01</td>\n",
       "      <td>listening   speeches   find       repeated ...</td>\n",
       "      <td>[mr, president, listening, speeches, today, fi...</td>\n",
       "      <td>2654</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1044031</th>\n",
       "      <td>3373921</td>\n",
       "      <td>mr speaker with its four core grantees the nat...</td>\n",
       "      <td>1987-04-09</td>\n",
       "      <td>Mr. CONIMS</td>\n",
       "      <td>1603</td>\n",
       "      <td>1987</td>\n",
       "      <td>4</td>\n",
       "      <td>1987-04-01</td>\n",
       "      <td>four core grantees   endowment  democracy ...</td>\n",
       "      <td>[mr, speaker, four, core, grantees, national, ...</td>\n",
       "      <td>840</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1383325</th>\n",
       "      <td>3713215</td>\n",
       "      <td>mr president benazir bhutto prime minister of ...</td>\n",
       "      <td>1989-10-13</td>\n",
       "      <td>Mr. KERRY</td>\n",
       "      <td>2162</td>\n",
       "      <td>1989</td>\n",
       "      <td>10</td>\n",
       "      <td>1989-10-01</td>\n",
       "      <td>benazir bhutto prime minister   islamic repu...</td>\n",
       "      <td>[mr, president, benazir, bhutto, prime, minist...</td>\n",
       "      <td>1226</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>867164</th>\n",
       "      <td>3197054</td>\n",
       "      <td>mr president on december 16 1983 president rea...</td>\n",
       "      <td>1985-12-06</td>\n",
       "      <td>Mr. HATCH</td>\n",
       "      <td>1862</td>\n",
       "      <td>1985</td>\n",
       "      <td>12</td>\n",
       "      <td>1985-12-01</td>\n",
       "      <td>december 16 1983  reagan speaking   ceremon...</td>\n",
       "      <td>[mr, president, december, 16, 1983, president,...</td>\n",
       "      <td>981</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1309407</th>\n",
       "      <td>3639297</td>\n",
       "      <td>mr president i wanted to say to my friend from...</td>\n",
       "      <td>1989-04-13</td>\n",
       "      <td>Mr. WALLOP</td>\n",
       "      <td>1213</td>\n",
       "      <td>1989</td>\n",
       "      <td>4</td>\n",
       "      <td>1989-04-01</td>\n",
       "      <td>wanted     friend  south carolina  feel    ...</td>\n",
       "      <td>[mr, president, wanted, say, friend, south, ca...</td>\n",
       "      <td>543</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0.1                                             speech  \\\n",
       "1252707       3582597  mr president in listening to the speeches toda...   \n",
       "1044031       3373921  mr speaker with its four core grantees the nat...   \n",
       "1383325       3713215  mr president benazir bhutto prime minister of ...   \n",
       "867164        3197054  mr president on december 16 1983 president rea...   \n",
       "1309407       3639297  mr president i wanted to say to my friend from...   \n",
       "\n",
       "               date       speaker  word_count  year  month  month_year  \\\n",
       "1252707  1988-08-10  Mr. PACKWOOD        5661  1988      8  1988-08-01   \n",
       "1044031  1987-04-09    Mr. CONIMS        1603  1987      4  1987-04-01   \n",
       "1383325  1989-10-13     Mr. KERRY        2162  1989     10  1989-10-01   \n",
       "867164   1985-12-06     Mr. HATCH        1862  1985     12  1985-12-01   \n",
       "1309407  1989-04-13    Mr. WALLOP        1213  1989      4  1989-04-01   \n",
       "\n",
       "                                                stopworded  \\\n",
       "1252707     listening   speeches   find       repeated ...   \n",
       "1044031      four core grantees   endowment  democracy ...   \n",
       "1383325    benazir bhutto prime minister   islamic repu...   \n",
       "867164      december 16 1983  reagan speaking   ceremon...   \n",
       "1309407     wanted     friend  south carolina  feel    ...   \n",
       "\n",
       "                                                     words  wordcount  \\\n",
       "1252707  [mr, president, listening, speeches, today, fi...       2654   \n",
       "1044031  [mr, speaker, four, core, grantees, national, ...        840   \n",
       "1383325  [mr, president, benazir, bhutto, prime, minist...       1226   \n",
       "867164   [mr, president, december, 16, 1983, president,...        981   \n",
       "1309407  [mr, president, wanted, say, friend, south, ca...        543   \n",
       "\n",
       "         democracy_count  \n",
       "1252707               27  \n",
       "1044031               26  \n",
       "1383325               24  \n",
       "867164                21  \n",
       "1309407               19  "
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_democracy_speeches = democracy_speeches.nlargest(500, ['democracy_count'])\n",
    "top_democracy_speeches.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What words do the top speeches mentioning democracy use when they talk about democracy?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "democracy        3933\n",
       "democratic       1767\n",
       "nicaragua        1674\n",
       "political        1338\n",
       "freedom          1235\n",
       "central          1168\n",
       "aid              1096\n",
       "contras           926\n",
       "sandinistas       918\n",
       "peace             878\n",
       "human             720\n",
       "salvador          685\n",
       "el                680\n",
       "free              656\n",
       "countries         603\n",
       "elections         592\n",
       "chile             590\n",
       "opposition        525\n",
       "south             519\n",
       "communist         517\n",
       "power             510\n",
       "forces            510\n",
       "nicaraguan        507\n",
       "endowment         487\n",
       "philippines       457\n",
       "election          451\n",
       "international     440\n",
       "sandinista        438\n",
       "leaders           410\n",
       "toward            399\n",
       "Name: stopworded, dtype: int64"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "democracywordcount = top_democracy_speeches[\"stopworded\"].str.split().explode().dropna().value_counts()\n",
    "democracywordcount[:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does that give you something you could write about?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summing Up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a reprise of the code you just learned.  \n",
    "\n",
    "In the code that follows, we will use **str.count()** and **.nlargest()** to find the speeches that mention \"pineapples\" the most.\n",
    "* We will use **.loc** and **.iloc()** to find the speakers who gave those pineapple speeches, calling that list, *pineapplespeakers.*\n",
    "* We will use square brackets -- **[]** -- and **.isin()** to find all the speeches given by speakers who are in the *pineapplespeakers* list, calling the result *pineapplespeakersspeeches.*\n",
    "* We will narrow the *pineapplespeakersspeeches* to just the 1960s using **.isin(),** creating the dataframe *pineapple_speakers_sixties_data.*\n",
    "* We will clean that data using **.str.replace()** and **.lower()**.\n",
    "* We will add a column, *democracy_count*, to *pineapple_speakers_sixties_data*.  The new coumn uses **str.count** to count how many times the speakers who defended the American pineapple mentioned democracy. We will use **.nlargest()** to find the speeches where they mentioned democracy the most, calling the results, *top_speeches.*\n",
    "* We will then divide 'stopworded' column of  *top_speeches* into words. We will count the stopworded words using **str.split().explode().dropna().value_counts()**, producing the final dataset, *pineapple_democracywordcount.*\n",
    "\n",
    "Effectively, this seemingly silly research project will give us an alternative portrait of how American representatives talked about democracy -- one centered not on anti-communism in the Reagan congress of the 1980s, with its fears about spreading communism in El Salvador, but coming from the diverse, multi-ethnic, Pacific island of Hawaii in the 1960s.  \n",
    "\n",
    "Such a project will allow us to answer a question like this: What did Hawaii's representatives (who sometimes defended its pineapple industry) talk about when they spoke about democracy in the 1960s? How did their view of American democracy differ from that of Reagan-era America in the fight against communism?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /users/jguldi/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import pandas as pd\n",
    "import csv\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download('stopwords')\n",
    "stop = stopwords.words('english')\n",
    "stopwords_regex = r'\\b(?:{})\\b'.format('|'.join(stop))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/group/history/hist_3368-jguldi\n"
     ]
    }
   ],
   "source": [
    "cd /scratch/group/history/hist_3368-jguldi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "hawaii         64\n",
       "congress       40\n",
       "territory      37\n",
       "state          36\n",
       "statehood      34\n",
       "united         31\n",
       "states         30\n",
       "pacific        29\n",
       "islands        27\n",
       "people         27\n",
       "education      23\n",
       "world          23\n",
       "american       21\n",
       "us             21\n",
       "island         20\n",
       "one            20\n",
       "nation         19\n",
       "years          19\n",
       "status         18\n",
       "house          18\n",
       "would          17\n",
       "trust          17\n",
       "government     17\n",
       "democracy      16\n",
       "new            15\n",
       "first          15\n",
       "territories    15\n",
       "vote           14\n",
       "hawaiis        13\n",
       "citizens       13\n",
       "Name: stopworded, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load congress data\n",
    "congress = pd.read_csv(\"congress1967-2010.csv\")\n",
    "\n",
    "# get the speeches that mentioned pineapples the most\n",
    "pineapplespeeches = congress['speech'].str.count('pineapple').nlargest(5) \n",
    "\n",
    "# get the names of speakers who gave the most pineapple-y speeches \n",
    "pineapplespeakers = []\n",
    "for speechnumber in list(pineapplespeeches.index):\n",
    "        speaker = list(congress.loc[:, ['speaker']].iloc[speechnumber, ])[0]\n",
    "        if speaker not in pineapplespeakers:\n",
    "            pineapplespeakers.append(speaker)\n",
    "        \n",
    "# get all speeches of speakers who mention pineapples the most\n",
    "pineapplespeakersspeeches = congress[congress['speaker'].isin(pineapplespeakers)]\n",
    "\n",
    "# narrow to just the 1960s\n",
    "target_years = list(range(1960, 1969 + 1))  # List of the years 1980-1989\n",
    "pineapple_speakers_sixties_data = pineapplespeakersspeeches[pineapplespeakersspeeches['year'].isin(target_years)].copy().reset_index()  # filter our dataset to just this decade\n",
    "pineapple_speakers_sixties_data = pineapple_speakers_sixties_data.drop(['index', 'Unnamed: 0'], 1) #minor reformatting - drop extra columns\n",
    "\n",
    "# clean up the data\n",
    "pineapple_speakers_sixties_data['speech'] = pineapple_speakers_sixties_data['speech'].str.replace('[^\\w\\s]','').str.lower() # remove punctuation, lowercase\n",
    "pineapple_speakers_sixties_data['stopworded'] = pineapple_speakers_sixties_data['speech'].str.replace(stopwords_regex, '') #stopwording\n",
    "\n",
    "# count how many times the pineapple speaker mentioned democracy in each speech\n",
    "pineapple_speakers_sixties_data['democracy_count'] = pineapple_speakers_sixties_data['speech'].str.count('democracy')  # Create a new column for the count of the word democracydis\n",
    "top_speeches = pineapple_speakers_sixties_data.nlargest(10, ['democracy_count'])\n",
    "\n",
    "# count other words that the pineapple speakers used when they mentioned democracy\n",
    "pineapple_democracywordcount = top_speeches[\"stopworded\"].str.split().explode().dropna().value_counts()\n",
    "pineapple_democracywordcount[:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***One Interpretation:*** while 80's legislators were most concerned about contrasting American democracy with what they perceived as the threat of Marxist violence in Latin American, Hawaiian legislators in the 1960s had mostly been concerned with democracy as a reflection of pacific people and islanders, depicting democracy as a system closely bound up with the advantages of public education and citizenship.\n",
    "\n",
    "***Reflect on the Room for Other Interpretations.  What is fact here, and what is open to debate?*** *How might your interpretation differ from the interpretation I have given? Would you pay attention to the same words?  Would you interpret them the same way?  Would you measure the 'top words about democracy' in the same way? Would that produce different results?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Print out the first 500 words of the speech that mentions your favorite animal the highest number of times.  \n",
    "\n",
    "\n",
    "2) Choose one of the top 100 speakers from congress['speaker'].value_counts().\n",
    "   * Find the longest speech by that speaker. \n",
    "   * Show the first and last 500 words.  \n",
    "\n",
    "\n",
    "3) Limit the dataframe to one year of your choosing in the 1980s.  \n",
    "   * Find the longest speech by your speaker. \n",
    "   * Show the first 500 words.  \n",
    "\n",
    "\n",
    "4) Call up a list of your speaker's top hundred words.  \n",
    "   * Choose one word that you judge to be meaningful -- and possibly distinctive of that speaker (consult the list of overall top words for comparison).  \n",
    "   * Find the speech where your speaker mentions that word the greatest number of times. \n",
    "   * Show the first 500 words.  \n",
    "\n",
    "\n",
    "5) Find longest speech where your speaker mentions the word of your choosing.  \n",
    "   * The word should be mentioned at least three times. \n",
    "   *  Show the first 500 words.  \n",
    "\n",
    "For each part of the assignment, take a screenshot of the code and the results and upload it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10 (main, Feb 25 2022, 18:07:24) \n[Clang 13.0.0 (clang-1300.0.29.30)]"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}