finding-significant-terms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/CC_BY.png\"><br />\n",
    "\n",
    "Created by [Nathan Kelber](http://nkelber.com) and Ted Lawless for [JSTOR Labs](https://labs.jstor.org/) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)<br />\n",
    "For questions/comments/improvements, email nathan.kelber@ithaka.org.<br />\n",
    "___\n",
    "\n",
    "# Finding Significant Words Using TF/IDF\n",
    "\n",
    "**Description:**\n",
    "This [notebook](https://docs.constellate.org/key-terms/#jupyter-notebook) shows how to discover significant words. The method for finding significant terms is [tf-idf](https://docs.constellate.org/key-terms/#tf-idf).  The following processes are described:\n",
    "\n",
    "* An educational overview of TF-IDF, including how it is calculated\n",
    "* Using the `constellate` client to retrieve a dataset\n",
    "* Filtering based on a pre-processed ID list\n",
    "* Cleaning the tokens in the dataset\n",
    "* Creating a [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary)\n",
    "* Creating a [gensim](https://docs.constellate.org/key-terms/#gensim) [bag of words](https://docs.constellate.org/key-terms/#bag-of-words) [corpus](https://docs.constellate.org/key-terms/#corpus)\n",
    "* Computing the most significant words in your [corpus](https://docs.constellate.org/key-terms/#corpus) using [gensim](https://docs.constellate.org/key-terms/#gensim) implementation of [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf)\n",
    "\n",
    "**Use Case:** For Learners (Detailed explanation, not ideal for researchers)\n",
    "\n",
    "[Take me to the **Research Version** of this notebook ->](./finding-significant-terms-for-research.ipynb)\n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion time:** 60 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "* Python Basics Series ([Start Python Basics I](./python-basics-1.ipynb))\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "* [Exploring Metadata](./metadata.ipynb)\n",
    "* [Working with Dataset Files](./working-with-dataset-files.ipynb)\n",
    "* [Pandas I](./pandas-1.ipynb)\n",
    "* A familiarity with [gensim](https://docs.constellate.org/key-terms/#gensim) is helpful but not required.\n",
    "\n",
    "**Data Format:** [JSON Lines (.jsonl)](https://docs.constellate.org/key-terms/#jsonl)\n",
    "\n",
    "**Libraries Used:**\n",
    "* [constellate](https://docs.constellate.org/key-terms/#tdm-client) client to collect, unzip, and read our dataset\n",
    "* [pandas](https://constellate.org/docs/key-terms/#pandas) to load a preprocessing list\n",
    "* [gensim](https://docs.constellate.org/key-terms/#gensim) to help compute the [tf-idf](https://docs.constellate.org/key-terms/#tf-idf) calculations\n",
    "* [NLTK](https://docs.constellate.org/key-terms/#nltk) to create a stopwords list (if no list is supplied)\n",
    "\n",
    "**Research Pipeline:**\n",
    "\n",
    "1. Build a dataset\n",
    "2. Create a \"Pre-Processing CSV\" with [Exploring Metadata](./exploring-metadata.ipynb) (Optional)\n",
    "3. Complete the TF-IDF analysis with this notebook\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is \"Term Frequency- Inverse Document Frequency\" (TF-IDF)?\n",
    "\n",
    "[TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) is used in [machine learning](https://docs.constellate.org/key-terms/#machine-learning) and [natural language processing](https://docs.constellate.org/key-terms//#nlp) for measuring the significance of particular terms for a given document. It consists of two parts that are multiplied together:\n",
    "\n",
    "1. Term Frequency- A measure of how many times a given word appears in a document\n",
    "2. Inverse Document Frequency- A measure of how many times the same word occurs in other documents within the corpus\n",
    "\n",
    "**Before starting this lesson,** we recommend reading the [explanation of TF/IDF](https://ghost.constellate.org/what-is-tf-idf/) in the Constellate documentation.\n",
    "\n",
    "\n",
    "### TF-IDF Calculation in Plain English\n",
    "\n",
    "$$(Times-the-word-occurs-in-given-document) \\cdot \\mbox{log} \\frac{(Total-number-of-documents)}{(Number-of-documents-containing-word)}$$\n",
    "\n",
    "There are variations on the [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) formula, but this is the most widely-used version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing TF-IDF with your Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the `constellate` client to automatically retrieve the dataset in the JSON file format. \n",
    "\n",
    "Enter a [dataset ID](https://docs.constellate.org/key-terms/#dataset-ID) in the next code cell.\n",
    "\n",
    "If you don't have a dataset ID, you can:\n",
    "* Use the sample dataset ID already in the code cell\n",
    "* [Create a new dataset](https://constellate.org/builder)\n",
    "* [Use a dataset ID from other pre-built sample datasets](https://constellate.org/dataset/dashboard)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default dataset is \"Shakespeare Quarterly,\" 1950-present\n",
    "dataset_id = \"7e41317e-740f-e86a-4729-20dab492e925\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, import the `constellate` client, passing the `dataset_id` as an argument using the `get_dataset` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing your dataset with a dataset ID\n",
    "import constellate\n",
    "# Pull in the sampled dataset (1500 documents) that matches `dataset_id`\n",
    "# in the form of a gzipped JSON lines file.\n",
    "# The .get_dataset() method downloads the gzipped JSONL file\n",
    "# to the /data folder and returns a string for the file name and location\n",
    "dataset_file = constellate.get_dataset(dataset_id)\n",
    "\n",
    "# To download the full dataset (up to a limit of 25,000 documents),\n",
    "# request it first in the builder environment. See the Constellate Client\n",
    "# documentation at: https://constellate.org/docs/constellate-client\n",
    "# Then use the `constellate.download` method show below.\n",
    "#dataset_file = constellate.download(dataset_id, 'jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply Pre-Processing Filters (if available)\n",
    "If you completed pre-processing with the \"Exploring Metadata and Pre-processing\" notebook, you can use your CSV file of dataset IDs to automatically filter the dataset. Your pre-processed CSV file  must be in the root folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a pre-processed CSV file of filtered dataset IDs.\n",
    "# If you do not have a pre-processed CSV file, the analysis\n",
    "# will run on the full dataset and may take longer to complete.\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "pre_processed_file_name = f'data/pre-processed_{dataset_id}.csv'\n",
    "\n",
    "if os.path.exists(pre_processed_file_name):\n",
    "    df = pd.read_csv(pre_processed_file_name)\n",
    "    filtered_id_list = df[\"id\"].tolist()\n",
    "    use_filtered_list = True\n",
    "    print('Pre-Processed CSV found. Successfully read in ' + str(len(df)) + ' documents.')\n",
    "else: \n",
    "    use_filtered_list = False\n",
    "    print('No pre-processed CSV file found. Full dataset will be used.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a Unigram Processing Function\n",
    "In this step, we gather the unigrams. If there is a Pre-Processing Filter, we will only analyze documents from the filtered ID list. We will also process each unigram, assessing them individually. We will complete the following tasks:\n",
    "\n",
    "* Lowercase all tokens\n",
    "* Remove tokens in stopwords list\n",
    "* Remove tokens with fewer than 4 characters\n",
    "* Remove tokens with non-alphabetic characters\n",
    "\n",
    "We can define this process in a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function that will process individual tokens\n",
    "# Only a token that passes through all three `if` \n",
    "# statements will be returned. A `True` result for\n",
    "# any `if` statement does not return the token. \n",
    "\n",
    "def process_token(token):\n",
    "    token = token.lower()\n",
    "    if len(token) < 4: # If True, do not return token\n",
    "        return None\n",
    "    if not(token.isalpha()): # If True, do not return token\n",
    "        return None\n",
    "    return token # If all are False, return the lowercased token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect lists of Document IDs, Titles, and Unigrams\n",
    "\n",
    "Next, we process all the unigrams into a list called `documents`. For demonstration purposes, this code runs on a limit of 500 documents, but we can change this to process all the documents. We are also collecting the document titles and ids so we can reference them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [] # A list that will contain all of our unigrams\n",
    "document_ids = [] # A list that will contain all of our document ids\n",
    "document_titles = [] # A list that will contain all of our titles\n",
    "\n",
    "for document in constellate.dataset_reader(dataset_file):\n",
    "    processed_document = [] # Temporarily store the unigrams for this document\n",
    "    document_id = document['id'] # Temporarily store the document id for this document\n",
    "    document_title = document['title'] # Temporarily store the document title for this document\n",
    "    if use_filtered_list is True:\n",
    "        # Skip documents not in our filtered_id_list\n",
    "        if document_id not in filtered_id_list:\n",
    "            continue\n",
    "    unigrams = document.get(\"unigramCount\", [])\n",
    "    for gram, count in unigrams.items():\n",
    "        clean_gram = process_token(gram)\n",
    "        if clean_gram is None:\n",
    "            continue\n",
    "        processed_document += [clean_gram] * count # Add the unigram as many times as it was counted\n",
    "    if len(processed_document) > 0:\n",
    "        document_ids.append(document_id)\n",
    "        document_titles.append(document_title)\n",
    "        documents.append(processed_document)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we have unigrams collected for all our documents insde the `documents` list variable. Each index of our list is a single document, starting with `documents[0]`. Each document is, in turn, a list with a single stringe for each unigram.\n",
    "\n",
    "**Note:** As we collect the unigrams for each document, we are simply including them in a list of strings. This is not the same as collecting them into word counts, and we are not using a Counter() object here like the Word Frequencies notebook. \n",
    "\n",
    "The next cell demonstrates the contents of each item in our `document` list. Essentially, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Show the unigrams collected for a particular document\n",
    "# Change the value of n to see a different document\n",
    "n = 0\n",
    "\n",
    "print(document_titles[n])\n",
    "list(documents[n])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to see word frequencies, we could convert the lists at this point into `Counter()` objects. The next cell demonstrates that operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Convert a given document into a Counter object to determine\n",
    "# word frequencies count\n",
    "\n",
    "# Import counter to help count word frequencies\n",
    "from collections import Counter\n",
    "\n",
    "word_freq = Counter(documents[0]) # Change documents index to see a different document\n",
    "word_freq.most_common(25) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all the cleaned unigrams for every document in a list called `documents`, we can use Gensim to compute TF/IDF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Using Gensim to Compute \"Term Frequency- Inverse Document Frequency\"\n",
    "\n",
    "It will be helpful to remember the basic steps we did in the explanatory [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) example:\n",
    "\n",
    "1. Create a list of the frequency of every word in every document\n",
    "2. Create a list of every word in the [corpus](https://docs.constellate.org/key-terms/#corpus)\n",
    "3. Compute [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) based on that data\n",
    "\n",
    "So far, we have completed the first item by creating a list of the frequency of every word in every document. Now we need to create a list of every word in the corpus. In [gensim](https://docs.constellate.org/key-terms/#gensim), this is called a \"dictionary\". A [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) is similar to a [Python dictionary](https://docs.constellate.org/key-terms/#python-dictionary), but here it is called a [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) to show it is a specialized kind of dictionary.\n",
    "\n",
    "### Creating a Gensim Dictionary\n",
    "\n",
    "Let's create our [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary). A [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) is a kind of masterlist of all the words across all the documents in our corpus. Each unique word is assigned an ID in the gensim dictionary. The result is a set of key/value pairs of unique tokens and their unique IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "dictionary = gensim.corpora.Dictionary(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) stores a unique identifier (starting with 0) for every unique token in the corpus. The [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) does not contain information on word frequencies; it only catalogs all the unique words in the corpus. You can see the unique ID for each token in the text using the .token2id() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(dictionary.token2id.items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also look up the corresponding ID for a token using the ``.get`` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the value for the key 'people'. Return 0 if there is no token matching 'people'. \n",
    "# The number returned is the gensim dictionary ID for the token. \n",
    "\n",
    "dictionary.token2id.get('people', 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of example, we could also discover a particular token using just the ID number. This is not something likely to happen in practice, but it serves here as a demonstration of the connection between tokens and their ID number.\n",
    "\n",
    "Normally, [Python dictionaries](https://docs.constellate.org/key-terms/#python-dictionary) only map from keys to values (not from values to keys). However, we can write a quick for loop to go the other direction. This cell is simply to demonstrate how the [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) is connected to the list entries in the [gensim](https://docs.constellate.org/key-terms/#gensim) ``bow_corpus``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the token associated with a token id number\n",
    "token_id = 100\n",
    "\n",
    "# If the token id matches, print out the associated token\n",
    "for dict_id, token in dictionary.items():\n",
    "    if dict_id == token_id:\n",
    "        print(token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Bag of Words Corpus\n",
    "\n",
    "The next step is to connect our word frequency data found within ``documents`` to our [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) token IDs. For every document, we want to know how many times a word (notated by its ID) occurs. We will create a [Python list](https://docs.constellate.org/key-terms/#python-list) called ``bow_corpus`` that will turn our word counts into a series of [tuples](https://docs.constellate.org/key-terms/#tuple) where the first number is the [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) token ID and the second number is the word frequency.\n",
    "\n",
    "![Combining Gensim dictionary with documents list to create Bag of Words Corpus](https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/bag-of-words-creation.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a bag of words corpus\n",
    "bow_corpus = []\n",
    "\n",
    "for document in documents:\n",
    "    bow_corpus.append(dictionary.doc2bow(document))\n",
    "\n",
    "print('Bag of words corpus created successfully.')\n",
    "\n",
    "# The for loop could also be written as a list comprehension\n",
    "# bow_corpus = [dictionary.doc2bow(document) for document in documents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Examine the bag of words corpus for a specific document n\n",
    "# Change the value of n to see another document\n",
    "n = 100\n",
    "\n",
    "list(bow_corpus[n][:25]) # List out a slice of the first 25 items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using IDs can seem a little abstract, but we can discover the word associated with a particular ID. For demonstration purposes, the following code prints the associated token with the token counts. Each line printed below matches the token id and and count from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each id and count in the bag of words corpus\n",
    "# Print the corresponding word from the Gensim dictionary and count\n",
    "for id, count in bow_corpus[n]:\n",
    "    print(dictionary[id].ljust(15), count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the `TfidfModel`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to create the [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) model which will set the parameters for our implementation of [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf). In our [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) example, the formula for [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) was:\n",
    "\n",
    "$$(Times-the-word-occurs-in-given-document) \\cdot \\mbox{log} \\frac{(Total-number-of-documents)}{(Number-of-documents-containing-word)}$$\n",
    "\n",
    "In [gensim](https://docs.constellate.org/key-terms/#gensim), the default formula for measuring [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) uses log base 2 instead of log base 10, as shown:\n",
    "\n",
    "$$(Times-the-word-occurs-in-given-document) \\cdot \\log_{2} \\frac{(Total-number-of-documents)}{(Number-of-documents-containing-the-word)}$$\n",
    "\n",
    "If you would like to use a different formula for your [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) calculation, there is a description of [parameters you can pass](https://radimrehurek.com/gensim/models/tfidfmodel.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create our gensim TF-IDF model\n",
    "model = gensim.models.TfidfModel(bow_corpus) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we apply our model to the ``bow_corpus`` to create our results in ``corpus_tfidf``. The ``corpus_tfidf`` is a python list of each document similar to ``bow_document``. Instead of listing the frequency next to the [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) ID, however, it contains the [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) score for the associated token. Below, we display the first document in ``corpus_tfidf``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create TF-IDF scores for the ``bow_corpus`` using our model\n",
    "corpus_tfidf = model[bow_corpus]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List out the TF-IDF scores for the nth document's first 10 tokens\n",
    "# Change n to change the document\n",
    "n = 0\n",
    "\n",
    "list(corpus_tfidf[n][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display the tokens instead of the [gensim dictionary](https://docs.constellate.org/key-terms/#gensim-dictionary) IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for id, score in corpus_tfidf[n][:10]:\n",
    "    print(dictionary[id].ljust(20), score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find Top Terms in a Single Document\n",
    "Finally, let's sort the terms by their [TF-IDF](https://docs.constellate.org/key-terms/#tf-idf) weights to find the most significant terms in the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Sort the tuples in our tf-idf scores list\n",
    "\n",
    "# Choosing a document by its index number\n",
    "# Change n to see a different document\n",
    "n = 0\n",
    "\n",
    "def Sort(tfidf_tuples):\n",
    "    \"This sorts based on the second value in our tuple, the tf-idf score\"\n",
    "    tfidf_tuples.sort(key = lambda x: x[1], reverse=True)\n",
    "    return tfidf_tuples \n",
    "\n",
    "# Print the document id and title\n",
    "print('Title: ', document_titles[n])\n",
    "print('ID: ', document_ids[n])\n",
    "print('----------------------------------------')\n",
    "\n",
    "# List the top twenty tokens in our example document by their TF-IDF scores\n",
    "# First we sort the tokens with their scores\n",
    "most_significant_terms = Sort(corpus_tfidf[n])[:20]\n",
    "\n",
    "# Next we print the list, replacing the token ids with the tokens\n",
    "for id, score in most_significant_terms:\n",
    "    print(dictionary[id].ljust(20), score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also analyze across the entire corpus to find the most unique terms. These are terms that appear in a particular text, but rarely or never appear in other texts. (Often, these will be proper names since a particular article may mention a name often but the name may rarely appear in other articles. There's also a fairly good chance these will be typos or errors in optical character recognition.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = {}\n",
    "for document in corpus_tfidf:\n",
    "    for token_id, score in document:\n",
    "        current_score = td.get(dictionary.get(token_id), 0)\n",
    "        if current_score < score:\n",
    "            td.update([(dictionary.get(token_id), score)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the items of ``td`` into a new variable ``sorted_td``\n",
    "# the ``reverse`` starts from highest to lowest\n",
    "sorted_td = sorted(td.items(), key=lambda kv: kv[1], reverse=True) \n",
    "\n",
    "for term, weight in sorted_td[:25]: # Print the top 25 terms in the entire corpus\n",
    "    print(term, weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display Most Significant Term for each Document\n",
    "We can see the most significant term in every document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# For each document, print the ID, most significant/unique word, and TF/IDF score\n",
    "\n",
    "n = 0\n",
    "\n",
    "for n, doc in enumerate(corpus_tfidf):\n",
    "    if len(doc) < 1:\n",
    "        continue\n",
    "    word_id, score = max(doc, key=lambda x: x[1])\n",
    "    print(document_ids[n], dictionary.get(word_id), score)\n",
    "    if n >= 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ranking documents by TF-IDF Score for a Search Word\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set a limit on the number of documents analyzed\n",
    "limit = 500\n",
    "\n",
    "from collections import defaultdict\n",
    "terms_to_docs = defaultdict(list)\n",
    "for doc_id, doc in enumerate(corpus_tfidf):\n",
    "    for term_id, value in doc:\n",
    "        term = dictionary.get(term_id)\n",
    "        terms_to_docs[term].append((doc_id, value))\n",
    "    if doc_id >= limit:\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick a unigram to discover its score across documents\n",
    "search_term = 'coriolanus'\n",
    "\n",
    "# Display a list of documents and scores for the search term\n",
    "\n",
    "matching = terms_to_docs.get(search_term)\n",
    "\n",
    "try: \n",
    "    for doc_id, score in sorted(matching, key=lambda x: x[1], reverse=True):\n",
    "        print(document_ids[doc_id], score)\n",
    "except:\n",
    "    print('Search term not found. Change the term or expand the corpus size.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}