model-visualize.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python38064bittrlnworkshop2020virtualenv3d7570ac459b45329af28755a1ebad02",
   "display_name": "Python 3.8.0 64-bit ('TRLN-Workshop-2020': virtualenv)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Custom Discovery for Digitized Collections Using Computational Methods\n",
    "\n",
    "[Scott Bailey](mailto:csbaile3@ncsu.edu)\n",
    "\n",
    "TRLN Annual Meeting 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Goals\n",
    "\n",
    "Based on the pre-workshop survey, there are a lot of different learning goals that people have coming to this workshop. Some folks want to really see how the code works and expand their knowledge of Python and visualization. Others want to see what's possible with discovery and machine learning techniques. My goal in leading this workshop today is to take you through a workflow for building a custom discovery visualization for a digitized collection. Throughout, I'll talk through the code and the libraries we're using, offer ideas on different types of models and machine learning techniques, and try to point out moments where we should reflect critically on machine learning methods and libraries. My hope is that working through this example together will create opportunities and possibilities for you, in line with and alongside the goals you bring.\n",
    "\n",
    "## Questions during the workshop\n",
    "\n",
    "During the workshop, please do ask questions in the Zoom chat. I'll be trying to keep an eye on that, and will also give a few occasions during the workshop for folks to ask questions out loud. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Colab set-up steps\n",
    "\n",
    "Click the \"Copy to Drive\" button or otherwise go to \"File\" -> \"Save a Copy in Drive\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only run this code if you are in this notebook on Google Colab\n",
    "!git clone https://github.com/csbailey5t/TRLN-Workshop-2020.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only run this code if you are in this notebook on Google Colab\n",
    "!mv TRLN-Workshop-2020/model/ .\n",
    "!mv TRLN-Workshop-2020/data/ .\n",
    "!mv TRLN-Workshop-2020/output/ .\n",
    "!mv TRLN-Workshop-2020/texts/ ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only run this code if you are in this notebook on Google Colab\n",
    "!pip install pyldavis gensim==3.8.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages we're using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# semantic modeling\n",
    "import gensim\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# visualization\n",
    "import pyLDAvis\n",
    "import pyLDAvis.gensim\n",
    "from sklearn.manifold import TSNE\n",
    "from bokeh.io import output_file, output_notebook, save, show\n",
    "from bokeh.models import ColumnDataSource, OpenURL, TapTool\n",
    "from bokeh.palettes import viridis\n",
    "from bokeh.plotting import figure\n",
    "\n",
    "\n",
    "# data manipulation\n",
    "import pandas as pd\n",
    "\n",
    "# general utility\n",
    "import glob\n",
    "import itertools\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the corpus and determining approaches\n",
    "\n",
    "Let's begin by just taking a look at some of the individual OCR files to get a sense of what they might be like. We could look at the items by way of the Libraries' website, but anytime I'm doing text analysis work, I like to see the text I'll be working with directly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_dir = \"texts\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fns = glob.glob(\"texts/*.txt\")\n",
    "print(len(fns))\n",
    "fns[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(fns[0], 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you see in this text? Does the OCR look good? Are there parts of the text that you think shouldn't be included in a model that helps with discovering texts?\n",
    "\n",
    "Try picking a different file and reading through it with the same questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know a bit about what types of texts we have, what goals would you have in providing discovery for this collection? What aspects of the documents would you want to focus on to expose to scholars?\n",
    "\n",
    "One of the main advantages of using any sort of machine learning process is that we can show relationships between and features of the items in a collection that we had not otherwise known or shown. The types of features could vary greatly. Maybe we want to show relationships based on the content of the documents in some cases. Maybe we want to expose something in the metadata of the documents. We might want to do both. \n",
    "\n",
    "Here we're going to focus on the content, and specifically one type of model that allows us to make connections across the collection: topic modeling. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is topic modeling?\n",
    "\n",
    "According to [David Blei](http://www.cs.columbia.edu/~blei/topicmodeling.html), topic models are a \"suite of algorithms that uncover the hidden thematic structure in document collections.\" Topic models operate on the idea that for any given document collection, or corpus, there is a finite number of themes, or topics, from which the corpus draws and each document is composed of words that are associated with some number of those topics. While we don't necessarily think of an author simply dipping into buckets (topics) of words and putting them together to create a document, it's turned out to be a useful model for understanding collections of documents according to the themes that cut across the collection.\n",
    "\n",
    "There are quite a few types of topic models, but we'll focus on one of the most common forms: latent dirichlet allocation (LDA). LDA topic modeling is a form of unsupervised machine learning, wherein we provide an unlabeled corpus of texts to the algorithm, which then produces the model, though we often provide the number of topics that the algorithm should use for the model. While there are processes for determining the \"correct\" number of topics, many consider this part of topic modeling a bit of an art that is determined as much by the research questions of the person running the model as it is by the corpus and model. Other types of topic models highlight different aspects and problematics of types of corpora, such as temporal differentiation and author bias. \n",
    "\n",
    "A topic model gives us a number of data objects. We'll have a list of topics, which are distributions over terms, though we could think of topics somewhat simply as sets of regularly co-occuring terms. We'll also have a representation of each document in the corpus as a vector denoting the composition of the document according to the topics, that is, we'll have an account of how much of each document is associated with each topic. \n",
    "\n",
    "Key resource: [Probabilistic Topic Models](http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf) by David Blei. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other approaches that could be useful for discovery, especiallly for text collections.\n",
    "\n",
    "- Keyword extraction: especially for long texts or large collections where creating individual content metadata isn't within reach, keyword extraction can offer a way to generate metadata that could aid in modeling a collection\n",
    "- Automated summarization: we increasingly have full text search for digitized collections, such as through OCR or the IIIF search API, but having longer summarization could help in identifying documents that are really relevant rather than just inclusive of searched terms.  \n",
    "- Entity extraction, including geospatial data: identifying named entities can create metadata for specific approaches to a collection, an open opportunity to link to gazetteers and linked open data sources. Combined with interactive maps, extraction of geospatial entities can combine with other semantic modeling approaches to create information rich geospatial discovery.\n",
    "- Clustering and similarity: especially with primary sources, we often want to find or show related documents. We can do this through metadata and faceting, but have new avenues with content based similarity measures, though these (and other methods) often depend on having a defined corpus for context.\n",
    "- Distant viewing: my focus is on text, but there are a lot of people working on semantic information extraction from audio-visual materials, such as Lauren Tilton and Taylor Arnold with the [Distant Viewing project](https://distantviewing.org/).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling the corpus\n",
    "\n",
    "Now that we know what type of model, we'll use, let's jump in to building the model itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in and cleaning the documents\n",
    "\n",
    "How we're going to read in and clean our texts is somewhat particular to `gensim`, the library we're using for our topic model. You could absolutely approach this part of the process in different ways, but we'll stick with an approach recommended by the author of `gensim` so that if you're looking for documentation and help later, it will be easier to find. For this part we'll hew closely to the code in the following tutorial.\n",
    "\n",
    "[Radim Řehůřek's topic modeling tutorial](https://radimrehurek.com/topic_modeling_tutorial/2%20-%20Topic%20Modeling.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since corpora can be large, it's often a good idea to approach reading in data with streaming in mind. Rather than reading in all of our data at once and then processing it, we'll read each item in and process it one at a time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def head(stream, n=10):\n",
    "    \"\"\"Given a stream of data items, return just the first n as a list\"\"\"\n",
    "    return list(itertools.islice(stream, n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We currently have filenames that include the item id. We'll want to associate the processed texts with just item id, so we need to pull it out of the filename.\n",
    "def get_item_id(fn):\n",
    "    \"\"\"Given a filename, return just the item id\"\"\"\n",
    "    return os.path.split(fn)[1].split(\".\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cleaning texts is often iterative, and how much you clean your corpus depends on the model you use. For topic modeling, I typically start without cleaning at all, get results, then piece by piece add in the minimum necessary cleaning to get sensible results. What we'll do in the function below is based on that minimal approach. \n",
    "\n",
    "Just due to how `gensim` builds corpora for processing, we need to break each text down into its component tokens, which in this case are just the individual words of the corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"Given a text, tokenize it while removing stopwords, non-alpha characters, and one letter words\"\"\"\n",
    "    tokens = [token for token in word_tokenize(text) if token.lower() not in STOPWORDS]\n",
    "    cleaned = [token for token in tokens if token.isalpha()]\n",
    "    cleaned_greater_1 = [token for token in cleaned if len(token) > 1]\n",
    "    return cleaned_greater_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_stream(text_dir):\n",
    "    \"\"\"Given a directory of plain text files, return a stream of tuples with the item id from the filename and the cleaned, tokenized text\"\"\"\n",
    "    for fn in glob.glob(f\"{text_dir}/*.txt\"):\n",
    "        item_id = get_item_id(fn)\n",
    "        with open(fn, 'r') as f:\n",
    "            document = f.read()\n",
    "            yield(item_id, tokenize(document))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've defined all the functions we need to read in our documents and process them. We'll use the `head` utility function we wrote above to look at the first file and see how well our processing worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head(text_stream(text_dir), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also just look at the first or last bunch of tokens for each text to get a sense of the processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for item_id, tokens in head(text_stream(text_dir), n=5):\n",
    "    print(item_id, tokens[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for item_id, tokens in head(text_stream(text_dir), n=5):\n",
    "    print(item_id, tokens[-10:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to build the model, we now actually need to break apart the pieces of data that we put together in our text_stream function: the item id and the processed text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You could extract the item_ids from the full text_stream, but in order to not\n",
    "# tokenize everything when we don't yet need to we'll pull them directly from the filenames\n",
    "item_ids = [get_item_id(fn) for fn in fns]\n",
    "head(item_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a generator comprehension/expression. \n",
    "# What we get back is a generator that when called will provide the tokens for a single text, one at a time. \n",
    "doc_stream = (tokens for _, tokens in text_stream(text_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the model \n",
    "\n",
    "At this point, we have preprocessed texts that exist as lists of tokens, and a correspondingly ordered list of item ids. From here we'll build the different pieces of our model. \n",
    "\n",
    "First, we build a dictionary for the corpus, that is, a collection of the unique tokens (words) from the whole collection of documents. \n",
    "\n",
    "This will take a few minutes, so this is a great time for any questions you might have. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%time id2word_items = gensim.corpora.Dictionary(doc_stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(id2word_items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at those first few tokens, we can see we definitely have some words or abbreviations we would want to remove from a production model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter words based on occurence in docs\n",
    "# https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.filter_extremes\n",
    "# The filtered dictionary will only contain words that appear in at least two documents. We could also filter out words that appear in more than a percentage of the documents, and/or keep only the most frequent words. Sometimes this is good for a model, and sometimes not. It all depends on what you want your model to do. \n",
    "id2word_items.filter_extremes(no_below=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(id2word_items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corpus dictionary is now substantially smaller. It makes sense to filter out words that appear in only one document because we want to understand themes or topics that exist in the collection as a whole. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're building this for use in the LDA model and so we can save it to disk for re-use\n",
    "class ItemCorpus(object):\n",
    "    def __init__(self, text_dir, dictionary):\n",
    "        self.text_dir = text_dir\n",
    "        self.dictionary = dictionary\n",
    "        \n",
    "    def __iter__(self):\n",
    "        self.item_ids = []\n",
    "        for item_id, tokens in text_stream(text_dir):\n",
    "            self.item_ids.append(item_id)\n",
    "            yield self.dictionary.doc2bow(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_corpus = ItemCorpus(text_dir, id2word_items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save serialized corpus for later use\n",
    "# mm here is the Market Matrix format that Gensim prefers, though there are a few different formats that Gensim can work with.\n",
    "#%time gensim.corpora.MmCorpus.serialize(\"model/animal_turn_bow.mm\", item_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've saved our corpus to disk, we can easily reload it. Especially if we are working iteratively on our model, it's a great idea to save the corpora in multiple states of pre-processing in case we ever need to go back to a previous version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "loaded_corpus = gensim.corpora.MmCorpus(\"model/animal_turn_bow.mm\")\n",
    "print(loaded_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# DON'T run this cell during the workshop; it takes a bit too long for the live workshop.\n",
    "# We'll build an LDA topic model. LdaMulticore allows us to use multiple CPU cores to build the model. \n",
    "#%time lda_model = gensim.models.LdaMulticore(loaded_corpus, num_topics=40, id2word=id2word_items, passes=50, workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lda_model.save('model/animalturn_40.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the model\n",
    "\n",
    "Since the model takes 20ish minutes to train with 40 topics, we're going to just load the model that I've already trained rather than train a new one here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model = gensim.models.LdaModel.load(\"model/animalturn_40.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model.print_topics(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not the easiest way to read through the topics, and we'll keep exploring different views on the topics. \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def explore_topic(lda_model, topic_number, topn, output=True):\n",
    "    \"\"\"\n",
    "    accepts an ldamodel, a topic number and topn terms of interest\n",
    "    prints a formatted list of the topn terms\n",
    "    \"\"\"\n",
    "    terms = []\n",
    "    for term, frequency in lda_model.show_topic(topic_number, topn=topn):\n",
    "        terms += [term]\n",
    "        if output:\n",
    "            print(u'{:20} {:.3f}'.format(term, round(frequency, 3)))\n",
    "    return terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topic_summaries = []\n",
    "print(u'{:20} {}'.format(u'term', u'frequency') + u'\\n')\n",
    "for i in range(40):\n",
    "    print(\"Topic \" + str(i) + \" |---------------------\\n\")\n",
    "    tmp = explore_topic(lda_model, topic_number=i, topn=10, output=True)\n",
    "    topic_summaries += [tmp[:5]]\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a minute though to read through the lists of words above. Topic models don't give you any sort of title for a topic or tell you how they cohere. This is the part of the interpreter or expert. Pick out a couple of topics, and think about you might name the topic or describe it. \n",
    "\n",
    "Do any stand out to you? Does it seem like setting 40 topics was the right choice or does it seem like the topics are too general or too granular? \n",
    "\n",
    "Let's look at one other way of visualizing our model before we shift to thinking about visualizing the corpus. We'll take advantage of the great `pyLDAvis` library, which is a Python version of a previous R library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyLDAvis.enable_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will likely take a few minutes to load up, so feel free to ask any questions you might have been holding on to! \n",
    "pyLDAvis.gensim.prepare(lda_model, loaded_corpus, id2word_items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a great way to explore the relations between the topics, and to understand some of the thematic shape of the corpus. We can get a sense of how closely the topics relate to each other, and whether there are strands of themes that are significantly different than others or less frequent on the whole. This visualization may also just uncover particular words that surprise you and cause you to delve back into the collection, and into specific documents to understand what's happening. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some ways, this visualization is already a visualization of the corpus. We can think about any given collection as being presented in many ways. The original images or videos or texts are one way. A presentation of the human-created metadata in tabular format would be a different way. This type of model and its visualization might be another way. We could think about designing discovery in ways that show these different layers of presentation, a sort of thick, rich description of the collection as a phenomenon. \n",
    "\n",
    "But, what we haven't done yet is use this model as a way to visualize the individual items in the collection as a collection. We'll turn there now. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the corpus\n",
    "\n",
    "Let's start just with looking at the topic distribution for a single document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model.get_document_topics(loaded_corpus[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that for the first document in our corpus, the model considers it to be about 79% about topic 2, based on the words in the document. It considers it roughly 15% about topic 25, 4% about topic 35, and 1% about topic 18. If we need a reminder of the most significant topics we can check that.\n",
    "\n",
    "N.B. the topic numbers in pyLDAvis can be different from the topics numbers in the model (unless they've fixed that). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model.show_topic(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could take the numbers from `get_document_topics` and generate a visualization for each document that shows the significant topics for each document according to the model, even putting in nice tooltips or click events to show the top words for each topic. \n",
    "\n",
    "We're going to hold off on that, though, and stay with the idea of visualizating the whole collection. To do that, we'll build up a tabular dataset where every row is a document in the collection, and every column a topic proportion. The value at any given cell of the data will be the proportion of that document's words that are associated with that topic. A row would represent the vector of that document in the vector space, or feature space, of the collection model.   \n",
    "\n",
    "We'll use `pandas` to build this data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the list of item ids as a pandas Series\n",
    "source_id = pd.Series(item_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Above, `get_document_topics` only showed significant topics, but we can set a minimum probability to get numbers for all topics\n",
    "lda_model.get_document_topics(loaded_corpus[0], minimum_probability=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create headers for the DataFrame\n",
    "headers = [\"source_id\"]\n",
    "for i in range(40):\n",
    "    headers.append(f\"topic-{i}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that our last topic header is correctly numbered\n",
    "headers[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the DataFrame\n",
    "df = pd.DataFrame(columns=headers)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generally, building pandas DataFrames row by row is not a best practice, but it makes sense here given the gensim function that gives us the topic distribution for a document\n",
    "for i in range(len(item_ids)):\n",
    "    item_id = item_ids[i]\n",
    "    new_row = [item_id]\n",
    "    for _, prob in lda_model.get_document_topics(loaded_corpus[i], minimum_probability=0):\n",
    "        new_row.append(prob)\n",
    "    df.loc[item_id] = new_row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our data to a csv file \n",
    "df.to_csv(\"model/doc_topic_probs_model_40.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our collection as a set of vectors in a high-dimensional space. What does that mean and how do we visualize it?\n",
    "\n",
    "A lot of the time when we build graphs or visualizations, we have two dimensions: `x` and `y`. Anything that can be shown in that graph has a value for those two dimensions, and might be represented as a point or a vector. We could look at that data as a table where there are two columns, one for `x` and one for `y`. \n",
    "\n",
    "In our collection and topic model, though, our data has 40 columns. Each of those columns is a dimension in the space that the model defines. But, humans aren't so good at seeing in 40 dimensions, so we need to be able to map the 40 dimensional space to a 2 or 3 dimensional space. \n",
    "\n",
    "There is a whole set of algorithms and approaches to let us do this, much of it going under the name dimensionality reduction. We're only going to look at one approach, [t-distributed Stochastic Neighbor Embedding, or t-SNE](https://lvdmaaten.github.io/tsne/). t-SNE is built for dimensionality reduction for visualization purposes.\n",
    "\n",
    "Other algorithms loosely include: Principle Component Analysis (PCA), Multi-Dimensional Scaling (MDS) (such as Principle Coordinates Analysis (PCoA)), and Uniform Manifold Approximation and Project (UMAP). \n",
    "\n",
    "We'll run our process and check out a visualization of the collection, then talk through what's going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of the TSNE class with default parameters, including perplexity=30. Perplexity is sort of like the number of nearest neighbors. We don't have a guess right now at what the appropriate value is, so we'll stick with the default, but could go back and rerun with different values. \n",
    "tsne = TSNE()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need our data in the form of a numpy array, without the source_id\n",
    "lda_data = df.drop([\"source_id\"], axis=1).to_numpy() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_embedding = tsne.fit_transform(lda_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll turn our two dimensional array into a pandas dataframe for ease of use in visualization\n",
    "# We'll also add in a hue column that maps to the most significant topic for each document. `argmax` works here because the columns of the array correspond to the topics in our model\n",
    "tsne_df = pd.DataFrame(tsne_embedding, columns=[\"x\", \"y\"])\n",
    "tsne_df[\"hue\"] = lda_data.argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our data now reduced to two dimensions, we can plot it. We'll use `Bokeh`, a great interactive data visualization library in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we plot, though, let's layer in a bit more information that we would want in our visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_df = pd.read_csv(\"data/item-info.csv\")\n",
    "item_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_base = \"https://d.lib.ncsu.edu/collections/catalog/\"\n",
    "item_df = item_df.drop([\"Unnamed: 0\", \"prefix\", \"ocr_url\"], axis=1)\n",
    "item_urls = [f\"{url_base}{id}\" for id in item_df.id]\n",
    "item_df[\"item_url\"] = item_urls\n",
    "\n",
    "item_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "item_data_df = pd.merge(df, item_df, left_on=\"source_id\", right_on=\"id\")\n",
    "item_data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show our plot in the notebook\n",
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the data that sources the visualization\n",
    "source = ColumnDataSource(\n",
    "    data = dict(\n",
    "        x = tsne_df.x,\n",
    "        y = tsne_df.y, \n",
    "        source_id = item_data_df[\"source_id\"],\n",
    "        title = item_data_df[\"title\"],\n",
    "        url = item_data_df[\"item_url\"],\n",
    "        colors = [viridis(40)[i] for i in tsne_df.hue],\n",
    "        hue = tsne_df.hue\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to see the item id and top topic when we hover over each point in our plot, so we'll create tooltips. A bit of Bokeh magic maps anything with an `@` to a column in the `ColumnDataSource`\n",
    "tooltips = [\n",
    "    (\"source_id\", \"@source_id\"),\n",
    "    (\"topic\", \"@hue\"),\n",
    "    (\"title\", \"@title\")\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_title = \"t-SNE embedding of Animal Turn 40-topic Topic Model\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_tsne = figure(plot_width=800, plot_height=800, title=plot_title, tooltips=tooltips, tools=\"pan,tap,wheel_zoom,box_zoom,reset\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_tsne.circle(\"x\", \"y\", source = source, fill_color=\"colors\", fill_alpha=0.8, size=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"@url\"\n",
    "taptool = plot_tsne.select(type=TapTool)\n",
    "taptool.callback = OpenURL(url=url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(plot_tsne)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our plot as an html file\n",
    "save(plot_tsne, \"output/animal_turn_tsne_40_workshop.html\", title=\"animal_turn_tsne\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Critical Reflection?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}