3-Clustering-and-Topic-Modeling/3-Clustering-and-Topic-Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Week 3 - Clustering & Topic Modeling\n",
    "\n",
    "This week, we take a text corpus that we have developed, and we first break it into discrete document chunks through a process known as clustering or partitioning. We will pilot this here both with a well-known *flat* clustering method, `kmeans`, and also a *hierarchical* approach, `Ward's (minimum variance) method`. We will demonstrate a simple (graphical) approach to identifying optimal cluster number, the sillhouette method, and evaluate the quality of unsupervised clusters on labeled data. Next, we will explore a method of content clustering called topic modeling. This statistical technique models and computationally induces *topics* from data, which are sparse distributions over (nonexclusive clusters of) words, from which documents can formally be described as sparse mixtures. We will explore these topics and consider their utility for understanding trends within a corpus. Finally, we will consider how to construct models that take document cluster and topic loading as predictive features.\n",
    "\n",
    "For this notebook we will be using the following packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#All these packages need to be installed from pip\n",
    "#These are all for the cluster detection\n",
    "import sklearn\n",
    "import sklearn.feature_extraction.text\n",
    "import sklearn.pipeline\n",
    "import sklearn.preprocessing\n",
    "import sklearn.datasets\n",
    "import sklearn.cluster\n",
    "import sklearn.decomposition\n",
    "import sklearn.metrics\n",
    "\n",
    "import scipy #For hierarchical clustering and some visuals\n",
    "#import scipy.cluster.hierarchy\n",
    "import gensim#For topic modeling\n",
    "import nltk #the Natural Language Toolkit\n",
    "import requests #For downloading our datasets\n",
    "import numpy as np #for arrays\n",
    "import pandas #gives us DataFrames\n",
    "import matplotlib.pyplot as plt #For graphics\n",
    "import matplotlib.cm #Still for graphics\n",
    "import seaborn as sns #Makes the graphics look nicer\n",
    "\n",
    "#This 'magic' command makes the plots work better\n",
    "#in the notebook, don't use it outside of a notebook.\n",
    "#Also you can ignore the warning, it\n",
    "%matplotlib inline\n",
    "\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting our corpora\n",
    "\n",
    "To begin, we will use a well known corpus of testing documents from the *20 Newsgroups corpus*, a dataset commonly used to illustrate text applications of text clustering and classification. This comes packaged with sklearn and comprises approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 newsgroups. It was originally collected by Ken Lang, probably for his 1995 *Newsweeder: Learning to filter netnews* paper. The data is organized into 20 distinct newsgroups, each corresponding to a different topic. Some of the newsgroups are very closely related (e.g. comp.sys.ibm.pc.hardware / comp.sys.mac.hardware), while others are unrelated (e.g misc.forsale / soc.religion.christian). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroups = sklearn.datasets.fetch_20newsgroups(subset='train') #, data_home = '~/shared/sklearnData'\n",
    "print(dir(newsgroups))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can ascertain the categories with `target_names` or the actual files with `filenames`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(newsgroups.target_names)\n",
    "print(len(newsgroups.data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by converting the provided data into pandas DataFrames.\n",
    "\n",
    "First we reduce our dataset for this analysis by dropping some extraneous information and converting it into a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroupsCategories = ['comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos']\n",
    "\n",
    "newsgroupsDF = pandas.DataFrame(columns = ['text', 'category', 'source_file'])\n",
    "\n",
    "for category in newsgroupsCategories:\n",
    "    print(\"Fetching data for: {}\".format(category))\n",
    "    ng = sklearn.datasets.fetch_20newsgroups(subset='train', categories = [category], remove=['headers', 'footers', 'quotes'])#, data_home = '~/shared/sklearnData'\n",
    "    newsgroupsDF = newsgroupsDF.append(pandas.DataFrame({'text' : ng.data, 'category' : [category] * len(ng.data), 'source_file' : ng.filenames}), ignore_index=True)\n",
    "\n",
    "#Creating an explicit index column for later\n",
    "\n",
    "#newsgroupsDF['index'] = range(len(newsgroupsDF))\n",
    "#newsgroupsDF.set_index('index', inplace = True)\n",
    "print(len(newsgroupsDF))\n",
    "newsgroupsDF[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can convert the documents into word count vectors (e.g., *soc.religion.christian message a* might contain 3 mentions of \"church\", 2 of \"jesus\", 1 of \"religion\", etc., yielding a CountVector=[3,2,1,...])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#First it needs to be initialized\n",
    "ngCountVectorizer = sklearn.feature_extraction.text.CountVectorizer()\n",
    "#Then trained\n",
    "newsgroupsVects = ngCountVectorizer.fit_transform(newsgroupsDF['text'])\n",
    "print(newsgroupsVects.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a matrix with row a document and each column a word. The matrix is mostly zeros, so we store it as a sparse matrix, a data structure that contains and indexes only the nonzero entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroupsVects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the normal operations on this sparse matrix or convert it to normal matrix (not recommended for large sparse matrices :-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroupsVects[:10,:20].todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also lookup the indices of different words using the Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ngCountVectorizer.vocabulary_.get('vector')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some more interesting things to do...\n",
    "\n",
    "Lets start with [term frequency–inverse document frequency](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html)(tf-idf), a method for weighting document-distinguishing words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#initialize\n",
    "newsgroupsTFTransformer = sklearn.feature_extraction.text.TfidfTransformer().fit(newsgroupsVects)\n",
    "#train\n",
    "newsgroupsTF = newsgroupsTFTransformer.transform(newsgroupsVects)\n",
    "print(newsgroupsTF.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us the tf-idf for each word in each text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list(zip(ngCountVectorizer.vocabulary_.keys(), newsgroupsTF.data))[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first glance, there appears to be a lot of garbage littering this unordered list with unique words and stopwords. Note, however, that words like *apple*, *rgb*, and *voltage* distinguish this newsgroup document, while stopwords post a much lower weight. Note that we could filter out stop words, stem and lem our data before vectorizering, or we can instead use tf-idf to filter our data (or **both**). For exact explanation of all options look [here](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). To prune this matrix of features, we now limit our word vector to 1000 words with at least 3 occurrences, which do not occur in more than half of the documents. There is an extensive science and art to feature engineering for machine learning applications like clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#initialize\n",
    "ngTFVectorizer = sklearn.feature_extraction.text.TfidfVectorizer(max_df=0.5, max_features=1000, min_df=3, stop_words='english', norm='l2')\n",
    "#train\n",
    "newsgroupsTFVects = ngTFVectorizer.fit_transform(newsgroupsDF['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroupsTFVects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The matrix is much smaller now, only 1000 words, but the same number of documents\n",
    "\n",
    "We can still look at the words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    print(ngTFVectorizer.vocabulary_['vector'])\n",
    "except KeyError:\n",
    "    print('vector is missing')\n",
    "    print('The available words are: {} ...'.format(list(ngTFVectorizer.vocabulary_.keys())[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a reasonable matrix of features with which to begin identifying clusters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Flat Clustering with K-means\n",
    "\n",
    "Lets start with k-means, an approach that begins with random clusters of predefined number, then iterates cluster reassignment and evaluates the new clusters relative to an objective function, recursively.\n",
    "\n",
    "To do this we will need to know how many clusters we are looking for. Here the *true number* of clusters is 4. Of course, in most cases you would not know the number in advance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "numClusters = len(set(newsgroupsDF['category']))\n",
    "numClusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can initialize our cluster finder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#k-means++ is a better way of finding the starting points\n",
    "#We could also try providing our own\n",
    "km = sklearn.cluster.KMeans(n_clusters=numClusters, init='k-means++')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can calculate the clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "km.fit(newsgroupsTFVects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the clusters, we can evaluate them with a variety of metrics that sklearn provides. We will look at a few."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The available metrics are: {}\".format([s for s in dir(sklearn.metrics) if s[0] != '_']))\n",
    "print(\"for our clusters:\")\n",
    "print(\"Homogeneity: {:0.3f}\".format(sklearn.metrics.homogeneity_score(newsgroupsDF['category'], km.labels_)))\n",
    "print(\"Completeness: {:0.3f}\".format(sklearn.metrics.completeness_score(newsgroupsDF['category'], km.labels_)))\n",
    "print(\"V-measure: {:0.3f}\".format(sklearn.metrics.v_measure_score(newsgroupsDF['category'], km.labels_)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the contents of the clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "terms = ngTFVectorizer.get_feature_names()\n",
    "print(\"Top terms per cluster:\")\n",
    "order_centroids = km.cluster_centers_.argsort()[:, ::-1]\n",
    "for i in range(numClusters):\n",
    "    print(\"Cluster %d:\" % i)\n",
    "    for ind in order_centroids[i, :10]:\n",
    "        print(' %s' % terms[ind])\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's construct a visualization of the clusters. First, we will first reduce the\n",
    "dimensionality of the data using principal components analysis (PCA)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PCA = sklearn.decomposition.PCA\n",
    "pca = PCA(n_components = 2).fit(newsgroupsTFVects.toarray())\n",
    "reduced_data = pca.transform(newsgroupsTFVects.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The cell below is optional. It allows you to do a biplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "components = pca.components_\n",
    "keyword_ids = list(set(order_centroids[:,:10].flatten())) #Get the ids of the most distinguishing words(features) from your kmeans model.\n",
    "words = [terms[i] for i in keyword_ids]#Turn the ids into words.\n",
    "x = components[:,keyword_ids][0,:] #Find the coordinates of those words in your biplot.\n",
    "y = components[:,keyword_ids][1,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's build a color map for the true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "colordict = {\n",
    "'comp.sys.mac.hardware': 'red',\n",
    "'comp.windows.x': 'orange',\n",
    "'misc.forsale': 'green',\n",
    "'rec.autos': 'blue',\n",
    "    }\n",
    "colors = [colordict[c] for c in newsgroupsDF['category']]\n",
    "print(\"The categories' colors are:\\n{}\".format(colordict.items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the data using the true labels as the colors of our data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_frame_on(False)\n",
    "ax.scatter(reduced_data[:, 0], reduced_data[:, 1], color = colors, alpha = 0.5, label = colors)\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('True Classes')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One nice thing about PCA is that we can also do a biplot and map our feature\n",
    "vectors to the same space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (16,9))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_frame_on(False)\n",
    "ax.scatter(reduced_data[:, 0], reduced_data[:, 1], color = colors, alpha = 0.3, label = colors)\n",
    "for i, word in enumerate(words):\n",
    "    ax.annotate(word, (x[i],y[i]))\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('True Classes')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do it again with predicted clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "colors_p = [colordict[newsgroupsCategories[l]] for l in km.labels_]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_frame_on(False)\n",
    "plt.scatter(reduced_data[:, 0], reduced_data[:, 1], color = colors_p, alpha = 0.5)\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('Predicted Clusters\\n k = 4')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try with 3 clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "km3 = sklearn.cluster.KMeans(n_clusters= 3, init='k-means++')\n",
    "km3.fit(newsgroupsTFVects.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "colors_p = [colordict[newsgroupsCategories[l]] for l in km3.labels_]\n",
    "fig = plt.figure(figsize = (10,6))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_frame_on(False)\n",
    "plt.scatter(reduced_data[:, 0], reduced_data[:, 1], color = colors_p, alpha = 0.5)\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('Predicted Clusters\\n k = 3')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selecting Cluster Number\n",
    "\n",
    "Now we demonstrate the Silhouette method, one approach by which optimal number of clusters can be ascertained. Many other methods exist (e.g., Bayesian Information Criteria or BIC, the visual \"elbow criteria\", etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "range_n_clusters = [2, 3, 4, 5, 6]\n",
    "X = newsgroupsTFVects.toarray()\n",
    "\n",
    "for n_clusters in range_n_clusters:\n",
    "    # Create a subplot with 1 row and 2 columns\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2)\n",
    "    fig.set_size_inches(18, 7)\n",
    "\n",
    "    # The 1st subplot is the silhouette plot\n",
    "    # The silhouette coefficient can range from -1, 1 but in this example all\n",
    "    # lie within [-0.1, 1]\n",
    "    ax1.set_xlim([-0.1, 1])\n",
    "    # The (n_clusters+1)*10 is for inserting blank space between silhouette\n",
    "    # plots of individual clusters, to demarcate them clearly.\n",
    "    ax1.set_ylim([0, len(X) + (n_clusters + 1) * 10])\n",
    "\n",
    "    # Initialize the clusterer with n_clusters value and a random generator\n",
    "    # seed of 10 for reproducibility.\n",
    "    clusterer = sklearn.cluster.KMeans(n_clusters=n_clusters, random_state=10)\n",
    "    cluster_labels = clusterer.fit_predict(X)\n",
    "\n",
    "    # The silhouette_score gives the average value for all the samples.\n",
    "    # This gives a perspective into the density and separation of the formed\n",
    "    # clusters\n",
    "    silhouette_avg = sklearn.metrics.silhouette_score(X, cluster_labels)\n",
    "    print(\"For n_clusters =\", n_clusters,\n",
    "          \"The average silhouette_score is :\", silhouette_avg)\n",
    "\n",
    "    # Compute the silhouette scores for each sample\n",
    "    sample_silhouette_values = sklearn.metrics.silhouette_samples(X, cluster_labels)\n",
    "\n",
    "    y_lower = 10\n",
    "    for i in range(n_clusters):\n",
    "        # Aggregate the silhouette scores for samples belonging to\n",
    "        # cluster i, and sort them\n",
    "        ith_cluster_silhouette_values = \\\n",
    "            sample_silhouette_values[cluster_labels == i]\n",
    "\n",
    "        ith_cluster_silhouette_values.sort()\n",
    "\n",
    "        size_cluster_i = ith_cluster_silhouette_values.shape[0]\n",
    "        y_upper = y_lower + size_cluster_i\n",
    "\n",
    "        color = matplotlib.cm.spectral(float(i) / n_clusters)\n",
    "        ax1.fill_betweenx(np.arange(y_lower, y_upper),\n",
    "                          0, ith_cluster_silhouette_values,\n",
    "                          facecolor=color, edgecolor=color, alpha=0.7)\n",
    "\n",
    "        # Label the silhouette plots with their cluster numbers at the middle\n",
    "        ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))\n",
    "\n",
    "        # Compute the new y_lower for next plot\n",
    "        y_lower = y_upper + 10  # 10 for the 0 samples\n",
    "\n",
    "    ax1.set_title(\"The silhouette plot for the various clusters.\")\n",
    "    ax1.set_xlabel(\"The silhouette coefficient values\")\n",
    "    ax1.set_ylabel(\"Cluster label\")\n",
    "\n",
    "    # The vertical line for average silhouette score of all the values\n",
    "    ax1.axvline(x=silhouette_avg, color=\"red\", linestyle=\"--\")\n",
    "\n",
    "    ax1.set_yticks([])  # Clear the yaxis labels / ticks\n",
    "    ax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])\n",
    "\n",
    "    # 2nd Plot showing the actual clusters formed\n",
    "    colors = matplotlib.cm.spectral(cluster_labels.astype(float) / n_clusters)\n",
    "    ax2.scatter(reduced_data[:, 0], reduced_data[:, 1], marker='.', s=30, lw=0, alpha=0.7,\n",
    "                c=colors)\n",
    "\n",
    "    # Labeling the clusters\n",
    "    centers = clusterer.cluster_centers_\n",
    "    projected_centers = pca.transform(centers)\n",
    "    # Draw white circles at cluster centers\n",
    "    ax2.scatter(projected_centers[:, 0], projected_centers[:, 1],\n",
    "                marker='o', c=\"white\", alpha=1, s=200)\n",
    "\n",
    "    for i, c in enumerate(projected_centers):\n",
    "        ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1, s=50)\n",
    "\n",
    "    ax2.set_title(\"The visualization of the clustered data.\")\n",
    "    ax2.set_xlabel(\"PC 1\")\n",
    "    ax2.set_ylabel(\"PC 2\")\n",
    "\n",
    "    plt.suptitle((\"Silhouette analysis for KMeans clustering on sample data \"\n",
    "                  \"with n_clusters = %d\" % n_clusters),\n",
    "                 fontsize=14, fontweight='bold')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, the silhouette scores above suggests that 3 is a better number of clusters than 4, which would be accurate if we (reasonsably) grouped the two computer-themed groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting arbitrary text data\n",
    "\n",
    "Lets start by defining the same function as last lesson and loading a few press releases from 10 different senators into a DataFrame. The code to do this is below, but commented out as we've already downloaded the data to the data directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getGithubFiles(target, maxFiles = 100):\n",
    "    #We are setting a max so our examples don't take too long to run\n",
    "    #For converting to a DataFrame\n",
    "    releasesDict = {\n",
    "        'name' : [], #The name of the file\n",
    "        'text' : [], #The text of the file, watch out for binary files\n",
    "        'path' : [], #The path in the git repo to the file\n",
    "        'html_url' : [], #The url to see the file on Github\n",
    "        'download_url' : [], #The url to download the file\n",
    "    }\n",
    "\n",
    "    #Get the directory information from Github\n",
    "    r = requests.get(target)\n",
    "\n",
    "    #Check for rate limiting\n",
    "    if r.status_code != 200:\n",
    "        raise RuntimeError(\"Github didn't like your request, you have probably been rate limited.\")\n",
    "    filesLst = json.loads(r.text)\n",
    "\n",
    "    for fileDict in filesLst[:maxFiles]:\n",
    "        #These are provided by the directory\n",
    "        releasesDict['name'].append(fileDict['name'])\n",
    "        releasesDict['path'].append(fileDict['path'])\n",
    "        releasesDict['html_url'].append(fileDict['html_url'])\n",
    "        releasesDict['download_url'].append(fileDict['download_url'])\n",
    "\n",
    "        #We need to download the text though\n",
    "        text = requests.get(fileDict['download_url']).text\n",
    "        releasesDict['text'].append(text)\n",
    "\n",
    "    return pandas.DataFrame(releasesDict)\n",
    "\n",
    "targetSenator = 'Kennedy'# = ['Voinovich', 'Obama', 'Whitehouse', 'Snowe', 'Rockefeller', 'Murkowski', 'McCain', 'Kyl', 'Baucus', 'Frist']\n",
    "\"\"\"\n",
    "#Uncomment this to download your own data\n",
    "senReleasesTraining = pandas.DataFrame()\n",
    "\n",
    "print(\"Fetching {}'s data\".format(targetSenator))\n",
    "targetDF = getGithubFiles('https://api.github.com/repos/lintool/GrimmerSenatePressReleases/contents/raw/{}'.format(targetSenator), maxFiles = 2000)\n",
    "targetDF['targetSenator'] = targetSenator\n",
    "senReleasesTraining = senReleasesTraining.append(targetDF, ignore_index = True)\n",
    "\n",
    "#Watch out for weird lines when converting to csv\n",
    "#one of them had to be removed from the Kennedy data so it could be re-read\n",
    "senReleasesTraining.to_csv(\"data/senReleasesTraining.csv\")\n",
    "\"\"\"\n",
    "\n",
    "senReleasesTraining = pandas.read_csv(\"data/senReleasesTraining.csv\")\n",
    "\n",
    "senReleasesTraining[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the files we can tokenize and normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The normalized text is good, but we know that the texts will have a large amount of overlap so we can use tf-idf to remove some of the most frequent words. Before doing that, there is one empty cell, let's remove that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "senReleasesTraining = senReleasesTraining.dropna(axis=0, how='any')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Similar parameters to before, but stricter max df and no max num occurrences\n",
    "senTFVectorizer = sklearn.feature_extraction.text.TfidfVectorizer(max_df=100, min_df=2, stop_words='english', norm='l2')\n",
    "senTFVects = senTFVectorizer.fit_transform(senReleasesTraining['text'])\n",
    "senTFVectorizer.vocabulary_.get('senat', 'Missing \"Senate\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering with our new data\n",
    "\n",
    "One nice thing about using DataFrames for everything is that we can quickly convert code from one input to another. Below we are redoing the cluster detection with our senate data. If you setup your DataFrame the same way it should be able to run on this code, without much work.\n",
    "\n",
    "First we will define what we will be working with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "targetDF = senReleasesTraining\n",
    "textColumn = 'text'\n",
    "numCategories = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tf-IDf vectorizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "exampleTFVectorizer = sklearn.feature_extraction.text.TfidfVectorizer(max_df=0.5, max_features=1000, min_df=3, stop_words='english', norm='l2')\n",
    "#train\n",
    "exampleTFVects = ngTFVectorizer.fit_transform(targetDF[textColumn])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running k means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "exampleKM = sklearn.cluster.KMeans(n_clusters = numCategories, init='k-means++')\n",
    "exampleKM.fit(exampleTFVects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And visualize, this is more up to you, but we will do one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "examplePCA = sklearn.decomposition.PCA(n_components = 2).fit(exampleTFVects.toarray())\n",
    "reducedPCA_data = examplePCA.transform(exampleTFVects.toarray())\n",
    "\n",
    "colors = list(plt.cm.rainbow(np.linspace(0,1, numCategories)))\n",
    "colors_p = [colors[l] for l in exampleKM.labels_]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(1)\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_frame_on(False)\n",
    "plt.scatter(reducedPCA_data[:, 0], reducedPCA_data[:, 1], color = colors_p, alpha = 0.5)\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('Predicted Clusters\\n k = {}'.format(numCategories))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, there are probably two clusters. You can check with a Silhouette analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your Turn*</span>\n",
    "\n",
    "<span style=\"color:red\">Construct cells immediately below this that construct features and cluster your documents using K-means and a variety of cluster numbers. Interrogate the cluster contents in terms of both documents and features. Identify the optimal cluster number with Silhouette analysis. Plot the clusters and features after reducing with PCA. What does this cluster structure reveal about the structure of documents in your corpora? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hierarchical Clustering with Wald's Method\n",
    "\n",
    "Next we approach a hierchical clustering method, which proposes nested clusters at any resolution (at the finest resolution, every document is its own cluster).\n",
    "\n",
    "Here we must begin by calculating how similar the documents are to one another.\n",
    "\n",
    "As a first pass, we take our matrix of word counts per document\n",
    "`newsgroupsTFVects` and create a word occurrence matrix measuring how similar\n",
    "the documents are to each other based on their number of shared words. (Note one could perform the converse operation, a document occurrence matrix measuring how similar  words are to each other based on their number of collocated documents)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "newsgroupsCoocMat = newsgroupsTFVects * newsgroupsTFVects.T\n",
    "#set the diagonal to 0 since we don't care how similar texts are to themselves\n",
    "newsgroupsCoocMat.setdiag(0)\n",
    "#Another way of relating the texts is with their cosine similarity\n",
    "#newsgroupsCosinMat1 = 1 - sklearn.metrics.pairwise.cosine_similarity(newsgroupsTFVects)\n",
    "#But generally word occurrence is more accurate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compute a tree of nested clusters. Here we will only look at the first 100 texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "linkage_matrix = scipy.cluster.hierarchy.ward(newsgroupsCoocMat[:100, :100].toarray())\n",
    "linkage_matrix[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can visualize the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax = scipy.cluster.hierarchy.dendrogram(linkage_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot may seem somewhat unwieldy. To make it easier to read, we can cut the tree after a number of branchings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax = scipy.cluster.hierarchy.dendrogram(linkage_matrix, p=4, truncate_mode='level')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the tree is colored to show the clusters based on their ['distance'](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.cluster.hierarchy.dendrogram.html#scipy.cluster.hierarchy.dendrogram) from one another, but there are other ways of forming hierarchical clusters.\n",
    "\n",
    "Another approach involves cutting the tree into `n` branches. We can do this with [`fcluster()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.fcluster.html#scipy.cluster.hierarchy.fcluster). Lets break the tree into 4 clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hierarchicalClusters = scipy.cluster.hierarchy.fcluster(linkage_matrix, 4, 'maxclust')\n",
    "hierarchicalClusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us an array giving each element of `linkage_matrix`'s cluster. The leader function below is actually quite misleading. In this case, the ids it returns are actually not document ids but non-singleton clusters. You can ignore this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clusterLeaders = scipy.cluster.hierarchy.leaders(linkage_matrix, hierarchicalClusters)\n",
    "clusterLeaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now let's do it with our new data\n",
    "\n",
    "We can also do hierarchical clustering with the Senate data. Let's start by creating the linkage matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "exampleCoocMat = exampleTFVects * exampleTFVects.T\n",
    "exampleCoocMat.setdiag(0)\n",
    "examplelinkage_matrix = scipy.cluster.hierarchy.ward(exampleCoocMat[:100, :100].toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And visualize the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax = scipy.cluster.hierarchy.dendrogram(examplelinkage_matrix, p=5, truncate_mode='level')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, it's not very interesting :-)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your Turn*</span>\n",
    "\n",
    "<span style=\"color:red\">Construct cells immediately below this that hierarchically cluster your documents using two approaches, and visualize them with a tree. Interrogate the recursive cluster contents in terms of both documents and closenesses. What does this nested cluster structure reveal about the structure of documents in your corpora? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim\n",
    "\n",
    "To do topic modeling we will also be using data from the [grimmer press releases corpus](ttps://github.com/lintool/GrimmerSenatePressReleases). To use the texts with gensim we need to create a `corpua` object, this takes a few steps. First we create a `Dictionary` that maps tokens to ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Define the same function as last week\n",
    "def normlizeTokens(tokenLst, stopwordLst = None, stemmer = None, lemmer = None):\n",
    "    #We can use a generator here as we just need to iterate over it\n",
    "\n",
    "    #Lowering the case and removing non-words\n",
    "    workingIter = (w.lower() for w in tokenLst if w.isalpha())\n",
    "\n",
    "    #Now we can use the semmer, if provided\n",
    "    if stemmer is not None:\n",
    "        workingIter = (stemmer.stem(w) for w in workingIter)\n",
    "\n",
    "    #And the lemmer\n",
    "    if lemmer is not None:\n",
    "        workingIter = (lemmer.lemmatize(w) for w in workingIter)\n",
    "\n",
    "    #And remove the stopwords\n",
    "    if stopwordLst is not None:\n",
    "        workingIter = (w for w in workingIter if w not in stopwordLst)\n",
    "    #We will return a list with the stopwords removed\n",
    "    return list(workingIter)\n",
    "\n",
    "#initialize our stemmer and our stop words\n",
    "stop_words_nltk = nltk.corpus.stopwords.words('english')\n",
    "snowball = nltk.stem.snowball.SnowballStemmer('english')\n",
    "\n",
    "#Apply our functions\n",
    "senReleasesTraining['tokenized_text'] = senReleasesTraining['text'].apply(lambda x: nltk.word_tokenize(x))\n",
    "senReleasesTraining['normalized_tokens'] = senReleasesTraining['tokenized_text'].apply(lambda x: normlizeTokens(x, stopwordLst = stop_words_nltk, stemmer = snowball))\n",
    "\n",
    "senReleasesTraining[::100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dropMissing(wordLst, vocab):\n",
    "    return [w for w in wordLst if w in vocab]\n",
    "\n",
    "senReleasesTraining['reduced_tokens'] = senReleasesTraining['normalized_tokens'].apply(lambda x: dropMissing(x, senTFVectorizer.vocabulary_.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dictionary = gensim.corpora.Dictionary(senReleasesTraining['reduced_tokens'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then for each of the texts we create a list of tuples containing each token and its count. We will only use the first half of our dataset for now and will save the remainder for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus = [dictionary.doc2bow(text) for text in senReleasesTraining['reduced_tokens']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we serialize the corpus as a file and load it. This is an important step when the corpus is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gensim.corpora.MmCorpus.serialize('data/senate.mm', corpus)\n",
    "senmm = gensim.corpora.MmCorpus('data/senate.mm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a correctly formatted corpus that we can use for topic modeling and induction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "senlda = gensim.models.ldamodel.LdaModel(corpus=senmm, id2word=dictionary, num_topics=10, alpha='auto', eta='auto')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the degree to which distinct texts load on different topics. Here is one of the texts from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sen1Bow = dictionary.doc2bow(senReleasesTraining['reduced_tokens'][0])\n",
    "sen1lda = senlda[sen1Bow]\n",
    "print(\"The topics of the text: {}\".format(senReleasesTraining['name'][0]))\n",
    "print(\"are: {}\".format(sen1lda))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see which topics our model predicts press releases load on and make this into a `dataFrame` for later analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ldaDF = pandas.DataFrame({\n",
    "        'name' : senReleasesTraining['name'],\n",
    "        'topics' : [senlda[dictionary.doc2bow(l)] for l in senReleasesTraining['reduced_tokens']]\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a bit unwieldy so lets make each topic its own column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Dict to temporally hold the probabilities\n",
    "topicsProbDict = {i : [0] * len(ldaDF) for i in range(senlda.num_topics)}\n",
    "\n",
    "#Load them into the dict\n",
    "for index, topicTuples in enumerate(ldaDF['topics']):\n",
    "    for topicNum, prob in topicTuples:\n",
    "        topicsProbDict[topicNum][index] = prob\n",
    "\n",
    "#Update the DataFrame\n",
    "for topicNum in range(senlda.num_topics):\n",
    "    ldaDF['topic_{}'.format(topicNum)] = topicsProbDict[topicNum]\n",
    "\n",
    "ldaDF[1::100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's visualize this for several (e.g., 10) documents in the corpus. First we'll subset the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ldaDFV = ldaDF[:10][['topic_%d' %x for x in range(10)]]\n",
    "ldaDFVisN = ldaDF[:10][['name']]\n",
    "ldaDFVis = ldaDFV.as_matrix(columns=None)\n",
    "ldaDFVisNames = ldaDFVisN.as_matrix(columns=None)\n",
    "ldaDFV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we can visualize as a stacked bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "N = 10\n",
    "ind = np.arange(N)\n",
    "K = senlda.num_topics  # N documents, K topics\n",
    "ind = np.arange(N)  # the x-axis locations for the novels\n",
    "width = 0.5  # the width of the bars\n",
    "plots = []\n",
    "height_cumulative = np.zeros(N)\n",
    "\n",
    "for k in range(K):\n",
    "    color = plt.cm.coolwarm(k/K, 1)\n",
    "    if k == 0:\n",
    "        p = plt.bar(ind, ldaDFVis[:, k], width, color=color)\n",
    "    else:\n",
    "        p = plt.bar(ind, ldaDFVis[:, k], width, bottom=height_cumulative, color=color)\n",
    "    height_cumulative += ldaDFVis[:, k]\n",
    "    plots.append(p)\n",
    "    \n",
    "\n",
    "plt.ylim((0, 1))  # proportions sum to 1, so the height of the stacked bars is 1\n",
    "plt.ylabel('Topics')\n",
    "\n",
    "plt.title('Topics in Press Releases')\n",
    "plt.xticks(ind+width/2, ldaDFVisNames, rotation='vertical')\n",
    "\n",
    "plt.yticks(np.arange(0, 1, 10))\n",
    "topic_labels = ['Topic #{}'.format(k) for k in range(K)]\n",
    "plt.legend([p[0] for p in plots], topic_labels, loc='center left', frameon=True,  bbox_to_anchor = (1, .5))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualize as a heat map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.pcolor(ldaDFVis, norm=None, cmap='Blues')\n",
    "plt.yticks(np.arange(ldaDFVis.shape[0])+0.5, ldaDFVisNames);\n",
    "plt.xticks(np.arange(ldaDFVis.shape[1])+0.5, topic_labels);\n",
    "\n",
    "# flip the y-axis so the texts are in the order we anticipate (Austen first, then Brontë)\n",
    "plt.gca().invert_yaxis()\n",
    "\n",
    "# rotate the ticks on the x-axis\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "# add a legend\n",
    "plt.colorbar(cmap='Blues')\n",
    "plt.tight_layout()  # fixes margins\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the top words from each topic to get a sense of the semantic (or syntactic) domain they represent. To look at the terms with the highest LDA weight in topic `1` we can do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "senlda.show_topic(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we want to make a dataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "topicsDict = {}\n",
    "for topicNum in range(senlda.num_topics):\n",
    "    topicWords = [w for w, p in senlda.show_topic(topicNum)]\n",
    "    topicsDict['Topic_{}'.format(topicNum)] = topicWords\n",
    "\n",
    "wordRanksDF = pandas.DataFrame(topicsDict)\n",
    "wordRanksDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that several of the topics have the same top words, but there are definitely differences. We can try and make the topics more distinct by changing the $\\alpha$ and $\\eta$ parameters of the model. $\\alpha$ controls the sparsity of document-topic loadings, and $\\eta$ controls the sparsity of topic-word loadings.\n",
    "\n",
    "We can make a visualization of the distribution of words over any single topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "topic1_df = pandas.DataFrame(senlda.show_topic(1, topn=50))\n",
    "plt.figure()\n",
    "topic1_df.plot.bar(legend = False)\n",
    "plt.title('Probability Distribution of Words, Topic 1')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how different $\\eta$ values can change the shape of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "senlda1 = gensim.models.ldamodel.LdaModel(corpus=senmm, id2word=dictionary, num_topics=10, eta = 0.00001)\n",
    "senlda2 = gensim.models.ldamodel.LdaModel(corpus=senmm, id2word=dictionary, num_topics=10, eta = 0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "topic11_df = pandas.DataFrame(senlda1.show_topic(1, topn=50))\n",
    "topic21_df = pandas.DataFrame(senlda2.show_topic(1, topn=50))\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2)\n",
    "fig.set_size_inches(18, 7)\n",
    "topic11_df.plot.bar(legend = False, ax = ax1, title = '$\\eta$  = 0.00001')\n",
    "topic21_df.plot.bar(legend = False, ax = ax2, title = '$\\eta$  = 0.9')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your Turn*</span>\n",
    "\n",
    "<span style=\"color:red\">Construct cells immediately below this that topic model documents related to your anticipated final project. Interrogate and visually plot (e.g., as a bar graph?) the topic-word loadings and the document-topic loadings. What does this topic structure reveal about the distribution of contents across your documents?\n",
    "\n",
    "<span style=\"color:red\">**Stretch**: Cluster your documents, but instead of using words alone, use their topic loadings as an additional set of features. Do these topic loadings increase the apparent semantic coherence of your clusters?</span> "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}