unsupervised/unsupervised_datacamp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised Learning DataCamp\n",
    "\n",
    "### Cluster points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import KMeans\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# Create a KMeans instance with 3 clusters: model\n",
    "model = KMeans(n_clusters=3)\n",
    "\n",
    "# Fit model to points\n",
    "model.fit(points)\n",
    "\n",
    "# Determine the cluster labels of new_points: labels\n",
    "labels = model.predict(new_points)\n",
    "\n",
    "# Print cluster labels of new_points\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pyplot\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Assign the columns of new_points: xs and ys\n",
    "xs = new_points[:,0]\n",
    "ys = new_points[:,1]\n",
    "\n",
    "# Make a scatter plot of xs and ys, using labels to define the colors\n",
    "plt.scatter(xs, ys, c=labels, alpha=0.5)\n",
    "\n",
    "# Assign the cluster centers: centroids\n",
    "centroids = model.cluster_centers_\n",
    "\n",
    "# Assign the columns of centroids: centroids_x, centroids_y\n",
    "centroids_x = centroids[:,0]\n",
    "centroids_y = centroids[:,1]\n",
    "\n",
    "# Make a scatter plot of centroids_x and centroids_y\n",
    "plt.scatter(centroids_x, centroids_y, marker='D', s=50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating clustering quality\n",
    "- good clustering has tight clusters\n",
    "- spread is measured by the inertia: how far samples are from their centroids\n",
    "- lower values of the inertia are better - not as spread out\n",
    "- inertia is available as the inertia attribute\n",
    "- picking the number for k:\n",
    "    - Plot k by inertia\n",
    "    - pick an elbow in the plot\n",
    "    - this is when the inertia starts to decrease more slowly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ks = range(1, 6)\n",
    "inertias = []\n",
    "\n",
    "for k in ks:\n",
    "    # Create a KMeans instance with k clusters: model\n",
    "    model = KMeans(n_clusters=k)\n",
    "\n",
    "    # Fit model to samples\n",
    "    model.fit(samples)\n",
    "\n",
    "    # Append the inertia to the list of inertias\n",
    "    inertias.append(model.inertia_)\n",
    "\n",
    "# Plot ks vs inertias\n",
    "plt.plot(ks, inertias, '-o')\n",
    "plt.xlabel('number of clusters, k')\n",
    "plt.ylabel('inertia')\n",
    "plt.xticks(ks)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create model using 'elbow' number for k & calculate crosstab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a KMeans model with 3 clusters: model\n",
    "model = KMeans(n_clusters=3)\n",
    "\n",
    "# Use fit_predict to fit model and obtain cluster labels: labels\n",
    "# Using .fit_predict() is the same as using .fit() followed by .predict()\n",
    "labels = model.fit_predict(samples)\n",
    "\n",
    "# Create a DataFrame with labels and varieties as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'varieties': varieties})\n",
    "\n",
    "# Create crosstab: ct\n",
    "# Use the pd.crosstab() function on df['labels'] and df['varieties']\n",
    "# to count the number of times each grain variety coincides with each cluster label\n",
    "ct = pd.crosstab(df['labels'], df['varieties'])\n",
    "\n",
    "# Display ct\n",
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalise & create pipeline\n",
    "- feature variance = feature influence\n",
    "- scikit-learn: `standardscaler` transforms each feature to have a mean of 0 and variance of 1\n",
    "- This gives you standardised features\n",
    "\n",
    "#### Scaling fish data for clustering\n",
    "You are given an array samples giving measurements of fish. Each row represents an individual fish. The measurements, such as weight in grams, length in centimeters, and the percentage ratio of height to length, have very different scales. In order to cluster this data effectively, you'll need to standardize these features first. In this exercise, you'll build a pipeline to standardize and cluster the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# Create scaler: scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create KMeans instance: kmeans\n",
    "kmeans = KMeans(n_clusters=4)\n",
    "\n",
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, kmeans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Fit the pipeline to samples\n",
    "pipeline.fit(samples)\n",
    "\n",
    "# Calculate the cluster labels: labels\n",
    "labels = pipeline.predict(samples)\n",
    "\n",
    "# Create a DataFrame with labels and species as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'species': species})\n",
    "\n",
    "# Create crosstab: ct\n",
    "ct = pd.crosstab(df['labels'], df['species'])\n",
    "\n",
    "# Display ct\n",
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering stocks using KMeans\n",
    "\n",
    "In this exercise, you'll cluster companies using their daily stock price movements (i.e. the dollar difference between the closing and opening prices for each trading day). You are given a NumPy array movements of daily price movements from 2010 to 2015 (obtained from Yahoo! Finance), where each row corresponds to a company, and each column corresponds to a trading day.\n",
    "\n",
    "Some stocks are more expensive than others. To account for this, include a Normalizer at the beginning of your pipeline. The Normalizer will separately transform each company's stock price to a relative scale before the clustering begins.\n",
    "\n",
    "Note that Normalizer() is different to StandardScaler(), which you used in the previous exercise. While StandardScaler() standardizes features (such as the features of the fish data from the previous exercise) by removing the mean and scaling to unit variance, Normalizer() rescales each sample - here, each company's stock price - independently of the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Normalizer\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "# Create a normalizer: normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "# Create a KMeans model with 10 clusters: kmeans\n",
    "kmeans = KMeans(n_clusters=10)\n",
    "\n",
    "# Make a pipeline chaining normalizer and kmeans: pipeline\n",
    "pipeline = make_pipeline(normalizer, kmeans)\n",
    "\n",
    "# Fit pipeline to the daily price movements\n",
    "pipeline.fit(movements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which stocks move together?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Predict the cluster labels: labels\n",
    "labels = pipeline.predict(movements)\n",
    "\n",
    "# Create a DataFrame aligning labels and companies: df\n",
    "df = pd.DataFrame({'labels': labels, 'companies': companies})\n",
    "\n",
    "# Display df sorted by cluster label\n",
    "print(df.sort_values('labels'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical clustering\n",
    "- Arranges clusters into a hierarchy of clusters\n",
    "- Process\n",
    "    - Each sample starts as a seperate cluster\n",
    "    - With each step, the closest clusters are merged\n",
    "    - Continues until only a single cluster\n",
    "    - This is referred to as \"agglomerative\" clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from scipy.cluster.hierarchy import linkage, dendrogram\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(samples, method='complete')\n",
    "\n",
    "# Plot the dendrogram, using varieties as labels\n",
    "dendrogram(mergings,\n",
    "           labels=varieties,\n",
    "           leaf_rotation=90,\n",
    "           leaf_font_size=6,\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import normalize\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "# Normalize the movements: normalized_movements\n",
    "normalized_movements = normalize(movements)\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(normalized_movements, method='complete')\n",
    "\n",
    "# Plot the dendrogram\n",
    "dendrogram(mergings,\n",
    "           labels=companies,\n",
    "           leaf_rotation=90,\n",
    "           leaf_font_size=6,\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cluster labels from hierarchical clustering\n",
    "- Cluster labels at intermediate stages can be recovered\n",
    "- Can use these labels for additional analysis and can be reviewed using cross-tabulations\n",
    "- Choose intermediate stage by specifying a height on the dendrogram\n",
    "- Height: distance between merging clusters\n",
    "- Can specify not to merge clusters futher apart than a certain height\n",
    "- Distance between clusters is defined using a linkage method\n",
    "- Can modify method paramter e.g. complete linkage where the distance between clusters is the max distance between their samples\n",
    "- Multiple different linkage methods\n",
    "- Can use the fcluster method to get a numpy array containing cluster labels\n",
    "\n",
    "### Notes\n",
    "- scipy cluster labels start from 1 rather than 0\n",
    "\n",
    "### Different linkage, different hierarchical clustering\n",
    "In the video, you saw a hierarchical clustering of the voting countries at the Eurovision song contest using 'complete' linkage. Now, perform a hierarchical clustering of the voting countries with 'single' linkage, and compare the resulting dendrogram with the one in the video. Different linkage, different hierarchical clustering!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.cluster.hierarchy import linkage, dendrogram\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(samples, method='single')\n",
    "\n",
    "# Plot the dendrogram\n",
    "dendrogram(mergings,\n",
    "           labels=country_names,\n",
    "           leaf_rotation=90,\n",
    "           leaf_font_size=6,\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the cluster labels\n",
    "In the previous exercise, you saw that the intermediate clustering of the grain samples at height 6 has 3 clusters. Now, use the fcluster() function to extract the cluster labels for this intermediate clustering, and compare the labels with the grain varieties using a cross-tabulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "import pandas as pd\n",
    "from scipy.cluster.hierarchy import fcluster\n",
    "\n",
    "# Use fcluster to extract labels: labels\n",
    "# max height of 6\n",
    "labels = fcluster(mergings, 6, criterion='distance')\n",
    "\n",
    "# Create a DataFrame with labels and varieties as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'varieties': varieties})\n",
    "\n",
    "# Create crosstab: ct\n",
    "ct = pd.crosstab(df['labels'], df['varieties'])\n",
    "\n",
    "# Display ct\n",
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## t-SNE\n",
    "- Maps samples from higher dimensions to 2D/3D\n",
    "- Great aid for inspecting datasets\n",
    "- Has fit_transform method\n",
    "- No seperate fit & transform methods\n",
    "- Can't extend to new samples, have to start over each time\n",
    "- Choose learning rate for dataset\n",
    "    - Bad choice gives you all your values bunched together\n",
    "    - Experiment with values between 50 and 200\n",
    "- Axes have no meaning and are different each time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TSNE\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# Create a TSNE instance: model\n",
    "model = TSNE(learning_rate=200)\n",
    "\n",
    "# Apply fit_transform to samples: tsne_features\n",
    "tsne_features = model.fit_transform(samples)\n",
    "\n",
    "# Select the 0th feature: xs\n",
    "xs = tsne_features[:,0]\n",
    "\n",
    "# Select the 1st feature: ys\n",
    "ys = tsne_features[:,1]\n",
    "\n",
    "# Scatter plot, coloring by variety_numbers\n",
    "plt.scatter(xs, ys, c=variety_numbers)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TSNE\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# Create a TSNE instance: model\n",
    "model = TSNE(learning_rate=50)\n",
    "\n",
    "# Apply fit_transform to normalized_movements: tsne_features\n",
    "tsne_features = model.fit_transform(normalized_movements)\n",
    "\n",
    "# Select the 0th feature: xs\n",
    "xs = tsne_features[:,0]\n",
    "\n",
    "# Select the 1th feature: ys\n",
    "ys = tsne_features[:,1]\n",
    "\n",
    "# Scatter plot\n",
    "plt.scatter(xs, ys, alpha=0.5)\n",
    "\n",
    "# Annotate the points\n",
    "for x, y, company in zip(xs, ys, companies):\n",
    "    plt.annotate(company, (x, y), fontsize=5, alpha=0.75)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimension reduction\n",
    "- Finds patterns in data that allow you to express it in a compressed form\n",
    "- More efficient\n",
    "- Removes 'noise' features\n",
    "\n",
    "### PCA\n",
    "- Rotates the samples so they're aligned with the coordinate axes\n",
    "- Shifts data to have mean 0\n",
    "- No information is lost\n",
    "- PCA features are not correlated: the columns are no longer linearly correlated\n",
    "- Principal components of the data: direction in which the samples vary the most\n",
    "- Discards low variance PCA features\n",
    "- Assumes high variance features are informative\n",
    "    \n",
    "![](../../resources/pca_decorrelate.png)\n",
    "\n",
    "#### sklearn\n",
    "- Implementation of PCA\n",
    "- fit: learns how to shift and rotate samples but doesn't change them\n",
    "- transform applies the learned transformation and can be applied to new data\n",
    "- Principal components are available as the `components_` attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlated data in nature\n",
    "\n",
    "You are given an array grains giving the width and length of samples of grain. You suspect that width and length will be correlated. To confirm this, make a scatter plot of width vs length and measure their Pearson correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import pearsonr\n",
    "\n",
    "# Assign the 0th column of grains: width\n",
    "width = grains[:,0]\n",
    "\n",
    "# Assign the 1st column of grains: length\n",
    "length = grains[:,1]\n",
    "\n",
    "# Scatter plot width vs length\n",
    "plt.scatter(width, length)\n",
    "plt.axis('equal')\n",
    "plt.show()\n",
    "\n",
    "# Calculate the Pearson correlation\n",
    "correlation, pvalue = pearsonr(width, length)\n",
    "\n",
    "# Display the correlation\n",
    "print(correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decorrelating the grain measurements with PCA\n",
    "\n",
    "You observed in the previous exercise that the width and length measurements of the grain are correlated. Now, you'll use PCA to decorrelate these measurements, then plot the decorrelated points and measure their Pearson correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Create PCA instance: model\n",
    "model = PCA()\n",
    "\n",
    "# Apply the fit_transform method of model to grains: pca_features\n",
    "pca_features = model.fit_transform(grains)\n",
    "\n",
    "# Assign 0th column of pca_features: xs\n",
    "xs = pca_features[:,0]\n",
    "\n",
    "# Assign 1st column of pca_features: ys\n",
    "ys = pca_features[:,1]\n",
    "\n",
    "# Scatter plot xs vs ys\n",
    "plt.scatter(xs, ys)\n",
    "plt.axis('equal')\n",
    "plt.show()\n",
    "\n",
    "# Calculate the Pearson correlation of xs and ys\n",
    "correlation, pvalue = pearsonr(xs, ys)\n",
    "\n",
    "# Display the correlation\n",
    "print(correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intrinsic dimension\n",
    "- Number of features needed to approximate a dataset\n",
    "- Essential idea behind dimension reduction: tells you how much it can be compressed\n",
    "- Example: iris dataset:\n",
    "    - Plot 3 features = 3D\n",
    "    - Can see that they're mostly lying flat, like a sheet, therefore the intrinsic dimension is 2\n",
    "    - Only need 2 coordinates wtihout losing much additional information, need PCA for more\n",
    "    - PCA will rotate and shift the features to align them with the coordinate axis\n",
    "    - This expresses the features using 3 PCA features\n",
    "    - PCA features are arranged in order of their variance\n",
    "    - The third dimension for the iris dataset has very low variation\n",
    "- PCA identifies the intrinsic dimension when samples have any number of features\n",
    "- Intrinsic dimension = number of PCA features with significant variance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a scatter plot of the untransformed points\n",
    "plt.scatter(grains[:,0], grains[:,1])\n",
    "\n",
    "# Create a PCA instance: model\n",
    "model = PCA()\n",
    "\n",
    "# Fit model to points\n",
    "model.fit(grains)\n",
    "\n",
    "# Get the mean of the grain samples: mean\n",
    "mean = model.mean_\n",
    "\n",
    "# Get the first principal component: first_pc\n",
    "first_pc = model.components_[0,:]\n",
    "\n",
    "# Plot first_pc as an arrow, starting at mean\n",
    "plt.arrow(mean[0], mean[1], first_pc[0], first_pc[1], color='red', width=0.01)\n",
    "\n",
    "# Keep axes on same scale\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance of the PCA features\n",
    "\n",
    "The fish dataset is 6-dimensional. But what is its intrinsic dimension? Make a plot of the variances of the PCA features to find out. As before, samples is a 2D array, where each row represents a fish. You'll need to standardize the features first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create scaler: scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create a PCA instance: pca\n",
    "pca = PCA()\n",
    "\n",
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, pca)\n",
    "\n",
    "# Fit the pipeline to 'samples'\n",
    "pipeline.fit(samples)\n",
    "\n",
    "# Plot the explained variances\n",
    "features = range(pca.n_components_)\n",
    "plt.bar(features, pca.explained_variance_)\n",
    "plt.xlabel('PCA feature')\n",
    "plt.ylabel('variance')\n",
    "plt.xticks(features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimension reduction of the fish measurements\n",
    "\n",
    "In a previous exercise, you saw that 2 was a reasonable choice for the \"intrinsic dimension\" of the fish measurements. Now use PCA for dimensionality reduction of the fish measurements, retaining only the 2 most important components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Create a PCA model with 2 components: pca\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "# Fit the PCA instance to the scaled samples\n",
    "pca.fit(scaled_samples)\n",
    "\n",
    "# Transform the scaled samples: pca_features\n",
    "pca_features = pca.transform(scaled_samples)\n",
    "\n",
    "# Print the shape of pca_features\n",
    "print(pca_features.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternatives to PCA\n",
    "- Work frequency arrays are an example of when you can't really use PCA\n",
    "- Rows correspond to documents, and columns to words\n",
    "- Entries: how often each word occurs in each document (most are 0)\n",
    "- Array = sparse: use `scipy.sparse_csr_matrix` instead of NumPy array: remember only non-zero entries\n",
    "- PCA doesn't work with sparse matrics, use sklearn's TruncatedSVD instead\n",
    "- Performs the same transformation\n",
    "\n",
    "### A tf-idf word-frequency array\n",
    "\n",
    "In this exercise, you'll create a tf-idf word frequency array for a toy collection of documents. For this, use the TfidfVectorizer from sklearn. It transforms a list of documents into a word frequency array, which it outputs as a csr_matrix. It has fit() and transform() methods like other sklearn objects.\n",
    "\n",
    "- tf: frequency of the word in the document: 10% = tf of 0.1\n",
    "- idf: weighting scheme that reduces the influence of frequent words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Create a TfidfVectorizer: tfidf\n",
    "tfidf = TfidfVectorizer() \n",
    "\n",
    "# Apply fit_transform to document: csr_mat\n",
    "csr_mat = tfidf.fit_transform(documents)\n",
    "\n",
    "# Print result of toarray() method\n",
    "print(csr_mat.toarray())\n",
    "\n",
    "# Get the words: words\n",
    "words = tfidf.get_feature_names()\n",
    "\n",
    "# Print words\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering Wikipedia\n",
    "\n",
    "You saw in the video that TruncatedSVD is able to perform PCA on sparse arrays in csr_matrix format, such as word-frequency arrays. Combine your knowledge of TruncatedSVD and k-means to cluster some popular pages from Wikipedia. In this exercise, build the pipeline. In the next exercise, you'll apply it to the word-frequency array of some Wikipedia articles.\n",
    "\n",
    "Create a Pipeline object consisting of a TruncatedSVD followed by KMeans. (This time, we've precomputed the word-frequency matrix for you, so there's no need for a TfidfVectorizer).\n",
    "\n",
    "Next, you are given an array articles of tf-idf word-frequencies of some popular Wikipedia articles, and a list titles of their titles. Use your pipeline to cluster the Wikipedia articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.pipeline import make_pipeline\n",
    "import pandas as pd\n",
    "\n",
    "# Create a TruncatedSVD instance: svd\n",
    "svd = TruncatedSVD(n_components=50)\n",
    "\n",
    "# Create a KMeans instance: kmeans\n",
    "kmeans = KMeans(n_clusters=6)\n",
    "\n",
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(svd, kmeans)\n",
    "\n",
    "# Fit the pipeline to articles\n",
    "pipeline.fit(articles)\n",
    "\n",
    "# Calculate the cluster labels: labels\n",
    "labels = pipeline.predict(articles)\n",
    "\n",
    "# Create a DataFrame aligning labels and titles: df\n",
    "df = pd.DataFrame({'label': labels, 'article': titles})\n",
    "\n",
    "# Display df sorted by cluster label\n",
    "print(df.sort_values('label'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-negative matrix factorization (NMF)\n",
    "- Dimension reduction technique like PCA\n",
    "- NMF models are interpretable (unlike PCA)\n",
    "- All sample features must be non-negative\n",
    "- Works by decomposing samples as sums of their parts\n",
    "- Unlike PCA, must always specify the number of components\n",
    "- Works with NumPy and sparse arrays\n",
    "- NMF has components it learns from the samples like PCA learns principal components\n",
    "- Also like PCA, the dimension of the components = dimension of samples\n",
    "- NMF entries and feature values are always non-negative\n",
    "- Features and components of model can be used to reconstruct the original data samples\n",
    "- Sample reconstruction: multiply components by feature values, add up: product of matrices\n",
    "- NMF learns interpretable parts\n",
    "- Patterns of NMF represents patterns of frequently occuring samples\n",
    "- For images, NMF components are parts of images\n",
    "\n",
    "### NMF applied to Wikipedia articles\n",
    "\n",
    "Now it's your turn to apply NMF, this time using the tf-idf word-frequency array of Wikipedia articles, given as a csr matrix articles. Here, fit the model and transform the articles. In the next exercise, you'll explore the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NMF\n",
    "from sklearn.decomposition import NMF\n",
    "\n",
    "# Create an NMF instance: model\n",
    "model = NMF(n_components=6)\n",
    "\n",
    "# Fit the model to articles\n",
    "model.fit(articles)\n",
    "\n",
    "# Transform the articles: nmf_features\n",
    "nmf_features = model.transform(articles)\n",
    "\n",
    "# Print the NMF features\n",
    "print(nmf_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF features of the Wikipedia articles\n",
    "\n",
    "Now you will explore the NMF features you created in the previous exercise. A solution to the previous exercise has been pre-loaded, so the array nmf_features is available. Also available is a list titles giving the title of each Wikipedia article.\n",
    "\n",
    "When investigating the features, notice that for both actors, the NMF feature 3 has by far the highest value. This means that both articles are reconstructed using mainly the 3rd NMF component. In the next video, you'll see why: NMF components represent topics (for instance, acting!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Create a pandas DataFrame: df\n",
    "df = pd.DataFrame(nmf_features, index=titles)\n",
    "\n",
    "# Print the row for 'Anne Hathaway'\n",
    "print(df.loc['Anne Hathaway'])\n",
    "\n",
    "# Print the row for 'Denzel Washington'\n",
    "print(df.loc['Denzel Washington'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF learns topics of documents\n",
    "\n",
    "In the video, you learned when NMF is applied to documents, the components correspond to topics of documents, and the NMF features reconstruct the documents from the topics. Verify this for yourself for the NMF model that you built earlier using the Wikipedia articles. Previously, you saw that the 3rd NMF feature value was high for the articles about actors Anne Hathaway and Denzel Washington. In this exercise, identify the topic of the corresponding NMF component.\n",
    "\n",
    "The NMF model you built earlier is available as model, while words is a list of the words that label the columns of the word-frequency array.\n",
    "\n",
    "After you are done, take a moment to recognise the topic that the articles about Anne Hathaway and Denzel Washington have in common!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Create a DataFrame: components_df\n",
    "components_df = pd.DataFrame(model.components_, columns=words)\n",
    "\n",
    "# Print the shape of the DataFrame\n",
    "print(components_df.shape)\n",
    "\n",
    "# Select row 3: component\n",
    "component = components_df.iloc[3]\n",
    "\n",
    "# Print result of nlargest\n",
    "print(component.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the LED digits dataset\n",
    "\n",
    "In the following exercises, you'll use NMF to decompose grayscale images into their commonly occurring patterns. Firstly, explore the image dataset and see how it is encoded as an array. You are given 100 images as a 2D array samples, where each row represents a single 13x8 image. The images in your dataset are pictures of a LED digital display."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pyplot\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Select the 0th row: digit\n",
    "digit = samples[0,:]\n",
    "\n",
    "# Print digit\n",
    "print(digit)\n",
    "\n",
    "# Reshape digit to a 13x8 array: bitmap\n",
    "bitmap = digit.reshape(13, 8)\n",
    "\n",
    "# Print bitmap\n",
    "print(bitmap)\n",
    "\n",
    "# Use plt.imshow to display bitmap\n",
    "plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF learns the parts of images\n",
    "\n",
    "Now use what you've learned about NMF to decompose the digits dataset. You are again given the digit images as a 2D array samples. This time, you are also provided with a function show_as_image() that displays the image encoded by any 1D array:\n",
    "\n",
    "```\n",
    "def show_as_image(sample):\n",
    "    bitmap = sample.reshape((13, 8))\n",
    "    plt.figure()\n",
    "    plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "    plt.colorbar()\n",
    "    plt.show()\n",
    "```\n",
    "\n",
    "After you are done, take a moment to look through the plots and notice how NMF has expressed the digit as a sum of the components!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NMF\n",
    "from sklearn.decomposition import NMF\n",
    "\n",
    "# Create an NMF model: model\n",
    "# 7 = number of cells in an led display\n",
    "model = NMF(n_components=7)\n",
    "\n",
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(samples)\n",
    "\n",
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)\n",
    "\n",
    "# Assign the 0th row of features: digit_features\n",
    "digit_features = features[0,:]\n",
    "\n",
    "# Print digit_features\n",
    "print(digit_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PCA doesn't learn parts\n",
    "\n",
    "Unlike NMF, PCA doesn't learn the parts of things. Its components do not correspond to topics (in the case of documents) or to parts of images, when trained on images. Verify this for yourself by inspecting the components of a PCA model fit to the dataset of LED digit images from the previous exercise. The images are available as a 2D array samples. Also available is a modified version of the show_as_image() function which colors a pixel red if the value is negative.\n",
    "\n",
    "After submitting the answer, notice that the components of PCA do not represent meaningful parts of images of LED digits!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Create a PCA instance: model\n",
    "model = PCA(n_components=7)\n",
    "\n",
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(samples)\n",
    "\n",
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building recommender systems using NMF\n",
    "- Apply NMF to word-frequency array\n",
    "- NMF feature values describe topic: similar documents have similar NMF feature values\n",
    "- Compare cosine similarity to identify similar documents: similar ones have a linear relationship\n",
    "- Higher values indicate greater similarity: max = 1\n",
    "\n",
    "### Which articles are similar to 'Cristiano Ronaldo'?\n",
    "\n",
    "In the video, you learned how to use NMF features and the cosine similarity to find similar articles. Apply this to your NMF model for popular Wikipedia articles, by finding the articles most similar to the article about the footballer Cristiano Ronaldo. The NMF features you obtained earlier are available as nmf_features, while titles is a list of the article titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "# Normalize the NMF features: norm_features\n",
    "norm_features = normalize(nmf_features)\n",
    "\n",
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=titles)\n",
    "\n",
    "# Select the row corresponding to 'Cristiano Ronaldo': article\n",
    "article = df.loc['Cristiano Ronaldo']\n",
    "\n",
    "# Compute the dot products: similarities\n",
    "# Apply the .dot() method of df to article to calculate the cosine similarity of every row with article\n",
    "similarities = df.dot(article)\n",
    "\n",
    "# Display those with the largest cosine similarity\n",
    "print(similarities.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommend musical artists\n",
    "\n",
    "In this exercise and the next, you'll use what you've learned about NMF to recommend popular music artists! You are given a sparse array artists whose rows correspond to artists and whose column correspond to users. The entries give the number of times each artist was listened to by each user.\n",
    "\n",
    "In this exercise, build a pipeline and transform the array into normalized NMF features. The first step in the pipeline, MaxAbsScaler, transforms the data so that all users have the same influence on the model, regardless of how many different artists they've listened to. In the next exercise, you'll use the resulting normalized NMF features for recommendation!\n",
    "\n",
    "Suppose you were a big fan of Bruce Springsteen - which other musicial artists might you like? Use your NMF features from the previous exercise and the cosine similarity to find similar musical artists. A solution to the previous exercise has been run, so norm_features is an array containing the normalized NMF features as rows. The names of the musical artists are available as the list artist_names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.decomposition import NMF\n",
    "from sklearn.preprocessing import Normalizer, MaxAbsScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "import pandas as pd\n",
    "\n",
    "# Create a MaxAbsScaler: scaler\n",
    "scaler = MaxAbsScaler()\n",
    "\n",
    "# Create an NMF model: nmf\n",
    "nmf = NMF(n_components=20)\n",
    "\n",
    "# Create a Normalizer: normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, nmf, normalizer)\n",
    "\n",
    "# Apply fit_transform to artists: norm_features\n",
    "norm_features = pipeline.fit_transform(artists)\n",
    "\n",
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=artist_names)\n",
    "\n",
    "# Select row of 'Bruce Springsteen': artist\n",
    "artist = df.loc['Bruce Springsteen']\n",
    "\n",
    "# Compute cosine similarities: similarities\n",
    "similarities = df.dot(artist)\n",
    "\n",
    "# Display those with highest cosine similarity\n",
    "print(similarities.nlargest())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}