03A_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Natural Language Processing (NLP) in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word embeddings, or word vectors, provide a way of mapping words from a vocabulary into a low-dimensional space, where words with similar meanings are close together. Let's play around with a set of pre-trained word vectors, to get used to their properties. There exist many sets of pretrained word embeddings; here, we use ConceptNet Numberbatch, which provides a relatively small download in an easy-to-work-with format (h5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download word vectors\n",
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "if not os.path.isfile('mini.h5'):\n",
    "    print(\"Downloading Conceptnet Numberbatch word embeddings...\")\n",
    "    conceptnet_url = 'http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/17.06/mini.h5'\n",
    "    urlretrieve(conceptnet_url, 'mini.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read an `h5` file, we'll need to use the `h5py` package. Below, we use the package to open the `mini.h5` file we just downloaded. We extract from the file a list of utf-8-encoded words, as well as their $300$-dimensional vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all_words dimensions: 362891\n",
      "all_embeddings dimensions: (362891, 300)\n",
      "/c/de/lande\n"
     ]
    }
   ],
   "source": [
    "# Load the file and pull out words and embeddings\n",
    "import h5py\n",
    "with h5py.File('mini.h5', 'r') as f:\n",
    "    all_words = [word.decode('utf-8') for word in f['mat']['axis1'][:]]\n",
    "    all_embeddings = f['mat']['block0_values'][:]\n",
    "    \n",
    "print(\"all_words dimensions: {0}\".format(len(all_words)))\n",
    "print(\"all_embeddings dimensions: {0}\".format(all_embeddings.shape))\n",
    "\n",
    "print(all_words[10000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, `all_words` is a list of $V$ strings (what we call our *vocabulary*), and `all_embeddings` is a $V \\times 300$ matrix. The strings are of the form `/c/language_code/word`—for example, `/c/en/cat` and `/c/es/gato`.\n",
    "\n",
    "We are interested only in the English words. We use Python list comprehensions to pull out the indices of the English words, then extract just the English words (stripping the six-character `/c/en/` prefix) and their embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all_words dimensions: 150875\n",
      "all_embeddings dimensions: (150875, 300)\n",
      "bajillion\n"
     ]
    }
   ],
   "source": [
    "# Restrict our vocabulary to just the English words\n",
    "english_words = [word[6:] for word in all_words if word.startswith('/c/en/')]\n",
    "english_word_indices = [i for i, word in enumerate(all_words) if word.startswith('/c/en/')]\n",
    "english_embeddings = all_embeddings[english_word_indices]\n",
    "\n",
    "print(\"all_words dimensions: {0}\".format(len(english_words)))\n",
    "print(\"all_embeddings dimensions: {0}\".format(english_embeddings.shape))\n",
    "\n",
    "print(english_words[10000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to look up words easily, so we create a dictionary that maps us from a word to its index in the word embeddings matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = {word: i for i, word in enumerate(english_words)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each word has an associated 300-dimensional embedding, which we can directly show by calling the word embeddings with that dictionary word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat has word index 21398\n",
      "cat has word embedding [  0   0   2  -5   1   0  -1  -3   3  -2   2   1   4   0  -3   0  -1  -4\n",
      "   0   1   0  -2   0  -2  -2   3   1  -2  -2   3  -2   1   0   0  -3   1\n",
      "  -4   6  -1   2  -4   4  -3   0  -2  -3  -1  -5   5  -6   2   2  -3   0\n",
      "  -1   4  -1   2   2   1   3   1   3  -1  -8  -3   2   0   1   4  -1  -5\n",
      "   2   0  -5   1   0  -7  -4   2   3   4   3   3  -4   8   0   4  -2   0\n",
      "   3   1  -4   0   3   0  -2   0  -1  -8   0   0   2   0   2   4   7   3\n",
      "  -2   3  -3   3   2  -3  -5   0  -2   2   0  -3  10  -2   0  -3  -2  -5\n",
      "   4   0   0   3   2  -1  -1   5  -4  -2   0  -2  -8   0  -4   4   9  -6\n",
      "   5   4  -7  -1  -2  -3   2   4   0  -1  -6  -2  -4  -4  -2   0   4  -5\n",
      "   0   2   0   4   1   3   7  -3   4   0   6   0 -12  -4   0  -2  -6   1\n",
      "   6  -5   4   0   2  -6   2  -2   2   3  10   0   0   4   4  -2   0  -2\n",
      "   0   3   1   0  -9   3   3  -6   0   0   0  -8   1   5   2  -2   0   1\n",
      "  -3   4  -5  -5   3   1   4  -4   0   2  -2  -1  -4  -1   6  -2   3   3\n",
      "  -1  -1   2  -2   0   4   0   2   1  -2   0   1   0  -2   5  -9  -4   1\n",
      "   4   2   0   1   0  -3   1  -8   7   3  -4   3  -2   1  -1   2   0  -1\n",
      "   2   0   0  -2   2   1   4  -1   1  -2   6  -2   4  -1   0   1  -2  -1\n",
      "   2   0   3  -1   4   4   4   0   1   0  -1   0]\n"
     ]
    }
   ],
   "source": [
    "our_word='cat'\n",
    "our_word_index=index['cat']\n",
    "our_embedding=english_embeddings[our_word_index]\n",
    "print(our_word,'has word index',our_word_index)\n",
    "print(our_word, 'has word embedding',our_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The magnitude of a word vector is less important than its direction; the magnitude can be thought of as representing frequency of use, independent of the semantics of the word. \n",
    "Here, we will be interested in semantics, so we *normalize* our vectors, dividing each by its length. \n",
    "The result is that all of our word vectors are length 1, and as such, lie on a unit circle. \n",
    "The dot product of two vectors is proportional to the cosine of the angle between them, and provides a measure of similarity (the bigger the cosine, the smaller the angle).\n",
    "\n",
    "<img src=\"Figures/cosine_similarity.png\" alt=\"cosine\" style=\"width: 500px;\"/>\n",
    "<center>Figure adapted from *[Mastering Machine Learning with Spark 2.x](https://www.safaribooksonline.com/library/view/mastering-machine-learning/9781785283451/ba8bef27-953e-42a4-8180-cea152af8118.xhtml)*</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "norms = np.linalg.norm(english_embeddings, axis=1)\n",
    "normalized_embeddings = english_embeddings.astype('float32') / norms.astype('float32').reshape([-1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to measure the similarity between pairs of words. We use numpy to take dot products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "def similarity_score(w1, w2):\n",
    "    score = np.dot(normalized_embeddings[index[w1], :], normalized_embeddings[index[w2], :])\n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\tcat\t 1.0\n",
      "cat\tfeline\t 0.8199548\n",
      "cat\tdog\t 0.590724\n",
      "cat\tmoo\t 0.0039538266\n",
      "cat\tfreeze\t -0.030225186\n",
      "antonyms\topposites\t 0.3941065\n",
      "antonyms\tsynonyms\t 0.46883982\n"
     ]
    }
   ],
   "source": [
    "# A word is as similar with itself as possible:\n",
    "print('cat\\tcat\\t', similarity_score('cat', 'cat'))\n",
    "\n",
    "# Closely related words still get high scores:\n",
    "print('cat\\tfeline\\t', similarity_score('cat', 'feline'))\n",
    "print('cat\\tdog\\t', similarity_score('cat', 'dog'))\n",
    "\n",
    "# Unrelated words, not so much\n",
    "print('cat\\tmoo\\t', similarity_score('cat', 'moo'))\n",
    "print('cat\\tfreeze\\t', similarity_score('cat', 'freeze'))\n",
    "\n",
    "# Antonyms are still considered related, sometimes more so than synonyms\n",
    "print('antonyms\\topposites\\t', similarity_score('antonym', 'opposite'))\n",
    "print('antonyms\\tsynonyms\\t', similarity_score('antonym', 'synonym'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find, for instance, the most similar words to a given word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "def closest_to_vector(v, n):\n",
    "    all_scores = np.dot(normalized_embeddings, v)\n",
    "    best_words = map(lambda i: english_words[i], reversed(np.argsort(all_scores)))\n",
    "    return [next(best_words) for _ in range(n)]\n",
    "\n",
    "def most_similar(w, n):\n",
    "    return closest_to_vector(normalized_embeddings[index[w], :], n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cat', 'humane_society', 'kitten', 'feline', 'colocolo', 'cats', 'kitty', 'maine_coon', 'housecat', 'sharp_teeth']\n",
      "['dog', 'dogs', 'wire_haired_dachshund', 'doggy_paddle', 'lhasa_apso', 'good_friend', 'puppy_dog', 'bichon_frise', 'woof_woof', 'golden_retrievers']\n",
      "['duke', 'dukes', 'duchess', 'duchesses', 'ducal', 'dukedom', 'duchy', 'voivode', 'princes', 'prince']\n"
     ]
    }
   ],
   "source": [
    "print(most_similar('cat', 10))\n",
    "print(most_similar('dog', 10))\n",
    "print(most_similar('duke', 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Can you explain the following similarity scores?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8478777"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_score(\"sit\", \"sits\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.858501"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_score(\"want\", \"wants\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8664926"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_score(\"sleep\", \"sleeps\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42647985"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_score(\"leave\", \"leaves\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Can you find a polysemous word — a word with multiple meanings — so that the list of the top 10 most related words contains words that aren't themselves related to one another?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['leave', 'leaving', 'come_away', 'depart', 'beleave', 'go_forth', 'quit', 'go_away', 'vacate', 'departing']\n"
     ]
    }
   ],
   "source": [
    "print(most_similar('leave', 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['leaves', 'leaf', 'foliage', 'banana_leaf', 'leaflike', 'leaved', 'betel_leaf', 'thai_basil', 'oak_trees', 'aspidistra']\n"
     ]
    }
   ],
   "source": [
    "print(most_similar('leaves', 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['solution', 'solutions', 'aqueous_solution', 'subproblem', 'solve', 'aqueous_phase', 'virtuous_circle', 'solves', 'resolvent', 'solving']\n"
     ]
    }
   ],
   "source": [
    "print(most_similar('solution', 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `closest_to_vector` to find words \"nearby\" vectors that we create ourselves. This allows us to solve analogies. For example, in order to solve the analogy \"man : brother :: woman : ?\", we can compute a new vector `brother - man + woman`: the meaning of brother, minus the meaning of man, plus the meaning of woman. We can then ask which words are closest, in the embedding space, to that new vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def solve_analogy(a1, b1, a2):\n",
    "    b2 = normalized_embeddings[index[b1], :] - normalized_embeddings[index[a1], :] + normalized_embeddings[index[a2], :]\n",
    "    return closest_to_vector(b2, 5)\n",
    "def print_analogy(a1, b1,a2):\n",
    "    closest_words=solve_analogy(a1,b1,a2)\n",
    "    print(\"{0}:{1} as {2}:?\".format(a1,b1,a2))\n",
    "    print(\"Best guesses are: {}\".format(closest_words))\n",
    "    return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man:brother as woman:?\n",
      "Best guesses are: ['sister', 'brother', 'sisters', 'kid_sister', 'younger_brother']\n",
      "man:husband as woman:?\n",
      "Best guesses are: ['wife', 'husband', 'husbands', 'spouse', 'wifes']\n",
      "spain:madrid as france:?\n",
      "Best guesses are: ['paris', 'france', 'le_havre', 'in_france', 'montmartre']\n",
      "dog:golden_retriever as cat:?\n",
      "Best guesses are: ['cat', 'maine_coon', 'kitten', 'tabby', 'kitty']\n"
     ]
    }
   ],
   "source": [
    "print_analogy(\"man\", \"brother\", \"woman\")\n",
    "print_analogy(\"man\", \"husband\", \"woman\")\n",
    "print_analogy(\"spain\", \"madrid\", \"france\")\n",
    "print_analogy(\"dog\", \"golden_retriever\", \"cat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note that the new vector, `b2`, is not normalized. Does this matter?  Why or why not?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These three results are quite good, but in general, the results of these analogies can be disappointing. Try experimenting with other analogies, and see if you can think of ways to get around the problems you notice (i.e., modifications to the solve_analogy algorithm)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using word embeddings in deep models\n",
    "Word embeddings are fun to play around with, but their primary use is that they allow us to think of words as existing in a continuous, Euclidean space; we can then use an existing arsenal of techniques for machine learning with continuous numerical data (like logistic regression or neural networks) to process text.\n",
    "\n",
    "Let's take a look at an especially simple version of this. We'll perform *sentiment analysis* on a set of movie reviews: in particular, we will attempt to classify a movie review as positive or negative based on its text.\n",
    "\n",
    "We will use a simplified version of [Simple Word Embedding Model](http://people.ee.duke.edu/~lcarin/acl2018_swem.pdf) (SWEM, Shen et al. 2018) to do so. We will represent a review as the *mean* of the embeddings of the words in the review (SWEM would also update the word embeddings). Then we'll train a three-layer MLP (a neural network) to classify the review as positive or negative.\n",
    "\n",
    "Download the `movie-simple.txt` file from Google Classroom into this directory. Each line of that file contains \n",
    "\n",
    "1. the numeral 0 (for negative) or the numeral 1 (for positive), followed by\n",
    "2. a tab (the whitespace character), and then\n",
    "3. the review itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "remove_punct=str.maketrans('','',string.punctuation)\n",
    "\n",
    "# This function converts a line of our data file into\n",
    "# a tuple (x, y), where x is 300-dimensional representation\n",
    "# of the words in a review, and y is its label.\n",
    "def convert_line_to_example(line):\n",
    "    # Pull out the first character: that's our label (0 or 1)\n",
    "    y = int(line[0])\n",
    "    \n",
    "    # Split the line into words using Python's split() function\n",
    "    words = line[2:].translate(remove_punct).lower().split()\n",
    "    \n",
    "    # Look up the embeddings of each word, ignoring words not\n",
    "    # in our pretrained vocabulary.\n",
    "    embeddings = [normalized_embeddings[index[w]] for w in words\n",
    "                  if w in index]\n",
    "    \n",
    "    # Take the mean of the embeddings\n",
    "    x = np.mean(np.vstack(embeddings), axis=0)\n",
    "    return {'x': x, 'y': y}\n",
    "\n",
    "# Apply the function to each line in the file.\n",
    "with open(\"Resources/movie-simple.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    dataset = [convert_line_to_example(l) for l in f.readlines()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1411"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a dataset, let's shuffle it and do a train/test split. We use a quarter of the dataset for testing, 3/4 for training (but also ensure that we have a whole number of batches in our training set, to make the code nicer later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "random.shuffle(dataset)\n",
    "\n",
    "batch_size = 100\n",
    "total_batches = len(dataset) // batch_size\n",
    "train_batches = 3*total_batches // 4 \n",
    "train, test = dataset[:train_batches*batch_size], dataset[train_batches*batch_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to build our MLP in Tensorflow. We'll use placeholders for `X` and `y` as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Placeholders for input\n",
    "X = tf.placeholder(tf.float32, [None, 300])\n",
    "y = tf.placeholder(tf.float32, [None, 1])\n",
    "\n",
    "# Three-layer MLP\n",
    "h1 = tf.keras.layers.Dense(100, activation='relu')(X)\n",
    "h2 = tf.keras.layers.Dense(20, activation='relu')(h1)\n",
    "logits = tf.keras.layers.Dense(1)(h2)\n",
    "probabilities = tf.sigmoid(logits)\n",
    "\n",
    "# Loss and metrics\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.round(probabilities), y), tf.float32))\n",
    "\n",
    "# Training\n",
    "train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)\n",
    "\n",
    "# Initialization of variables\n",
    "init_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now begin a session and train our model. We'll train for 250 epochs. When we're finished, we'll evaluate our accuracy on all the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 \t Loss: 0.6889349222183228 \t Acc: 0.6200000047683716\n",
      "Epoch: 10 \t Loss: 0.6749433875083923 \t Acc: 0.5899999737739563\n",
      "Epoch: 20 \t Loss: 0.6782694458961487 \t Acc: 0.550000011920929\n",
      "Epoch: 30 \t Loss: 0.6783126592636108 \t Acc: 0.5199999809265137\n",
      "Epoch: 40 \t Loss: 0.6950281262397766 \t Acc: 0.44999998807907104\n",
      "Epoch: 50 \t Loss: 0.6358800530433655 \t Acc: 0.6399999856948853\n",
      "Epoch: 60 \t Loss: 0.6386836767196655 \t Acc: 0.6399999856948853\n",
      "Epoch: 70 \t Loss: 0.604857325553894 \t Acc: 0.6700000166893005\n",
      "Epoch: 80 \t Loss: 0.5425114631652832 \t Acc: 0.8399999737739563\n",
      "Epoch: 90 \t Loss: 0.5198315978050232 \t Acc: 0.800000011920929\n",
      "Epoch: 100 \t Loss: 0.4235253632068634 \t Acc: 0.8999999761581421\n",
      "Epoch: 110 \t Loss: 0.3741091787815094 \t Acc: 0.8999999761581421\n",
      "Epoch: 120 \t Loss: 0.29639628529548645 \t Acc: 0.9100000262260437\n",
      "Epoch: 130 \t Loss: 0.2626670300960541 \t Acc: 0.9300000071525574\n",
      "Epoch: 140 \t Loss: 0.21935081481933594 \t Acc: 0.9300000071525574\n",
      "Epoch: 150 \t Loss: 0.20941413938999176 \t Acc: 0.9200000166893005\n",
      "Epoch: 160 \t Loss: 0.11655229330062866 \t Acc: 0.9700000286102295\n",
      "Epoch: 170 \t Loss: 0.21014057099819183 \t Acc: 0.9200000166893005\n",
      "Epoch: 180 \t Loss: 0.1852555125951767 \t Acc: 0.9200000166893005\n",
      "Epoch: 190 \t Loss: 0.12112413346767426 \t Acc: 0.9800000190734863\n",
      "Epoch: 200 \t Loss: 0.17918312549591064 \t Acc: 0.9200000166893005\n",
      "Epoch: 210 \t Loss: 0.15702763199806213 \t Acc: 0.949999988079071\n",
      "Epoch: 220 \t Loss: 0.1150929257273674 \t Acc: 0.9700000286102295\n",
      "Epoch: 230 \t Loss: 0.12017910182476044 \t Acc: 0.9599999785423279\n",
      "Epoch: 240 \t Loss: 0.13769084215164185 \t Acc: 0.9200000166893005\n",
      "Final accuracy: 0.9489051103591919\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "sess = tf.Session()\n",
    "sess.run(init_op)\n",
    "\n",
    "for epoch in range(250):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = [sample['x'] for sample in data]\n",
    "        labels  = [sample['y'] for sample in data]\n",
    "        labels = np.array(labels).reshape([-1,1])\n",
    "        \n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "        \n",
    "    if epoch % 10 == 0:\n",
    "        print(\"Epoch: {0} \\t Loss: {1} \\t Acc: {2}\".format(epoch, l, acc))\n",
    "    \n",
    "    random.shuffle(train)\n",
    "        \n",
    "# Evaluate on test set\n",
    "test_reviews = [sample['x'] for sample in test]\n",
    "test_labels  = [sample['y'] for sample in test]\n",
    "test_labels  = np.array(test_labels).reshape([-1, 1])\n",
    "\n",
    "acc = sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "print(\"Final accuracy: {0}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now examine what our model has learned, seeing how it responds to word vectors for different words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "exciting [[0.9999815]]\n",
      "hated [[3.004384e-08]]\n",
      "boring [[5.114875e-06]]\n",
      "loved [[0.9999999]]\n",
      "extremely [[0.59495217]]\n",
      "rather [[0.00566179]]\n",
      "quite [[0.5447439]]\n"
     ]
    }
   ],
   "source": [
    "# Check some words\n",
    "words_to_test = [\"exciting\", \"hated\", \"boring\", \"loved\", \"extremely\", \"rather\", \"quite\"]\n",
    "\n",
    "for word in words_to_test:\n",
    "    print(word, sess.run(probabilities, feed_dict={X: normalized_embeddings[index[word]].reshape(1, 300)}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try some words of your own!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model works great for such a simple dataset, but does a little less well on something more complex. `movie-pang02.txt`, for instance, has 2000 longer, more complex movie reviews. It's in the same format as our simple dataset. On those longer reviews, this model achieves only 60-80% accuracy. (Increasing the number of epochs to, say, 1000, does help.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show this in practice, this is a condensed version of the same code above working on this second dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 Loss 0.69324297 Acc 0.47\n",
      "Epoch 10 Loss 0.6920747 Acc 0.67\n",
      "Epoch 20 Loss 0.6920169 Acc 0.54\n",
      "Epoch 30 Loss 0.6918787 Acc 0.56\n",
      "Epoch 40 Loss 0.6914005 Acc 0.57\n",
      "Epoch 50 Loss 0.6909299 Acc 0.56\n",
      "Epoch 60 Loss 0.6924898 Acc 0.48\n",
      "Epoch 70 Loss 0.6917926 Acc 0.58\n",
      "Epoch 80 Loss 0.69205946 Acc 0.53\n",
      "Epoch 90 Loss 0.6918338 Acc 0.49\n",
      "Epoch 100 Loss 0.6907446 Acc 0.53\n",
      "Epoch 110 Loss 0.69052535 Acc 0.6\n",
      "Epoch 120 Loss 0.68922544 Acc 0.65\n",
      "Epoch 130 Loss 0.688009 Acc 0.65\n",
      "Epoch 140 Loss 0.68838364 Acc 0.62\n",
      "Epoch 150 Loss 0.6877033 Acc 0.65\n",
      "Epoch 160 Loss 0.68835264 Acc 0.54\n",
      "Epoch 170 Loss 0.68397325 Acc 0.67\n",
      "Epoch 180 Loss 0.6879108 Acc 0.6\n",
      "Epoch 190 Loss 0.6832516 Acc 0.66\n",
      "Epoch 200 Loss 0.6884147 Acc 0.55\n",
      "Epoch 210 Loss 0.67951775 Acc 0.7\n",
      "Epoch 220 Loss 0.68232423 Acc 0.6\n",
      "Epoch 230 Loss 0.6805441 Acc 0.67\n",
      "Epoch 240 Loss 0.6713426 Acc 0.72\n",
      "Final accuracy: 0.596\n"
     ]
    }
   ],
   "source": [
    "# Apply the function to each line in the file.\n",
    "with open(\"Resources/movie-pang02.txt\", \"r\",encoding='utf-8') as f:\n",
    "    dataset = [convert_line_to_example(l) for l in f.readlines()]\n",
    "import random\n",
    "random.shuffle(dataset)\n",
    "batch_size = 100\n",
    "total_batches = len(dataset) // batch_size\n",
    "train_batches = 3 * total_batches // 4\n",
    "train, test = dataset[:train_batches*batch_size], dataset[train_batches*batch_size:]\n",
    "sess = tf.Session()\n",
    "initialize_all = tf.global_variables_initializer() \n",
    "sess.run(initialize_all)\n",
    "for epoch in range(250):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = [sample['x'] for sample in data]\n",
    "        labels  = [sample['y'] for sample in data]\n",
    "        labels = np.array(labels).reshape([-1, 1])\n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "    if epoch % 10 == 0:\n",
    "        print(\"Epoch\", epoch, \"Loss\", l, \"Acc\", acc)\n",
    "    random.shuffle(train)\n",
    "\n",
    "# Evaluate on test set\n",
    "test_reviews = [sample['x'] for sample in test]\n",
    "test_labels  = [sample['y'] for sample in test]\n",
    "test_labels = np.array(test_labels).reshape([-1, 1])\n",
    "acc = sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "print(\"Final accuracy:\", acc)\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will not plan to go through Recurrent Neural Networks in detail, but below is some example code on RNNs to provide some guidance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurrent Neural Networks (RNNs)\n",
    "\n",
    "In the context of deep learning, natural language is commonly modeled with Recurrent Neural Networks (RNNs).\n",
    "RNNs pass the output of a neuron back to the input of the next time step of the same neuron.\n",
    "These directed cycles in the RNN architecture gives them the ability to model temporal dynamics, making them particularly suited for modeling sequences (e.g. text).\n",
    "We can visualize an RNN layer as follows:\n",
    "\n",
    "<img src=\"Figures/basic_RNN.PNG\" alt=\"basic_RNN\" style=\"width: 80px;\"/>\n",
    "<center>Figure from *Understanding LSTMs*. https://colah.github.io/posts/2015-08-Understanding-LSTMs/</center>\n",
    "\n",
    "We can unroll an RNN through time, making the sequence aspect of them more obvious:\n",
    "\n",
    "<img src=\"Figures/unrolled_RNN.PNG\" alt=\"basic_RNN\" style=\"width: 400px;\"/>\n",
    "<center>Figure from *Understanding LSTMs*. https://colah.github.io/posts/2015-08-Understanding-LSTMs/</center>\n",
    "\n",
    "#### RNNs in TensorFlow\n",
    "How would we implement an RNN in TensorFlow? Given the different forms of RNNs, there are quite a few ways, but we'll stick to a simple one. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are dealing with a Recurrent Neural Network, we can have each word be a separate input to the network.  Given our word embeddings, that will be given by a matrix. The preprocessing pipeline will be slightly different than it was before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function converts a line of our data file into\n",
    "# a tuple (x, y, w), where x is 300-dimensional representation\n",
    "# of the words in a review,  y is its label, and w is the concatenated word embeddings\n",
    "def convert_line_to_example_rnn(line):\n",
    "    # Pull out the first character: that's our label (0 or 1)\n",
    "    y = int(line[0])\n",
    "    # Split the line into words using Python's split() function\n",
    "    words = line[2:].translate(remove_punct).lower().split()\n",
    "    # Look up the embeddings of each word, ignoring words not\n",
    "    # in our pretrained vocabulary.\n",
    "    embeddings = [normalized_embeddings[index[w]] for w in words\n",
    "                  if w in index]\n",
    "    # Take the mean of the embeddings\n",
    "    x = np.mean(np.vstack(embeddings), axis=0)\n",
    "    return {'x': x, 'y': y, 'w':embeddings, 'text':line[2:]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"Resources/movie-simple.txt\", \"r\",encoding='utf-8') as f:\n",
    "    dataset = [convert_line_to_example_rnn(l) for l in f.readlines()]\n",
    "import random\n",
    "random.seed(200)\n",
    "random.shuffle(dataset)\n",
    "batch_size = 1\n",
    "total_batches = len(dataset) // batch_size\n",
    "train_batches = 3 * total_batches // 4\n",
    "train, test = dataset[:train_batches*batch_size], dataset[train_batches*batch_size:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The DaVinci Code and Mission Impossible 3 are AWESOME.\n",
      "\n",
      "1\n",
      "(300,)\n",
      "(1, 9, 300)\n"
     ]
    }
   ],
   "source": [
    "print(train[0]['text'])\n",
    "print(train[0]['y'])\n",
    "print(train[0]['x'].shape)\n",
    "print(np.array(train[0]['w']).reshape([1,-1,300]).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying an RNN to the text reviews, starting with the easier data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "# sizes\n",
    "n_steps = None\n",
    "n_inputs = 300\n",
    "n_neurons = 5\n",
    "# Build RNN\n",
    "X= tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y= tf.placeholder(tf.float32, [None, 1])\n",
    "basic_cell = tf.contrib.rnn.BasicRNNCell(n_neurons,activation=tf.nn.tanh)\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)\n",
    "last_cell_output=outputs[:,-1,:]\n",
    "y_=tf.layers.dense(last_cell_output,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loss and metrics\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=y_, labels=y))\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.round(tf.sigmoid(y_)), y), tf.float32))\n",
    "\n",
    "# Training\n",
    "train_step = tf.train.AdamOptimizer(0.001).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 499 Loss 0.6888297 Acc 1.0\n",
      "batch 999 Loss 0.15129247 Acc 1.0\n",
      "Epoch 0 Loss 1.8025193 Acc 0.0\n",
      "batch 499 Loss 0.11805359 Acc 1.0\n",
      "batch 999 Loss 0.057079393 Acc 1.0\n",
      "Epoch 1 Loss 0.08673588 Acc 1.0\n",
      "batch 499 Loss 0.029195866 Acc 1.0\n",
      "batch 999 Loss 0.090121076 Acc 1.0\n",
      "Epoch 2 Loss 0.28285712 Acc 1.0\n",
      "batch 499 Loss 0.011758866 Acc 1.0\n",
      "batch 999 Loss 0.11506314 Acc 1.0\n",
      "Epoch 3 Loss 1.4820299 Acc 0.0\n",
      "batch 499 Loss 0.00388214 Acc 1.0\n",
      "batch 999 Loss 0.015506666 Acc 1.0\n",
      "Epoch 4 Loss 0.0733734 Acc 1.0\n",
      "batch 499 Loss 0.029356554 Acc 1.0\n",
      "batch 999 Loss 0.047062576 Acc 1.0\n",
      "Epoch 5 Loss 0.011984888 Acc 1.0\n",
      "batch 499 Loss 0.12030737 Acc 1.0\n",
      "batch 999 Loss 0.16438444 Acc 1.0\n",
      "Epoch 6 Loss 0.01008274 Acc 1.0\n",
      "batch 499 Loss 0.008522104 Acc 1.0\n",
      "batch 999 Loss 2.229559 Acc 0.0\n",
      "Epoch 7 Loss 0.005230967 Acc 1.0\n",
      "batch 499 Loss 0.13257532 Acc 1.0\n",
      "batch 999 Loss 0.0014965261 Acc 1.0\n",
      "Epoch 8 Loss 0.0025378007 Acc 1.0\n",
      "batch 499 Loss 0.004843535 Acc 1.0\n",
      "batch 999 Loss 0.93674695 Acc 0.0\n",
      "Epoch 9 Loss 0.005982762 Acc 1.0\n"
     ]
    }
   ],
   "source": [
    "initialize_all = tf.global_variables_initializer()\n",
    "sess = tf.InteractiveSession()\n",
    "sess.run(initialize_all)\n",
    "for epoch in range(10):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = np.array([sample['w'] for sample in data]).reshape([batch_size,-1,300])\n",
    "        labels  = np.array([sample['y'] for sample in data]).reshape([batch_size,1])\n",
    "        labels = np.array(labels).reshape([-1, 1])\n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "        if (batch+1) % 500 == 0:\n",
    "            print(\"batch\", batch, \"Loss\", l, \"Acc\", acc)\n",
    "    if epoch % 1 == 0:\n",
    "        print(\"Epoch\", epoch, \"Loss\", l, \"Acc\", acc)\n",
    "    random.shuffle(train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final accuracy: 0.9773371104815864\n"
     ]
    }
   ],
   "source": [
    "# Evaluate on test set\n",
    "test_acc=0\n",
    "n=0\n",
    "for sample in test:\n",
    "    test_reviews = np.array([sample['w'] ]).reshape([1,-1,300])\n",
    "    test_labels  = np.array([sample['y']]).reshape([1,1])\n",
    "    test_labels = np.array(test_labels).reshape([-1, 1])\n",
    "    test_acc += sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "    n+=1\n",
    "acc=test_acc/n \n",
    "print(\"Final accuracy:\", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Switching to an LSTM-based neural network is fairly easy in code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "# sizes\n",
    "n_steps = None\n",
    "n_inputs = 300\n",
    "n_neurons = 5\n",
    "# Build RNN\n",
    "X= tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y= tf.placeholder(tf.float32, [None, 1])\n",
    "basic_cell = tf.contrib.rnn.LSTMCell(n_neurons,activation=tf.nn.tanh)\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)\n",
    "last_cell_output=outputs[:,-1,:]\n",
    "y_=tf.layers.dense(last_cell_output,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loss and metrics\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=y_, labels=y))\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.round(tf.sigmoid(y_)), y), tf.float32))\n",
    "\n",
    "# Training\n",
    "train_step = tf.train.AdamOptimizer(0.0005).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 499 Loss 0.48607454 Acc 1.0\n",
      "batch 999 Loss 0.22863635 Acc 1.0\n",
      "Epoch 0 Loss 0.25951394 Acc 1.0\n",
      "batch 499 Loss 0.101040214 Acc 1.0\n",
      "batch 999 Loss 0.05657885 Acc 1.0\n",
      "Epoch 1 Loss 0.05204625 Acc 1.0\n",
      "batch 499 Loss 0.09560171 Acc 1.0\n",
      "batch 999 Loss 0.071655296 Acc 1.0\n",
      "Epoch 2 Loss 0.059827775 Acc 1.0\n",
      "batch 499 Loss 0.058376465 Acc 1.0\n",
      "batch 999 Loss 0.055644244 Acc 1.0\n",
      "Epoch 3 Loss 0.16751455 Acc 1.0\n",
      "batch 499 Loss 0.11454698 Acc 1.0\n",
      "batch 999 Loss 0.013678055 Acc 1.0\n",
      "Epoch 4 Loss 0.0361196 Acc 1.0\n",
      "batch 499 Loss 0.4248932 Acc 1.0\n",
      "batch 999 Loss 0.020090805 Acc 1.0\n",
      "Epoch 5 Loss 0.0067319837 Acc 1.0\n",
      "batch 499 Loss 0.024903579 Acc 1.0\n",
      "batch 999 Loss 0.0086562745 Acc 1.0\n",
      "Epoch 6 Loss 0.00056374667 Acc 1.0\n",
      "batch 499 Loss 0.040345907 Acc 1.0\n",
      "batch 999 Loss 0.005661821 Acc 1.0\n",
      "Epoch 7 Loss 0.029291177 Acc 1.0\n",
      "batch 499 Loss 0.008131191 Acc 1.0\n",
      "batch 999 Loss 0.0034976238 Acc 1.0\n",
      "Epoch 8 Loss 0.0014228114 Acc 1.0\n",
      "batch 499 Loss 0.017069545 Acc 1.0\n",
      "batch 999 Loss 0.0028547666 Acc 1.0\n",
      "Epoch 9 Loss 0.0045974283 Acc 1.0\n"
     ]
    }
   ],
   "source": [
    "initialize_all = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(initialize_all)\n",
    "for epoch in range(10):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = np.array([sample['w'] for sample in data]).reshape([batch_size,-1,300])\n",
    "        labels  = np.array([sample['y'] for sample in data]).reshape([batch_size,1])\n",
    "        labels = np.array(labels).reshape([-1, 1])\n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "        if (batch+1) % 500 == 0:\n",
    "            print(\"batch\", batch, \"Loss\", l, \"Acc\", acc)\n",
    "    if epoch % 1 == 0:\n",
    "        print(\"Epoch\", epoch, \"Loss\", l, \"Acc\", acc)\n",
    "    random.shuffle(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final accuracy: 0.9858356940509915\n"
     ]
    }
   ],
   "source": [
    "# Evaluate on test set\n",
    "test_acc=0\n",
    "n=0\n",
    "for sample in test:\n",
    "    test_reviews = np.array([sample['w'] ]).reshape([1,-1,300])\n",
    "    test_labels  = np.array([sample['y']]).reshape([1,1])\n",
    "    test_labels = np.array(test_labels).reshape([-1, 1])\n",
    "    test_acc += sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "    n+=1\n",
    "acc=test_acc/n \n",
    "print(\"Final accuracy:\", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, swapping out to a more complex dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"Resources/movie-pang02.txt\", \"r\") as f:\n",
    "    dataset = [convert_line_to_example_rnn(l) for l in f.readlines()]\n",
    "import random\n",
    "random.seed(42)\n",
    "random.shuffle(dataset)\n",
    "batch_size = 1\n",
    "total_batches = len(dataset) // batch_size\n",
    "train_batches = 3 * total_batches // 4\n",
    "train, test = dataset[:train_batches*batch_size], dataset[train_batches*batch_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By examining one of the reviews, we can see that these are much more complex reviews, and requiring greater nuance to analyze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " the central focus of michael winterbottom s   welcome to sarajevo   is sarajevo itself   the city under siege   and its different effect on the characters unfortunate enough to be stuck there    it proves the backdrop for a stunningly realized story which refreshingly strays from mythic portents     platoon       racial tumultuosness   the risible   the walking dead     or a tinge of schmaltziness     schindler s list        the two leads   stephen dillane as a reporter and emira nusevic as an orphan with a plight few can identify with   are extremely believable   not one moment with them involved rings false    the question is not what went right    the question is what went wrong    for one   the film fails to provide a political overview of the war as it progresses   the dillane characters reports an american plane departing from sarajevo as it departs   and that s about it        the assortment of high profile supporting actors   ranging from woody harrelson as a yankee reporter   into   liquor and cigarrettes to marisa tomei as a huggable children s aid or somesuch are incapable of rising above the sketchiness of their characters   albeit they strive    the interrupted use of authentic war footage somewhat hampers the rest of the film   it makes the fictional characters seem powerless by comparison    still   winterbottom eschews mawkishness through flashy   frantic editing and imaginative use of music    and it s a plus   because he doesn t toy with our emotions with sentimental blandness    he wants us to know that in war   no one is victorious    the sweet hereafter  starring ian holm   sarah polley   bruce greenwood   tom mccamus   gabrielle rose   arsinee khanjian   alberta watson   maury chaykin   caerthan banks    produced by atom egoyan and camelia frieberg    script by atom egoyan   based on the novel by russell banks      directed by atom egoyan    running time   110 mins    rated r                                                               atom egoyan s powerfully meditative   the sweet hereafter   is as anything as haunting and transcendental as i ve seen this year    it not only explores the aftermaths of a terrible tragedy with magnificent subtlety   with cold   and stunning shots of the backdrop of this mistfortune but also by telegraphing the sense of devastation that has permeated the small town with an enormous amount of dignity and respect for each and every soul affected    ian holm as the somber   lonely lawyer seeking compensation for the townspeople   has a decency and a restraint uncommon with lawyers   he s a lion for the wrong reasons   the pain for having lost his daughter   she s a druggie who frequently calls him begging for money   is reflected on the town    and in some mesmerizing flashbacks   we witness how she lost her purity    it s one of the most strikingly breathtaking takes on the loss of innocence i ve ever seen    but the most astounding part goes to sarah polly   the peaceful   benevolent girl who   in contrast to the other characters   remains the most mentally stable during past   present or future    her ability to convey concealed pain and unconditional love is the opposite   yet near revolutionary role that won emily watson an oscar nomination for   breaking the waves      both women are torn apart by an extraordinary incident but it is only then that we truly see   truly feel   their wordless   omnipotent love       the sweet hereafter   though   is atom egoyan s movie and as of now   he is emerging as the definite face of independent cinema    after surging to worldwide recognition with 1994 s   exotica       the sweet hereafter   makes him something few people in this industry can call themselves   an artist    he interweaves time with a delicacy and sees a soul with a purity that is just not common    \n",
      "\n",
      "1\n",
      "(300,)\n",
      "(1, 577, 300)\n"
     ]
    }
   ],
   "source": [
    "print(train[0]['text'])\n",
    "print(train[0]['y'])\n",
    "print(train[0]['x'].shape)\n",
    "print(np.array(train[0]['w']).reshape([1,-1,300]).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1500"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can retrain on the same LSTM that we were using before. Note that this is _much_ slower because the RNN now has to deal with much longer sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 499 Loss 0.63374853 Acc 1.0\n",
      "batch 999 Loss 0.65800625 Acc 1.0\n",
      "batch 1499 Loss 0.6028506 Acc 1.0\n",
      "Epoch 0 Loss 0.6028506 Acc 1.0\n",
      "batch 499 Loss 0.6138419 Acc 1.0\n",
      "batch 999 Loss 0.9523419 Acc 0.0\n",
      "batch 1499 Loss 0.8965019 Acc 0.0\n",
      "Epoch 1 Loss 0.8965019 Acc 0.0\n",
      "batch 499 Loss 0.6895683 Acc 1.0\n",
      "batch 999 Loss 0.3232401 Acc 1.0\n",
      "batch 1499 Loss 0.5422541 Acc 1.0\n",
      "Epoch 2 Loss 0.5422541 Acc 1.0\n",
      "batch 499 Loss 0.19960187 Acc 1.0\n",
      "batch 999 Loss 1.1023433 Acc 0.0\n",
      "batch 1499 Loss 0.4265583 Acc 1.0\n",
      "Epoch 3 Loss 0.4265583 Acc 1.0\n",
      "batch 499 Loss 0.8223858 Acc 0.0\n",
      "batch 999 Loss 0.30732954 Acc 1.0\n",
      "batch 1499 Loss 0.30599624 Acc 1.0\n",
      "Epoch 4 Loss 0.30599624 Acc 1.0\n",
      "batch 499 Loss 0.9944975 Acc 0.0\n",
      "batch 999 Loss 0.92760015 Acc 0.0\n",
      "batch 1499 Loss 1.3472576 Acc 0.0\n",
      "Epoch 5 Loss 1.3472576 Acc 0.0\n",
      "batch 499 Loss 0.25653654 Acc 1.0\n",
      "batch 999 Loss 0.28753564 Acc 1.0\n",
      "batch 1499 Loss 0.21350908 Acc 1.0\n",
      "Epoch 6 Loss 0.21350908 Acc 1.0\n",
      "batch 499 Loss 1.2260671 Acc 0.0\n",
      "batch 999 Loss 0.9523636 Acc 0.0\n",
      "batch 1499 Loss 0.85805506 Acc 0.0\n",
      "Epoch 7 Loss 0.85805506 Acc 0.0\n",
      "batch 499 Loss 0.38757777 Acc 1.0\n",
      "batch 999 Loss 0.9901401 Acc 0.0\n",
      "batch 1499 Loss 0.17422372 Acc 1.0\n",
      "Epoch 8 Loss 0.17422372 Acc 1.0\n",
      "batch 499 Loss 1.2793434 Acc 0.0\n",
      "batch 999 Loss 0.9387357 Acc 0.0\n",
      "batch 1499 Loss 0.34997463 Acc 1.0\n",
      "Epoch 9 Loss 0.34997463 Acc 1.0\n"
     ]
    }
   ],
   "source": [
    "initialize_all = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(initialize_all)\n",
    "for epoch in range(10):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = np.array([sample['w'] for sample in data]).reshape([batch_size,-1,300])\n",
    "        labels  = np.array([sample['y'] for sample in data]).reshape([batch_size,1])\n",
    "        labels = np.array(labels).reshape([-1, 1])\n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "        if (batch+1) % 500 == 0:\n",
    "            print(\"batch\", batch, \"Loss\", l, \"Acc\", acc)\n",
    "    if epoch % 1 == 0:\n",
    "        print(\"Epoch\", epoch, \"Loss\", l, \"Acc\", acc)\n",
    "    random.shuffle(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final accuracy: 0.788\n"
     ]
    }
   ],
   "source": [
    "# Evaluate on test set\n",
    "test_acc=0\n",
    "n=0\n",
    "for sample in test:\n",
    "    test_reviews = np.array([sample['w'] ]).reshape([1,-1,300])\n",
    "    test_labels  = np.array([sample['y']]).reshape([1,1])\n",
    "    test_labels = np.array(test_labels).reshape([-1, 1])\n",
    "    test_acc += sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "    n+=1\n",
    "acc=test_acc/n \n",
    "print(\"Final accuracy:\", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This final accuracy is not great, but it is _much higher_ than what we got with the MLP approach.  Playing around with the settings can properly improve this more (this really isn't that many epochs)."
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}