notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/microsoft_logo.png\" width=\"30%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning for Language\n",
    "\n",
    "**Presented by Adam Atkinson, SDE @ [Microsoft Research Montréal](https://www.microsoft.com/en-us/research/lab/microsoft-research-montreal/)** (formerly Maluuba)\n",
    "\n",
    "**Prepared for the [AI For Social Good Summer Lab, 2018](https://www.aiforsocialgood.ca/) run by OSMO, MILA, the McGill Reasoning and Learning Lab**\n",
    "\n",
    "**Code and notebook are on GitHub: https://github.com/Maluuba/sentiment-analysis-workshop**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal: \n",
    "\n",
    "**To familiarize you with the tools, techniques, and lingo when it comes to applying deep learning to language tasks.**\n",
    "\n",
    "## Prerequisites:\n",
    "\n",
    "- Machine learning 101: probabilities, vector & matrix calculus, training & inference, discriminative & generative models (supervised vs. unsupervised learning), evaluation setup.\n",
    "\n",
    "- Neural networks & deep learning 101: activation functions, loss functions, gradients, backpropagation, (stochastic) gradient descent, multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs).\n",
    "\n",
    "- Knowledge of Python and TensorFlow for these code samples.\n",
    "\n",
    "**Don't be intimidated! You don't need to be a neural engineer or have a PhD. If these topics are new or unfamiliar there's lots of great resources in print and online!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Social Good Applications\n",
    "\n",
    "Humanity is generating an increasingly enormous amount of unstructured text in the form of language. What does this text tell us about ourselves and how does this understanding help us address real problems?\n",
    "\n",
    "### Examples\n",
    "\n",
    "- Identifying health concerns and responding to crises.\n",
    "    - There's work done assessing mental health and for [suicide prevention](https://onlinelibrary.wiley.com/doi/abs/10.1111/sltb.12312).\n",
    "    \n",
    "    \n",
    "- Filtering toxic content.\n",
    "    - [Classifying hatespeech](http://www.aclweb.org/anthology/W17-1101).\n",
    "\n",
    "\n",
    "- Identifying fake news.\n",
    "    - See the [\"Liar, Liar, Pants on Fire\"](https://arxiv.org/abs/1705.00648) dataset and the paper's citations.\n",
    "    \n",
    "\n",
    "- Validating facts.\n",
    "    - See the [First Workshop on Information Extraction and Verification](http://fever.ai/).\n",
    "\n",
    "\n",
    "- Studying social biases with text analysis.\n",
    "    - See the excellent paper [\"Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings\"](https://arxiv.org/abs/1607.06520).\n",
    "\n",
    "<img src=\"assets/biased_word_embeddings.png\" width=\"100%\"/>\n",
    "\n",
    "- x-axis is the word embedding projected onto the vector difference between \"she\" and \"he\". This shows which gender the word is more often associated with, i.e. bias/skewness.\n",
    "- y-axis is the projection onto the gender neutralness component of the word embedding. Above the line means gender neutral.\n",
    "- **We need to move the gender neutral words (above the horizontal line) to be equidistant from either gender (to the vertical line).**  \n",
    "\n",
    "[Source: shameless Microsoft Research plug.](https://arxiv.org/abs/1607.06520)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tasks\n",
    "\n",
    "**Natural Language Processing (NLP)** encompasses tasks related to:\n",
    "1. **speech recognition** (mainly refining predictions from **automatic speech recognition (ASR)** models).\n",
    "1. the **syntax** of language.\n",
    "1. the **semantics** of language.\n",
    "1. language **generation**.\n",
    "1. analyzing **discourse**.\n",
    "\n",
    "NLP/NLU focuses more on language itself rather than extracting information - **information retrieval (IR)** - but the two fields intersect.\n",
    "\n",
    "Generally we want to _understand_ language by labelling it or generating new text.\n",
    "\n",
    "### Text classification\n",
    "\n",
    "1. Label a section of text **(N -> 1)**.\n",
    "1. Label each word or token, i.e. sequence labelling **(N -> N)**.\n",
    "\n",
    "E.g.\n",
    "- Sentiment analysis.\n",
    "- Classifying the intent of a sentence (slot filling).\n",
    "- Extracting entities.\n",
    "- Part-of-speech tagging.\n",
    "- Coreference resolution.\n",
    "- Logical entailment.\n",
    "- Extractive summarization.\n",
    "\n",
    "### Text generation:\n",
    "\n",
    "1. Consume some text and produce more, lengths don't need to match **(N -> M)**.\n",
    "\n",
    "E.g.\n",
    "- Machine translation.\n",
    "- Generating natural language answers to questions.\n",
    "- Image captioning.\n",
    "- Language modelling.\n",
    "- Producing responses in dialogue.\n",
    "- Abstractive summarization.\n",
    "\n",
    "\\* **Caveat**: natural language generation is challenging, especially for neural networks or without the use of templates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Deep Learning?\n",
    "\n",
    "**Natural language:**\n",
    "\n",
    "- has a high dimensional, sparse feature space because of large vocabularies, and this space becomes combinatorially large as you create features composed of multiple tokens.\n",
    "\n",
    "- has many nuanced rules, ambiguities, and exceptions that can't easily be captured by rule-based systems.\n",
    "\n",
    "- depends on order and context.\n",
    "\n",
    "- is really noisy!\n",
    "\n",
    "**Deep neural networks:**\n",
    "\n",
    "- learn dense representations in high dimensional feature spaces.\n",
    "\n",
    "- learn representations that capture complex relationships in the data.\n",
    "\n",
    "- learn compositional and contextual relationships.\n",
    "\n",
    "- perform well on large amounts of data, generalize well, and handle noise.\n",
    "\n",
    "Data for neural networks boils down to small normalized/standardized floating point values. This means you can combine text representations with representations for other types of data (e.g. images, audio, video) to create **multi-modal** models.\n",
    "\n",
    "There's lots of language data out there so robust representations can be learned, borrowed, and easily be applied to other tasks using **transfer learning** or **fine tuning**. This makes it easy to bootstrap your own models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features and Representations\n",
    "\n",
    "### Preprocessing\n",
    "\n",
    "Raw text is messy.\n",
    "\n",
    "- Fold case and accents.\n",
    "- Strip or compress whitespace.\n",
    "- **Tokenize** the text.\n",
    "- **Stem** or **lemmatize** tokens to remove affixes.\n",
    "- Heurisitic or regular expression substitution or deletion of tokens/characters. E.g. unicode empty spaces and emojis 😰.\n",
    "- Collect the **vocabulary** of tokens.\n",
    "- Add special tokens like sentence markers (**beginning/end of sentence** BOS/EOS) and **out-of-vocabulary** (OOV or UNK).\n",
    "\n",
    "### Feature Engineering\n",
    "\n",
    "- Calculate frequency based features likening text to a **bag-of-words**.\n",
    "    - e.g **term frequency inverse document frequency**, tf-idf\n",
    "    \n",
    "    $tfidf(t,d,D) = tf(t,d)*idf(t,D)$\n",
    "    \n",
    "    where $t$ = term, $d$ = a specific document, $D$ = the corpus of all documents.\n",
    "    \n",
    "    \n",
    "- Boolean or **one-hot vector** encoded flags for tokens or features.\n",
    "    - a one-hot encoding vector $V$ has $V[index] = 1$ where $index$ is the index of the word in the vocabulary, and $V[...] = 0$ otherwise\n",
    "    \n",
    "    \n",
    "- Compute features for n-length windows of tokens, called **n-grams**.\n",
    "    - E.g. \"artificial intelligence for social good\" has trigrams:\n",
    "        - (\"artificial\",\"intelligence\",\"for\")\n",
    "        - (\"intelligence\",\"for\",\"social\")\n",
    "        - (\"for\",\"social\",\"good\")\n",
    "\n",
    "\n",
    "- Fill a term by document frequency matrix where $M_{term,document}=f_{term,document}$ and apply dimensionality reduction or matrix factorization techniques to extract features. E.g. **singular value decomposition**.\n",
    "\n",
    "### Vector Representations\n",
    "\n",
    "Neural networks take **tensors** (N dimensional matrices) as input so we need to represent discrete tokens in this format. \n",
    "\n",
    "We can use a one-hot vector, but with a million or billion-word vocabulary we're going to run out of memory. Also the signals are sparse for the network. Besides, why do we need to hand-engineer special features?\n",
    "\n",
    "Instead, we want a denser vector with a lower fixed dimension, independent of the vocabulary size. How? **Learn it!**\n",
    "\n",
    "The idea here is to train a neural network to map focused keywords to their surround context words in a section of text, then use the neural net's weights as the word representations (**word vectors**).\n",
    "\n",
    "Think of the learned word vectors as columns in a matrix $W = [w_1, ... , w_n]$. To encode a word with a one-hot vector $x$ we multiply it by $W$, to get its embedding $y = Wx$. The neural network learns this weight matrix.\n",
    "\n",
    "<img src=\"assets/word_embeddings.png\" width=\"60%\"/>\n",
    "\n",
    "[Source](https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/)\n",
    "\n",
    "#### Methods\n",
    "\n",
    "**[Good] Continuous Bag Of Words (CBOW)**: Given the _context_, predict the _keyword_.\n",
    "\n",
    "**[Better] Skipgram**: Given the _keyword_ predict the _context_. E.g. **word2vec**.\n",
    "\n",
    "<img src=\"assets/cbow_skipgram.png\" width=\"67%\"/>\n",
    "\n",
    "[Source](http://rohanvarma.me/Word2Vec/)\n",
    "\n",
    "**[Best] Global Vectors for Word Embeddings [(GloVe)](https://nlp.stanford.edu/projects/glove/)**: Learn word vectors for terms such that their dot product is equal to their probability of co-occurrence. There's a good blog post [here](https://blog.acolyer.org/2016/04/22/glove-global-vectors-for-word-representation/).\n",
    "\n",
    "These learned representations capture semantic relationships, so words can be manipulated semantically using mathematical operators. These representations are learned in an **unsupervised** manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "Language is sequential and has contextual dependencies. Additionally linguistic features appear at different  offsets in the sequence so we need models that can be agnostic to the precise starting position of a feature. **Convolutional** and **recurrent** neural networks can do both of these.\n",
    "\n",
    "### Input\n",
    "\n",
    "These are the word vectors or standardized hand-crafted numerical features. \n",
    "\n",
    "Word vectors can also be loaded as rows in a weight matrix so sparse or one-hot token representations can be fed directly into the model. This is called an **embedding layer** and the embeddings can be tuned as weights of the network by setting them to be **trainable**.\n",
    "\n",
    "### CNNs\n",
    "\n",
    "Apply a mathematical transformation (i.e. a **convolution**) over all inputs and weights in a patch of a volume. These patches are computed over windows of the input, producing local features. These convolutional layers are followed by **max pooling** to be less sensitive to the precise location of features.\n",
    "\n",
    "See [Stanford's CS231n](http://cs231n.github.io/convolutional-networks/) for a good introduction.\n",
    "\n",
    "#### For Text\n",
    "\n",
    "We can treat a sequence of tokens like a one dimensional image where the channels are the components of the token representations (e.g. word vectors). The **filter size** and **stride** of the convolution determine the n-grams for which features are learned.\n",
    "\n",
    "[Denny Britz's blog](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/) explains this well.\n",
    "\n",
    "Convolutions are also highly parallelizable which makes CNNs faster to train on GPUs.\n",
    "\n",
    "<img src=\"assets/cnn_lang.png\" width=\"90%\"/>\n",
    "\n",
    "[Source](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)\n",
    "\n",
    "### RNNs\n",
    "\n",
    "Learn a representation at each timestep of a sequence using the input at time $t$ and the output and hidden states from the previous timestep $t-1$ (hence recurrence).\n",
    "\n",
    "Think of them as long or very deep neural nets. Their sequential nature makes them well-suited for text. **GRU** and **LSTM** are the most popular flavours of RNN units.\n",
    "\n",
    "See [Chris Olah's](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) and [Andrej Karpathy's](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) blogs.\n",
    "\n",
    "#### For Text\n",
    "\n",
    "Language tasks often use **bidirectional** RNNs to capture forward and backward dependencies in the data. A bidirectional RNN (or BidiRNN, BiRNN) has an RNN operating forward on the sequence $t_1$ -> $t_n$ and another independent RNN operating backward on the sequence $t_n$ -> $t_1$. The hidden states and outputs at each time step are the concatenation of those produced by the forward and backward RNNs.\n",
    "\n",
    "RNN layers are stacked by treating each RNN layer as an encoder, i.e. all outputs/hidden states for layer $L$ are computed and then passed as input to the next RNN layer $L+1$.\n",
    "\n",
    "<img src=\"assets/birnn.png\" width=\"70%\"/>\n",
    "\n",
    "[Source](http://www.cl.cam.ac.uk/~pv273/slides/LSTMslides.pdf)\n",
    "\n",
    "The final output and hidden state of an RNN is an encoding that captures information for the whole sequence.\n",
    "\n",
    "### Encoder-Decoder\n",
    "\n",
    "Neural network models for language generally follow an encoder-decoder recipe. \n",
    "\n",
    "- An **encoder network** reduces the source sequence to a tensor representation.\n",
    "- A **decoder network** expands the encoded representation to match the target sequence.\n",
    "\n",
    "The intermediate encoded representation can be fixed or variable length and can be used as a feature to decoders in other tasks.\n",
    "\n",
    "<img src=\"assets/encdec.jpg\" width=\"70%\"/>\n",
    "\n",
    "[Source](https://talbaumel.github.io/blog/attention/)\n",
    "\n",
    "### Output\n",
    "\n",
    "The encoded representation of the whole sequence is often fed into a **dense** or **fully connected** layer followed by a softmax. This produces an output probability distribution, the argmax of which is the correct class or token. This techique can also be applied independently at each time step to produce an output for each element of the sequence.\n",
    "\n",
    "In the case of text generation, the probability distribution has as many buckets as the vocabulary has tokens, so we often have to limit it to the top $K$ most significant components to make the softmax tractable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Neural Tools for Language\n",
    "\n",
    "### Attention\n",
    "\n",
    "We learn the weighted contribution of the surrounding tokens (the context) thereby learning to pay **attention** to specific words. Attention is usually applied in the decoder.\n",
    "\n",
    "Chris Olah has a great [blog](https://distill.pub/2016/augmented-rnns/) explaining this too.\n",
    "\n",
    "<img src=\"assets/attention.png\" width=\"80%\"/>\n",
    "\n",
    "[Source](https://distill.pub/2016/augmented-rnns/)\n",
    "\n",
    "### Beam Search\n",
    "\n",
    "In order to compute a likely output sequence tractably, we compute $K$ paths through the output distributions. \n",
    "For each tail node at time $t$ in the $K$ paths, we select the most likely node at time $t+1$ to add to the path. Here $K$ is our **beam width**.\n",
    "\n",
    "<img src=\"assets/beam_search.jpg\" width=\"90%\"/>\n",
    "\n",
    "[Source](https://talbaumel.github.io/blog/attention/)\n",
    "\n",
    "### Character-Level Embeddings\n",
    "\n",
    "We can model text as a sequence of characters instead of word tokens. We can learn character embeddings and use these as features instead. This eliminates out-of-vocabulary tokens and shrinks the output space, but requires lots of data to train and bigger networks to handle longer sequences.\n",
    "\n",
    "### Hierarchical Softmax\n",
    "\n",
    "We can avoid the big-softmax problem by modelling a component of the output probability distribution as the product of node probabilities in a path of a binary tree. This reduces the complexity to at most the depth of the tree $log_{2}(|vocabulary|)$. There's a great blog post [here](http://ruder.io/word-embeddings-softmax/index.html#hierarchicalsoftmax)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Note on Evaluation\n",
    "\n",
    "How do we make sure our model is decent?\n",
    "\n",
    "- For classification:\n",
    "    - Compute accuracy, **precision**, **recall**, **ROC curves**, **false positive** and **false negative** rates, and **F1** score.\n",
    "    - Plot a **confusion matrix** counting the number of times a ground truth label matches a predicted label for all possible labels.\n",
    "\n",
    "<img src=\"assets/confusion_matrix.png\" width=\"50%\"/>\n",
    "\n",
    "[Source](https://stackoverflow.com/questions/31324218/scikit-learn-how-to-obtain-true-positive-true-negative-false-positive-and-fal)\n",
    "    \n",
    "- For sequences:\n",
    "    - **Mean** and **absolute** error over the sequence.\n",
    "    - **Levenstein** or **edit distance**: how many edits are needed to transform one sequence to another.\n",
    "\n",
    "\n",
    "- For generated language:\n",
    "    - Examine the lengths of the generated phrases compared to those of the training targets.\n",
    "    - **Word Error Rate (WER)**: edit distance for language.\n",
    "    - Examine fluency by comparing the output to that of a language model. i.e. is the next word close to what a language model predicts. Fluency can be computed as the **perplexity** between the model output and the output of a language model. (See [MSR Montreal's question-generation paper](https://arxiv.org/abs/1705.02012)).\n",
    "    - **BLEU score**: correlates with human judgement score on the quality of generated text. Similarly **ROUGE** and **METEOR**.\n",
    "\n",
    "\n",
    "**Note**: We can't optimize these measures directly in a neural network since they aren't **differentiable**. (You can use Reinforcement Learning / REINFORCE if you're ambitious)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finally an Example!\n",
    "\n",
    "Here we'll classify a document as having a positive or negative sentiment. Our approach is as follows:\n",
    "\n",
    "1. **Features**: tokenize a dataset of text files and look up the GloVe vector for each token.\n",
    "\n",
    "1. **Model**: feed the vector representation of each token through a bidirectional recurrent neural network and use the final output of the RNN as input to a densely connected layer.\n",
    "\n",
    "1. **Training**: minimize the cross entropy between the sigmoid of the model output and the sentiment label, for each document.\n",
    "\n",
    "Our sentiment data will come from a subset of IMDB reviews, hosted [here](https://www.cs.cornell.edu/people/pabo/movie-review-data/). We include the 50-dimensional [GloVe](https://nlp.stanford.edu/projects/glove/) word vectors from the Wikipedia and Gigaword corpora [here](http://nlp.stanford.edu/data/glove.6B.zip).\n",
    "\n",
    "See `sentiment_rnn.py` for a standalone implementation.\n",
    "\n",
    "Note a pretrained model is provided at https://msrmtl-public-store.azureedge.net/ai4good/sentiment_90pct_639ep.tar.gz.\n",
    "\n",
    "This tutorial works with **TensorFlow 1.8**.\n",
    "\n",
    "-----\n",
    "\n",
    "First we read the GloVe vectors and build two lookup tables:\n",
    "1. One that maps word (string) -> token index (int), exposed by `lookup_word`.\n",
    "1. `glove`: token index (int) -> GloVe embedding (array of float).\n",
    "\n",
    "These use an `UNK` token for out-of-vocabulary words internally, and provide a padding token for shorter snippets of text. See `embedding.py` for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import numpy as np\n",
    "import os\n",
    "import shutil\n",
    "import tensorflow as tf\n",
    "\n",
    "from embedding import glove, look_up_word, PAD_TOKEN\n",
    "\n",
    "# Set to True to start from scratch, False to continue training an existing model\n",
    "FROM_SCRATCH = False\n",
    "\n",
    "TRAIN_BATCH_SIZE = 16\n",
    "EPOCHS = 2\n",
    "\n",
    "MAX_LENGTH = 200\n",
    "VALIDATION_SPLIT = 0.2\n",
    "EMBEDDING_DIMS = glove.shape[1]\n",
    "RNN_UNITS = 64\n",
    "\n",
    "logging.basicConfig(level=logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we get the raw dataset, tokenize it, and generate labels.\n",
    "\n",
    "For each batch we generate the token indices for embedding lookup, sequence lengths, and labels. We don't do any fancy tokenization here since the data is so clean.\n",
    "\n",
    "We define a generator we can use to create TensorFlow `Dataset` objects that feed our data through the computational graph. In order to evaluate our model on the whole training and validation sets after each epoch we create separate dataset initalizers and a dynamic batch size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache_dir = '.cache'\n",
    "\n",
    "if FROM_SCRATCH:\n",
    "    checkpoint_dir = 'experiment1'\n",
    "    if os.path.exists(checkpoint_dir):\n",
    "        shutil.rmtree(checkpoint_dir)\n",
    "    start_epoch = 0\n",
    "else:\n",
    "    # Put downloaded pretrained model here or your own trained one #\n",
    "    checkpoint_dir = 'sentiment_90pct_639ep'\n",
    "    start_epoch = 639\n",
    "\n",
    "for d in [cache_dir, checkpoint_dir]:\n",
    "    if not os.path.exists(d):\n",
    "        os.mkdir(d)\n",
    "\n",
    "RELATIVE_POLARITY_DATASET_SUBDIR = os.path.join('datasets', 'review_polarity')\n",
    "DATA_DIR = os.path.join(\n",
    "    cache_dir, RELATIVE_POLARITY_DATASET_SUBDIR, 'txt_sentoken')\n",
    "do_extract = not os.path.exists(DATA_DIR)\n",
    "\n",
    "dataset = tf.keras.utils.get_file(\n",
    "    fname='review_polarity.tar.gz',\n",
    "    cache_dir=cache_dir,\n",
    "    cache_subdir=RELATIVE_POLARITY_DATASET_SUBDIR,\n",
    "    origin='https://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.gz',\n",
    "    extract=do_extract)\n",
    "\n",
    "train_file_sents, val_file_sents = [], []\n",
    "\n",
    "for d, sent in [(os.path.join(DATA_DIR, sd), score) for sd, score in [('pos', 1.), ('neg', 0.)]]:\n",
    "    files = os.listdir(d)\n",
    "    split_index = int((1-VALIDATION_SPLIT)*len(files))\n",
    "    train_file_sents += [(os.path.join(d, f), sent)\n",
    "                         for f in files[:split_index]]\n",
    "    val_file_sents += [(os.path.join(d, f), sent)\n",
    "                       for f in files[split_index:]]\n",
    "\n",
    "\n",
    "def make_token_generator_for_files(src_file_sents):\n",
    "    def generator():\n",
    "        for f, sent in src_file_sents:\n",
    "            # Put your custom tokenizing code here\n",
    "            # Use nltk.word_tokenize, but in this case the dataset is processed so we don't need to\n",
    "            #   import nltk\n",
    "            #   nltk.download('punkt')\n",
    "            line_token_ids = [look_up_word(t.lower()) for ts in [line.split() for line in tf.gfile.GFile(\n",
    "                f, 'r').readlines()] for t in ts][:MAX_LENGTH]\n",
    "            token_ids_length = len(line_token_ids)\n",
    "            # Could also do `padded_batch` here\n",
    "            line_token_ids += [PAD_TOKEN] * (MAX_LENGTH - token_ids_length)\n",
    "            yield (line_token_ids, token_ids_length, sent)\n",
    "    return generator\n",
    "\n",
    "batch_size = tf.placeholder(tf.int64)\n",
    "\n",
    "train_set_size, val_set_size = len(train_file_sents), len(val_file_sents)\n",
    "\n",
    "train_dataset = tf.data.Dataset.from_generator(\n",
    "    make_token_generator_for_files(train_file_sents), (tf.int32, tf.int32, tf.float32), (tf.TensorShape([None]), tf.TensorShape(None), tf.TensorShape(None)))\\\n",
    "    .shuffle(train_set_size)\\\n",
    "    .batch(batch_size)\n",
    "\n",
    "val_dataset = tf.data.Dataset.from_generator(\n",
    "    make_token_generator_for_files(val_file_sents), (tf.int32, tf.int32, tf.float32), (tf.TensorShape([None]), tf.TensorShape(None), tf.TensorShape(None)))\\\n",
    "    .shuffle(val_set_size)\\\n",
    "    .batch(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an iterator that will choose a dataset based on the initializer used and will load batch data into the graph. Token embeddings are looked up through a table parameterized by the `glove` map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = tf.data.Iterator.from_structure(\n",
    "    train_dataset.output_types, train_dataset.output_shapes)\n",
    "\n",
    "train_init_op = iterator.make_initializer(train_dataset)\n",
    "val_init_op = iterator.make_initializer(val_dataset)\n",
    "\n",
    "batch_token_ids, batch_seq_lens, batch_labels = iterator.get_next()\n",
    "\n",
    "embedding_table = tf.get_variable(\"embedding_table\", initializer=glove)\n",
    "\n",
    "batch_embedding = tf.nn.embedding_lookup(embedding_table, batch_token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define our model. We feed all the batch sequences through a bidirectional recurrent neural network having forward and backward **gated recurrent unit (GRU)** RNN layers. We chose GRU here because it has fewer parameters making it faster to train, and its hidden state is equal to its output at each time step.\n",
    "\n",
    "We take the last hidden state (i.e. last output) of each of the forward and backward units concatenated since this is a representation of all the information learned over the whole sequence. These features are then fed as input to a linear layer to create **logits**. We don't apply a non-linearity or squashing function since TensorFlow can incorporate these directly into loss functions for numerical stability purposes and space efficency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fwd = tf.contrib.rnn.GRUCell(num_units=RNN_UNITS)\n",
    "bwd = tf.contrib.rnn.GRUCell(num_units=RNN_UNITS)\n",
    "\n",
    "_, final_rnn_state = tf.nn.bidirectional_dynamic_rnn(\n",
    "    fwd,\n",
    "    bwd,\n",
    "    batch_embedding,\n",
    "    sequence_length=batch_seq_lens,\n",
    "    dtype=tf.float32\n",
    ")\n",
    "\n",
    "fwd_state, bwd_state = final_rnn_state\n",
    "\n",
    "last_rnn_state = tf.concat([fwd_state, bwd_state], axis=1)\n",
    "\n",
    "sentiment_logits = tf.layers.dense(\n",
    "    last_rnn_state,\n",
    "    1,\n",
    "    use_bias=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our loss is the binary cross entropy between the squashed logits (predictions) and labels. One-hot encoding with softmax is mathematically equivalent here. We also define an `accuracy` operator used to evaluate our model's performance, independent of the loss. We use the Adam optimizer with default parameters because this generally works well. Finally we add a `Saver` to save and restore our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_op = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(\n",
    "    logits=sentiment_logits, labels=batch_labels))\n",
    "\n",
    "accuracy = tf.metrics.accuracy(\n",
    "    batch_labels,\n",
    "    tf.greater(sentiment_logits, tf.zeros(tf.shape(sentiment_logits)))\n",
    ")\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer()\n",
    "trainer = optimizer.minimize(loss_op)\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before our training loop we need to initalize global variables (`batch_size`), local variables (used by `tf.metrics.accuracy`), and lookup tables. We reinitialize our dataset iterators on each epoch, and when our training iterator is out of data we evaluate our model on the entire training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = tf.Session()\n",
    "\n",
    "if FROM_SCRATCH:\n",
    "    session.run(tf.global_variables_initializer())\n",
    "else:\n",
    "    saver.restore(session, os.path.join(checkpoint_dir, 'model-{0}'.format(start_epoch)))\n",
    "\n",
    "session.run(tf.local_variables_initializer())\n",
    "session.run(tf.tables_initializer())\n",
    "\n",
    "last_epoch = start_epoch + EPOCHS\n",
    "\n",
    "for i in range(start_epoch + 1, last_epoch + 1, 1):\n",
    "    session.run(train_init_op, feed_dict={batch_size: TRAIN_BATCH_SIZE})\n",
    "    logging.info('='*50)\n",
    "    logging.info('EPOCH %d ' % i + '-'*40)\n",
    "    # Iterate over batches\n",
    "    batchn = 0\n",
    "    while True:\n",
    "        try:\n",
    "            loss, acc, bs, _ = session.run(\n",
    "                [loss_op, accuracy, batch_size, trainer], feed_dict={batch_size: TRAIN_BATCH_SIZE})\n",
    "\n",
    "            # Print stats at the start and end of the batch for debugging\n",
    "            if batchn == 0 or batchn == ((train_set_size // bs) - 1):\n",
    "                logging.info('ep={}, batch={}, loss={:.5f}, acc={:.4f}'.format(\n",
    "                    i, batchn, loss, acc[0]))\n",
    "\n",
    "            batchn += 1\n",
    "\n",
    "        except tf.errors.OutOfRangeError:\n",
    "            break\n",
    "\n",
    "    session.run(train_init_op, feed_dict={batch_size: train_set_size})\n",
    "    loss, acc = session.run([loss_op, accuracy])\n",
    "    logging.info('-'*50)\n",
    "    logging.info(\n",
    "        'TRAIN RESULTS: loss={:.5f}, acc={:.4f}'.format(loss, acc[0]))\n",
    "\n",
    "    session.run(val_init_op, feed_dict={batch_size: val_set_size})\n",
    "    loss, acc = session.run([loss_op, accuracy])\n",
    "    logging.info(\n",
    "        'VALIDATION RESULTS: loss={:.5f}, acc={:.4f}'.format(loss, acc[0]))\n",
    "\n",
    "    saver.save(session, os.path.join(checkpoint_dir, 'model'), i)\n",
    "\n",
    "logging.info('Done training.')\n",
    "session.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's restore our model and try it out. We define a runtime method to feed data forward into the runtime graph without calling the `trainer` which backpropagates gradients. Note we don't have a sigmoid op in the graph so we look at the sign of the sentiment logit, where 0 is the decision boundary of a sigmoid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def interact(sess):\n",
    "    logging.info('='*40)\n",
    "    logging.info('Interactive runtime')\n",
    "    logging.info('='*40)\n",
    "    inp = input('Enter a phrase or `q` to quit: ')\n",
    "    while inp and inp != 'q':\n",
    "        logging.info('Query: %s' % inp)\n",
    "        line_token_ids = [look_up_word(t.lower())\n",
    "                          for t in inp.split()][:MAX_LENGTH]\n",
    "        token_ids_length = len(line_token_ids)\n",
    "        line_token_ids += [PAD_TOKEN] * (MAX_LENGTH - token_ids_length)\n",
    "\n",
    "        pred = sess.run([sentiment_logits], feed_dict={\n",
    "            batch_token_ids: [line_token_ids], batch_seq_lens: [token_ids_length], batch_size: 1})\n",
    "\n",
    "        if pred[0] >= 0:\n",
    "            logging.info('Result: POSTIVE (+)')\n",
    "        else:\n",
    "            logging.info('Result: NEGATIVE (-)')\n",
    "\n",
    "        inp = input('Enter a phrase or `q` to quit: ')\n",
    "        \n",
    "\n",
    "# Create a new session, load the model in, and try it out.\n",
    "new_session = tf.Session()\n",
    "print(checkpoint_dir)\n",
    "saver.restore(new_session, os.path.join(checkpoint_dir, 'model-{0}'.format(last_epoch)))\n",
    "new_session.run(tf.local_variables_initializer())\n",
    "new_session.run(tf.tables_initializer())\n",
    "\n",
    "interact(new_session)\n",
    "\n",
    "new_session.close()\n",
    "logging.info('All done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "#### Evaluate the model.\n",
    "\n",
    "- Test the model on different sentiment datasets, e.g. Rotten Tomatoes.\n",
    "- Look at different evaluation metrics defined above ^.\n",
    "\n",
    "\n",
    "#### Augment the data.\n",
    "\n",
    "- Train on varied sequence lengths. Augment the dataset by randomly taking snippets of different lengths from the documents.\n",
    "- Train on more data.\n",
    "\n",
    "\n",
    "#### Adjust the training setup.\n",
    "\n",
    "- Use actual sequence lengths so the whole RNN isn't unrolled or use [`padded_batch`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#padded_batch).\n",
    "- Dynamic learning rate or [batch size](https://arxiv.org/abs/1711.00489).\n",
    "- Adjust the batch size depending on the size of the training set.\n",
    "- Change the weight and bias initialization scheme.\n",
    "    \n",
    "\n",
    "#### Augment the model.\n",
    "\n",
    "- Add another dense layer or two to the output.\n",
    "- Stack more RNN layers.\n",
    "- Make the word embeddings trainable.\n",
    "- Use higher dimensional word embeddings.\n",
    "- Add dropout layers to prevent overfitting.\n",
    "- Apply **batch normalization**.\n",
    "- Add an **attention** mechanism over the RNN outputs / hidden states for each time step.\n",
    "- Add **convolution** over the input sequence where the filters are inputs to an RNN.\n",
    "- Add **character level embeddings**\n",
    "- Incorporate features learned by other embedding models (e.g. universal sentence encoder, NNLM, character level embeddings, ELMo) and **fine tune** them. Check out TensorFlow Hub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Great Resources\n",
    "\n",
    "- [A great collection of NLP tutorials and resources](https://alex-fabbri.github.io/TutorialBank/)\n",
    "- [Stanfords Deep Learning for NLP course](http://cs224d.stanford.edu/)\n",
    "- [Sebastian Ruder's blog](http://ruder.io/#open)\n",
    "- [Maluuba's QGen Workshop](https://github.com/Maluuba/qgen-workshop)\n",
    "- [Debugging your neural network](http://theorangeduck.com/page/neural-network-not-working)\n",
    "- [More advanced tutorial on TensorFlow datasets](https://towardsdatascience.com/how-to-use-dataset-in-tensorflow-c758ef9e4428)\n",
    "\n",
    "### Datasets\n",
    "\n",
    "- Lots of sources aggregated on GitHub:\n",
    "    - https://github.com/niderhoff/nlp-datasets\n",
    "    - https://github.com/karthikncode/nlp-datasets\n",
    "- [Maluuba Datasets](https://datasets.maluuba.com/)\n",
    "\n",
    "### Software\n",
    "\n",
    "- [Fast.ai](https://github.com/fastai/fastai) library and course for deeplearning & NLP using PyTorch.\n",
    "- [AllenNLP](https://github.com/allenai/allennlp) NLP and deep learning library.\n",
    "- [SpaCy](https://spacy.io/) library for classic NLP and text processing.\n",
    "- [Gensim](https://radimrehurek.com/gensim/), good for word vectors and topic modelling.\n",
    "- [NLTK](https://www.nltk.org/) library for clasic NLP and text processing.\n",
    "- [Pretrained models and embeddings in TensorFlow Hub](https://www.tensorflow.org/hub/modules/text)\n",
    "- [Maluuba's nlg-eval tools](https://github.com/Maluuba/nlg-eval)\n",
    "- [TensorFlow datasets](https://www.tensorflow.org/programmers_guide/datasets)\n",
    "\n",
    "**Training data for this example was originally curated for this work**:\n",
    "\n",
    "Pang, B., & Lee, L. (2004, July). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics (p. 271). Association for Computational Linguistics."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}