DeepLearning-07-Working-With-Text-Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN: Working with text data<a id=\"Top\"></a>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "\n",
    "<font size = 3>\n",
    "Table of Content\n",
    "<ul>\n",
    "<li>1. Tokenization</li>  \n",
    "<li>2. <a href=\"#Part_2\">One-hot text encoding</a></li>\n",
    "<li>3. <a href=\"#Part_3\">Word embeddings</a></li>\n",
    "    <ul>\n",
    "        <li> 3.1 <a href=\"#Part_3_1\">Learning word embeddings with the Keras Embedding layer</a></li>\n",
    "        <li> 3.2 <a href=\"#Part_3_2\">Using pretrained word embeddings</a></li>\n",
    "    </ul>\n",
    "</ul>    \n",
    "</font>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Tokenization\n",
    "\n",
    "Machine learning algorithms don't truly understand text in a human sense. A learning model only takes numerical\n",
    "inputs and trys to map out the input data's structure. So in order to work wiht text, which is one of the most\n",
    "widespread forms of sequential data, one has to decide a word representation that maps text into numerical\n",
    "tensors. Sometimes, this procudure is called vectorizing text. There are multiple possible ways:\n",
    "- Segment text into words, and transform each word into a vector.\n",
    "- Segment text into characters, and transform each character into a vector.\n",
    "- Extrac n-grams of words of characters, and transform each n-gram into a vector.\n",
    "\n",
    "The different units that one can break down text into (words, characters, n-grams) are called __tokens__.\n",
    "Breaking text into tokens is called __tokenization__. So tokenization is really a name that applies to all procedures\n",
    "that can associate numeric vectors with the generated tokens. An example of this process that goes from \n",
    "text to tokens to vectors is depicted by the following diagram\n",
    "\n",
    "<img src='./images/fig_RNN-TextData.png' width=350>\n",
    "\n",
    "There are two major tokenization schemes: one-hot encoding and token embedding (or word embedding).\n",
    "\n",
    "As a side note, word n-grams are groups of $N$ (or fewer) consecutive words that one can extract from a sentence.\n",
    "The same idea can be applied to characters. Take the sentence \"__This cat sat on the mat__\" as an example.\n",
    "It can be decomposed into the following set of 2-grams\n",
    "```python\n",
    "    {\"The\", \"The cat\", \"cat\", \"cat sat\", \"sat\", \"sat on\", \"on\", \"on the\", \"the\", \"the mat\", \"mat\"}\n",
    "```\n",
    "or the following 3-grams:\n",
    "```python\n",
    "    {\"The\", \"The cat\", \"cat\", \"cat sat\", \"The cat sat\",\n",
    "     \"sat\", \"sat on\", \"on\", \"cat sat on\", \"on the\", \"the\", \n",
    "     \"sat on the\", \"the mat\", \"mat\", \"on the mat\"}\n",
    "```\n",
    "\n",
    "One can see that n-gram tokenization is not an order-preserving method.\n",
    "\n",
    "\n",
    "# 2. One-hot text encoding<a id=\"Part_2\"></a>\n",
    "<a href=\"#Top\">Back to page top</a>\n",
    "\n",
    "One-hot encoding is the most common and basic way of turning a token into a vector. One-hot encoding is used\n",
    "all over the place in machine learning. For example, in the MNIST challenge, the targets are one-hot encoded\n",
    "before they are sent to the CNN model. One-hot encoding consists of attaching a unique integer index $i$ with \n",
    "every word, then turning this integer $i$ into a binary vector of size $N$, the size of the vocabulary. The\n",
    "vector is all zeros except for the $i$-th entry, which is 1. Of course, one-hot encoding can be applied at the \n",
    "character level too. Below is a code example from Chollet's book that performs word-level one-hot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " unique word: The         , token index:   1\n",
      " unique word: cat         , token index:   2\n",
      " unique word: sat         , token index:   3\n",
      " unique word: on          , token index:   4\n",
      " unique word: the         , token index:   5\n",
      " unique word: mat.        , token index:   6\n",
      " unique word: dog         , token index:   7\n",
      " unique word: ate         , token index:   8\n",
      " unique word: my          , token index:   9\n",
      " unique word: homework.   , token index:  10\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "token_index = {}\n",
    "for sample in samples:\n",
    "    for word in sample.split():\n",
    "        if word not in token_index:\n",
    "            token_index[word] = len(token_index) + 1\n",
    "            # Print out the unique words            \n",
    "            print(' unique word: {0:12s}, token index: {1:3d}'.format(word, token_index[word]))\n",
    "            \n",
    "# Consider the first 10 words in each sample            \n",
    "max_length = 10\n",
    "\n",
    "results = np.zeros(shape=(len(samples), \n",
    "                          max_length, \n",
    "                          max(token_index.values()) + 1)) \n",
    "\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index = token_index.get(word)\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the two sentences have total 11 words. But only 10 of them are unique ones. Note that \n",
    "the code can differenciate lower and upper cases. So \"The\" and \"the\" are two different tokens. Once the unique words\n",
    "have been figured out, the code went on to attach of the words an integer, then convert the interger into a\n",
    "binary vector. Note that the first index 0 of the vector is not associated with anything. The variable \n",
    "`max_length` limits the number of words to be tokenized. The following cell shows the vectorized sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras has built-in utilities for one-hot encoding text at the word or character level. The tools have a number\n",
    "of important features such as stripping special characters from strings and only taking into account the $N$ \n",
    "most common words in your dataset. So in general one should use Keras utilities. Below is an example using \n",
    "Keras utitlies on the sentences we just saw from the previous example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 9 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# Consider the first num_words in each sentence.\n",
    "tokenizer = Tokenizer(num_words=20) \n",
    "tokenizer.fit_on_texts(samples)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(samples)\n",
    "\n",
    "one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')\n",
    "word_index = tokenizer.word_index \n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'cat': 2,\n",
       " 'sat': 3,\n",
       " 'on': 4,\n",
       " 'mat': 5,\n",
       " 'dog': 6,\n",
       " 'ate': 7,\n",
       " 'my': 8,\n",
       " 'homework': 9}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 1. 1. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 1. 0. 0. 0. 0. 1. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "print(one_hot_results[0])\n",
    "print(one_hot_results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apparently, Keras `Tokenizer` is case insensitive, it treats \"The\" and \"the\" as identical. This makes sense.\n",
    "`Tokenizer` also returns a dictionary of unique words and corresponding index.\n",
    "\n",
    "If the number of unique tokens in the vocabulary is too large, one can hash words into vectors of fixed size.\n",
    "The advantage of this method is that it does not maintain an explicit list of word index, which saves memory\n",
    "and allows online encoding of the data. However, the method may suffer from hash collisions: two different\n",
    "words may end up with the same hash. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Word embeddings<a id=\"Part_3\"></a>\n",
    "<a href=\"#Top\">Back to page top</a>\n",
    "\n",
    "While one-hot encoding is straightford, the algorithm lacks representation capability. Each one-hot vectorization\n",
    "distinguishes a word in a vocabulary from every other word in the vocabulary. So basically one-hot encoding can \n",
    "not tell you the semantics of the text. Word embeddings, on the other hand, will group commonly co-occurring\n",
    "tokens together in the embedding space. Let's expand on the meaning of \"together.\"\n",
    "\n",
    "Word embeddings are algorithms that map human languages into a geometric space, called embedding space. As a\n",
    "result, it is possible to measure the distance between two tokens (word or character) in the embedding space. \n",
    "For example, a L2 diatance can be a measure of separation between two tokens. And since embedding space is a \n",
    "vector space, a vector that conntects two tokens thus has two properties: length and direction. Reasonable \n",
    "word embeddings should reflect the semantic relationships between the tokens in terms of the geometric \n",
    "distance measure. This relationship could be understood as a geometric transformation, as indicated by the\n",
    "example below.\n",
    "\n",
    "<img src='./images/fig_RNN-WordEmbedding.png' width=300>\n",
    "\n",
    "The diagram is a toy example of four words embedded in a two-dimensional embedding space. Some semantic \n",
    "relationships can be observed. Firstly, the translations from cat to tiger and from dog to wolf are given\n",
    "by the same vector (solid orange). Secondly, another vector (dashed orange) moves the point from dog to cat \n",
    "and from wolf to tiger. The first vector can be interpreted as the relation \"from pet to wild animal;\" \n",
    "while the second can be interpreted as a “from canine to feline” vector.\n",
    "\n",
    "One should keep in mind that __there is no one ideal word-embedding space__. As word embeddings are largely \n",
    "task-dependent. A good word-embedding space for English-language movie review may very likely look different \n",
    "from the one for French-language spam email detection.\n",
    "\n",
    "There are two ways to get word embeddings:\n",
    "- Learn word embeddings jointly with the main network training task (such as document classification or sentiment\n",
    "  prediction). In this setup, you start with random word vectors and then learn word vectors in the same way you\n",
    "  learn the weights of a neural network.\n",
    "- Pretrained word embeddings: load into your model word embeddings that were precomputed using a different \n",
    "  machine-learning task than the one you’re trying to solve. \n",
    "\n",
    "\n",
    "## 3.1 Learning word embeddings with the Keras `Embedding` layer<a id=\"Part_3_1\"></a>\n",
    "<a href=\"#Top\">Back to page top</a>\n",
    "\n",
    "Since a good word-embedding space varies from task to task, sometimes it's reasonable to learn a new embedding\n",
    "space for the task at hand. Keras offers an excellent tool to carry out the job: the `Embedding` layer. The `Embedding` layer can be understood as a dictionary that maps integer indices to dense vectors:\n",
    "\n",
    "$$ \\mbox{Word index}\\,\\Longrightarrow\\,\\mbox{Embedding layer}\\,\\Longrightarrow\\,\\mbox{Word vector} $$\n",
    "\n",
    "The Keras `Embedding` has the following functional interface:\n",
    "```python\n",
    "   Embedding(input_dim, output_dim, input_length=None) \n",
    "```    \n",
    "The meaning of the arguments:\n",
    "- `input_dim`: __Size of the vocabulary, i.e. the maximum number of tokens + 1__. For example, setting \n",
    "  `inpu_dim=1000` means the largest token index should be no larger than 999. \n",
    "- `output_dim`: __Dimension of the embedding space__.\n",
    "- `input_length`: This optional argument specifies __the length of input sequences__, when it is constant. \n",
    "  It is required if one is going to connect `Flatten()` then `Dense()` layers.\n",
    "  \n",
    "Let's look at an example:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 100, 8)            8000      \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 800)               0         \n",
      "=================================================================\n",
      "Total params: 8,000\n",
      "Trainable params: 8,000\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten\n",
    "\n",
    "maxlen=100\n",
    "model = Sequential()\n",
    "model.add( Embedding(1000, 8, input_length=maxlen) )\n",
    "model.add( Flatten() )\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this code snippet, \n",
    "1. `input_dim=1000` means we are considering 1000 unique tokens. So the maximal index should be no larger than 999.\n",
    "2. `output_dim=8` implies the embedding space dimsnsion is 8.\n",
    "3. `input_length=maxlen` suggests that we are considering the first `maxlen=100` words (among the 1000 tokens) in\n",
    "   the text.\n",
    "\n",
    "This model's input tensor has shape `(batch_size, input_length) = (batch_size, 100)`. After \n",
    "the `Embedding` layer activation, the output will have shape `(batch_size, 100, 8)`. The `Flatten()` layer turns\n",
    "the 3D tensor `(batch_size, 100, 8)` into a 2D one with shape `(batch_size, 100*8)`.\n",
    "  \n",
    "Let's look at another example where `input_length` is not specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, None, 8)           8000      \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 32)                5248      \n",
      "=================================================================\n",
      "Total params: 13,248\n",
      "Trainable params: 13,248\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import LSTM\n",
    "\n",
    "model = Sequential()\n",
    "model.add( Embedding(1000, 8) )\n",
    "model.add( LSTM(32) )\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now in this example, the model's input is of the shape `(batch_size, sequence_length)`, i.e. the second \n",
    "dimension is inferred dynamically from the input itself. As such, the model returns a floating point tensor \n",
    "of shape `(batch_size, sequence_length, output_dim)`.  This 3D tensor is then sent to an LSTM layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Using pretrained word embeddings<a id=\"Part_3_2\"></a>\n",
    "<a href=\"#Top\">Back to page top</a>\n",
    "\n",
    "In the case when there is little data available to learn an appropriate task-specific embedding space, one can\n",
    "load embedding vectors from a precomputed embedding space that is highly structured and has useful properties.\n",
    "This rationale of using pretrained word embeddings is basically the same as for using pretrained convolutional\n",
    "neural networks. \n",
    "\n",
    "There are various precomputed databases of word embeddings that one can use in a Keras `Embedding` layer.\n",
    "Examples include\n",
    "- __Google <a href='https://code.google.com/archive/p/word2vec/'>Word2vec</a>__. It seems that the Google Code\n",
    "  repository is no longer in use. Anyways, Google provides a nice \n",
    "  <a href='https://www.tensorflow.org/tutorials/representation/word2vec'>tutorial</a>. \n",
    "- __<a href='https://nlp.stanford.edu/projects/glove/'>GloVe</a>__, or Global Vectors for Word Representation. The\n",
    "  English tokens are obtained from Wikipedia data and Common Crawl data. It is argued \n",
    "  <a href='https://towardsdatascience.com/beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec'>here</a> that Word2vec only takes local contexts into account. But GloVe uses neural methods to\n",
    "  decompose the co-occurence matrix into more expressive and dense word vectors. However in practive, neither\n",
    "  GloVe or Word2vec has been shown to provide better results. Rather, they should both be evaluated for a given\n",
    "  dataset.\n",
    "- __<a href='https://github.com/facebookresearch/fastText'>Facebook fastText</a>__ is built on Word2vec by\n",
    "  learning representations for each word and the n-grams found within each word. FastText has been shown to\n",
    "  be more accurate than Word2vec vectors by several measures.\n",
    "- <a href='https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub'>TensorFlow Hub</a>'s\n",
    "  <a href='https://tfhub.dev/s?module-type=text-embedding'>Text Embedding module</a>.\n",
    "  \n",
    "How do we use pretrained word embeddings? From the Embedding layer training section, we know that an \n",
    "`Embedding` layer is essentially a dictiionary that maps a word index to a vector in the embedding space.\n",
    "Therefore, a pretrained word embedding is a 2D matrix of shape `(max_words, embedding_dim)` where each \n",
    "$i$ in the `max_word` entries contains the `emneddomg_dim`-dimensional vectors for the word of index $i$\n",
    "in the reference word index built during tokenization. The index 0 does not stand for and word or token, it's\n",
    "a placeholder.\n",
    "\n",
    "Let's download the GloVe word embeddings from 2014 English Wikipedia as an example and take a closer look at \n",
    "the precomputed embedding. The file name is `glove.6B.zip`. After the file is unzipped, we'll load the file\n",
    "`glove.6B.50d.txt`. As the file name suggests, the dimension of the embedding space is 50. The file has 400K \n",
    "tokens, but we'll read in the first 100 words for the purpose of demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>0.418000</td>\n",
       "      <td>0.249680</td>\n",
       "      <td>-0.41242</td>\n",
       "      <td>0.121700</td>\n",
       "      <td>0.345270</td>\n",
       "      <td>-0.044457</td>\n",
       "      <td>-0.49688</td>\n",
       "      <td>-0.178620</td>\n",
       "      <td>-0.000660</td>\n",
       "      <td>-0.656600</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.298710</td>\n",
       "      <td>-0.157490</td>\n",
       "      <td>-0.347580</td>\n",
       "      <td>-0.045637</td>\n",
       "      <td>-0.442510</td>\n",
       "      <td>0.187850</td>\n",
       "      <td>0.002785</td>\n",
       "      <td>-0.184110</td>\n",
       "      <td>-0.115140</td>\n",
       "      <td>-0.785810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>,</th>\n",
       "      <td>0.013441</td>\n",
       "      <td>0.236820</td>\n",
       "      <td>-0.16899</td>\n",
       "      <td>0.409510</td>\n",
       "      <td>0.638120</td>\n",
       "      <td>0.477090</td>\n",
       "      <td>-0.42852</td>\n",
       "      <td>-0.556410</td>\n",
       "      <td>-0.364000</td>\n",
       "      <td>-0.239380</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.080262</td>\n",
       "      <td>0.630030</td>\n",
       "      <td>0.321110</td>\n",
       "      <td>-0.467650</td>\n",
       "      <td>0.227860</td>\n",
       "      <td>0.360340</td>\n",
       "      <td>-0.378180</td>\n",
       "      <td>-0.566570</td>\n",
       "      <td>0.044691</td>\n",
       "      <td>0.303920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>.</th>\n",
       "      <td>0.151640</td>\n",
       "      <td>0.301770</td>\n",
       "      <td>-0.16763</td>\n",
       "      <td>0.176840</td>\n",
       "      <td>0.317190</td>\n",
       "      <td>0.339730</td>\n",
       "      <td>-0.43478</td>\n",
       "      <td>-0.310860</td>\n",
       "      <td>-0.449990</td>\n",
       "      <td>-0.294860</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.000064</td>\n",
       "      <td>0.068987</td>\n",
       "      <td>0.087939</td>\n",
       "      <td>-0.102850</td>\n",
       "      <td>-0.139310</td>\n",
       "      <td>0.223140</td>\n",
       "      <td>-0.080803</td>\n",
       "      <td>-0.356520</td>\n",
       "      <td>0.016413</td>\n",
       "      <td>0.102160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>of</th>\n",
       "      <td>0.708530</td>\n",
       "      <td>0.570880</td>\n",
       "      <td>-0.47160</td>\n",
       "      <td>0.180480</td>\n",
       "      <td>0.544490</td>\n",
       "      <td>0.726030</td>\n",
       "      <td>0.18157</td>\n",
       "      <td>-0.523930</td>\n",
       "      <td>0.103810</td>\n",
       "      <td>-0.175660</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.347270</td>\n",
       "      <td>0.284830</td>\n",
       "      <td>0.075693</td>\n",
       "      <td>-0.062178</td>\n",
       "      <td>-0.389880</td>\n",
       "      <td>0.229020</td>\n",
       "      <td>-0.216170</td>\n",
       "      <td>-0.225620</td>\n",
       "      <td>-0.093918</td>\n",
       "      <td>-0.803750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>0.680470</td>\n",
       "      <td>-0.039263</td>\n",
       "      <td>0.30186</td>\n",
       "      <td>-0.177920</td>\n",
       "      <td>0.429620</td>\n",
       "      <td>0.032246</td>\n",
       "      <td>-0.41376</td>\n",
       "      <td>0.132280</td>\n",
       "      <td>-0.298470</td>\n",
       "      <td>-0.085253</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.094375</td>\n",
       "      <td>0.018324</td>\n",
       "      <td>0.210480</td>\n",
       "      <td>-0.030880</td>\n",
       "      <td>-0.197220</td>\n",
       "      <td>0.082279</td>\n",
       "      <td>-0.094340</td>\n",
       "      <td>-0.073297</td>\n",
       "      <td>-0.064699</td>\n",
       "      <td>-0.260440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>and</th>\n",
       "      <td>0.268180</td>\n",
       "      <td>0.143460</td>\n",
       "      <td>-0.27877</td>\n",
       "      <td>0.016257</td>\n",
       "      <td>0.113840</td>\n",
       "      <td>0.699230</td>\n",
       "      <td>-0.51332</td>\n",
       "      <td>-0.473680</td>\n",
       "      <td>-0.330750</td>\n",
       "      <td>-0.138340</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.069043</td>\n",
       "      <td>0.368850</td>\n",
       "      <td>0.251680</td>\n",
       "      <td>-0.245170</td>\n",
       "      <td>0.253810</td>\n",
       "      <td>0.136700</td>\n",
       "      <td>-0.311780</td>\n",
       "      <td>-0.632100</td>\n",
       "      <td>-0.250280</td>\n",
       "      <td>-0.380970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>in</th>\n",
       "      <td>0.330420</td>\n",
       "      <td>0.249950</td>\n",
       "      <td>-0.60874</td>\n",
       "      <td>0.109230</td>\n",
       "      <td>0.036372</td>\n",
       "      <td>0.151000</td>\n",
       "      <td>-0.55083</td>\n",
       "      <td>-0.074239</td>\n",
       "      <td>-0.092307</td>\n",
       "      <td>-0.328210</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.486090</td>\n",
       "      <td>-0.008027</td>\n",
       "      <td>0.031184</td>\n",
       "      <td>-0.365760</td>\n",
       "      <td>-0.426990</td>\n",
       "      <td>0.421640</td>\n",
       "      <td>-0.116660</td>\n",
       "      <td>-0.507030</td>\n",
       "      <td>-0.027273</td>\n",
       "      <td>-0.532850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.217050</td>\n",
       "      <td>0.465150</td>\n",
       "      <td>-0.46757</td>\n",
       "      <td>0.100820</td>\n",
       "      <td>1.013500</td>\n",
       "      <td>0.748450</td>\n",
       "      <td>-0.53104</td>\n",
       "      <td>-0.262560</td>\n",
       "      <td>0.168120</td>\n",
       "      <td>0.131820</td>\n",
       "      <td>...</td>\n",
       "      <td>0.138130</td>\n",
       "      <td>0.369730</td>\n",
       "      <td>-0.642890</td>\n",
       "      <td>0.024142</td>\n",
       "      <td>-0.039315</td>\n",
       "      <td>-0.260370</td>\n",
       "      <td>0.120170</td>\n",
       "      <td>-0.043782</td>\n",
       "      <td>0.410130</td>\n",
       "      <td>0.179600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>\"</th>\n",
       "      <td>0.257690</td>\n",
       "      <td>0.456290</td>\n",
       "      <td>-0.76974</td>\n",
       "      <td>-0.376790</td>\n",
       "      <td>0.592720</td>\n",
       "      <td>-0.063527</td>\n",
       "      <td>0.20545</td>\n",
       "      <td>-0.573850</td>\n",
       "      <td>-0.290090</td>\n",
       "      <td>-0.136620</td>\n",
       "      <td>...</td>\n",
       "      <td>0.030498</td>\n",
       "      <td>-0.395430</td>\n",
       "      <td>-0.385150</td>\n",
       "      <td>-1.000200</td>\n",
       "      <td>0.087599</td>\n",
       "      <td>-0.310090</td>\n",
       "      <td>-0.346770</td>\n",
       "      <td>-0.314380</td>\n",
       "      <td>0.750040</td>\n",
       "      <td>0.970650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'s</th>\n",
       "      <td>0.237270</td>\n",
       "      <td>0.404780</td>\n",
       "      <td>-0.20547</td>\n",
       "      <td>0.588050</td>\n",
       "      <td>0.655330</td>\n",
       "      <td>0.328670</td>\n",
       "      <td>-0.81964</td>\n",
       "      <td>-0.232360</td>\n",
       "      <td>0.274280</td>\n",
       "      <td>0.242650</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.123420</td>\n",
       "      <td>0.659610</td>\n",
       "      <td>-0.518020</td>\n",
       "      <td>-0.829950</td>\n",
       "      <td>-0.082739</td>\n",
       "      <td>0.281550</td>\n",
       "      <td>-0.423000</td>\n",
       "      <td>-0.273780</td>\n",
       "      <td>-0.007901</td>\n",
       "      <td>-0.030231</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           1         2        3         4         5         6        7   \\\n",
       "0                                                                         \n",
       "the  0.418000  0.249680 -0.41242  0.121700  0.345270 -0.044457 -0.49688   \n",
       ",    0.013441  0.236820 -0.16899  0.409510  0.638120  0.477090 -0.42852   \n",
       ".    0.151640  0.301770 -0.16763  0.176840  0.317190  0.339730 -0.43478   \n",
       "of   0.708530  0.570880 -0.47160  0.180480  0.544490  0.726030  0.18157   \n",
       "to   0.680470 -0.039263  0.30186 -0.177920  0.429620  0.032246 -0.41376   \n",
       "and  0.268180  0.143460 -0.27877  0.016257  0.113840  0.699230 -0.51332   \n",
       "in   0.330420  0.249950 -0.60874  0.109230  0.036372  0.151000 -0.55083   \n",
       "a    0.217050  0.465150 -0.46757  0.100820  1.013500  0.748450 -0.53104   \n",
       "\"    0.257690  0.456290 -0.76974 -0.376790  0.592720 -0.063527  0.20545   \n",
       "'s   0.237270  0.404780 -0.20547  0.588050  0.655330  0.328670 -0.81964   \n",
       "\n",
       "           8         9         10    ...           41        42        43  \\\n",
       "0                                    ...                                    \n",
       "the -0.178620 -0.000660 -0.656600    ...    -0.298710 -0.157490 -0.347580   \n",
       ",   -0.556410 -0.364000 -0.239380    ...    -0.080262  0.630030  0.321110   \n",
       ".   -0.310860 -0.449990 -0.294860    ...    -0.000064  0.068987  0.087939   \n",
       "of  -0.523930  0.103810 -0.175660    ...    -0.347270  0.284830  0.075693   \n",
       "to   0.132280 -0.298470 -0.085253    ...    -0.094375  0.018324  0.210480   \n",
       "and -0.473680 -0.330750 -0.138340    ...    -0.069043  0.368850  0.251680   \n",
       "in  -0.074239 -0.092307 -0.328210    ...    -0.486090 -0.008027  0.031184   \n",
       "a   -0.262560  0.168120  0.131820    ...     0.138130  0.369730 -0.642890   \n",
       "\"   -0.573850 -0.290090 -0.136620    ...     0.030498 -0.395430 -0.385150   \n",
       "'s  -0.232360  0.274280  0.242650    ...    -0.123420  0.659610 -0.518020   \n",
       "\n",
       "           44        45        46        47        48        49        50  \n",
       "0                                                                          \n",
       "the -0.045637 -0.442510  0.187850  0.002785 -0.184110 -0.115140 -0.785810  \n",
       ",   -0.467650  0.227860  0.360340 -0.378180 -0.566570  0.044691  0.303920  \n",
       ".   -0.102850 -0.139310  0.223140 -0.080803 -0.356520  0.016413  0.102160  \n",
       "of  -0.062178 -0.389880  0.229020 -0.216170 -0.225620 -0.093918 -0.803750  \n",
       "to  -0.030880 -0.197220  0.082279 -0.094340 -0.073297 -0.064699 -0.260440  \n",
       "and -0.245170  0.253810  0.136700 -0.311780 -0.632100 -0.250280 -0.380970  \n",
       "in  -0.365760 -0.426990  0.421640 -0.116660 -0.507030 -0.027273 -0.532850  \n",
       "a    0.024142 -0.039315 -0.260370  0.120170 -0.043782  0.410130  0.179600  \n",
       "\"   -1.000200  0.087599 -0.310090 -0.346770 -0.314380  0.750040  0.970650  \n",
       "'s  -0.829950 -0.082739  0.281550 -0.423000 -0.273780 -0.007901 -0.030231  \n",
       "\n",
       "[10 rows x 50 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_path = './../Keras/glove.6B/glove.6B.50d.txt'\n",
    "GloveEmbedding = pd.read_table(glove_path, sep=\" \", index_col=0, header=None, quoting=csv.QUOTE_NONE, nrows=100)\n",
    "GloveEmbedding.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the dataframe shows, each token (e.g. `the`, `of`, `to`, ... etc and punctuations) has ite corresponding \n",
    "50-dimension vector. \n",
    "\n",
    "With the precomputed vectors, the next step is to build an embedding matrix that maps the tokens from\n",
    "tokens from tje text data to its corresponding vector found in the precomputed embedding. If a token\n",
    "is not found in the precomputed embedding, it is customary to set its vector as zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}