19_Jan_Glove.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Name:__ Tonson Praphabkul  \n",
    "__Student_Id:__ st123010"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|        Model       | Test_opposite Accruacy | Test_plural Accruacy | Semantic Accruacy | Synthetic Similarity | Train_time |\n",
    "|:------------------:|------------------------|:--------------------:|:-----------------:|----------------------|------------|\n",
    "| Glove              | 0                      | 0                    | 0                 | 25.57                | 3m 57s     |\n",
    "| Glove_1000*        | 0                      | 0                    | 0                 | 23.51                | 11m 22s    |\n",
    "| Cbow*               | 0                      | 0                    | 0                 | 29.40                | 3m 7s      |\n",
    "| Skipgrams          | 0                      | 0                    | 0                 | 23.95                | 3m 49s     |\n",
    "| Neg_Skipgrams      | 0                      | 0                    | 0                 | 25.50                | 4m 3s      |\n",
    "| Glove (Pretrained) | 0                      | 0                    | 0                 | 1.43                 | -          |"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inference\n",
    "Interestingly, we archieve accuracy of 0 in all of the test for sementic. I tried to look for the reasons and I found out that it is very difficult for the model to accurately predicted the exactly word. One of the main reasons is related to the dataset and model which I will not discussed about it here. The interesting part is that the model is not 100% wrongs, when I looked at top 10 probility sometimes the correct answer appear. I think if I change to top10 accruacy I am sure that atleast we can obtains non-zero accuracy.  \n",
    "\n",
    "For the synthetic similarity, I can not find the way to exactly compute the accruacy but we do know a way to compare which one are better. I used MSE errors to compare each models. The ytrue from the dataset have mean around 6 which mean all others model except Glove(pretrained) are worse than just simply predict the mean. \n",
    "\n",
    "##### Parameters\n",
    "\n",
    "batch_size     = 10  \n",
    "embedding_size = 100  \n",
    "num_neg        = 10 # num of negative sampling  \n",
    "Window_size    = 2 (Except CBOW that only used window size of 1 due to technical issues)  \n",
    "criterion = nn.CrossEntropyLoss()  \n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)  \n",
    "\n",
    "All of the models are train for 500 epochs excepts for Glove_1000 that trained for 1,000 epochs and pretrained model.\n",
    "\n",
    "*Test_opposite is the set of word that are opposite to each other  \n",
    "*Test_plurals basically mean singular and plural e.g women woman  \n",
    "See more at the code below\n",
    "*For Glove(Pretrained) I used the 'glove.6B.100d.txt' dataset and the code provided by Chaky\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries.\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import nltk"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ETL/Tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to /root/nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Use corpus from nltk\n",
    "# Amamda recommend this!\n",
    "from nltk.corpus import brown\n",
    "nltk.download('brown')\n",
    "corpus_sentence = nltk.corpus.brown.sents(categories=['government'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['the', 'office', 'of', 'business', 'economics', '(', 'obe', ')', 'of', 'the', 'u.s.', 'department', 'of', 'commerce', 'provides', 'basic', 'measures', 'of', 'the', 'national', 'economy', 'and', 'current', 'analysis', 'of', 'short-run', 'changes', 'in', 'the', 'economic', 'situation', 'and', 'business', 'outlook', '.'], ['it', 'develops', 'and', 'analyzes', 'the', 'national', 'income', ',', 'balance', 'of', 'international', 'payments', ',', 'and', 'many', 'other', 'business', 'indicators', '.'], ['such', 'measures', 'are', 'essential', 'to', 'its', 'job', 'of', 'presenting', 'business', 'and', 'government', 'with', 'the', 'facts', 'required', 'to', 'meet', 'the', 'objective', 'of', 'expanding', 'business', 'and', 'improving', 'the', 'operation', 'of', 'the', 'economy', '.'], ['contact'], ['for', 'further', 'information', 'contact', 'director', ',', 'office', 'of', 'business', 'economics', ',', 'u.s.', 'department', 'of', 'commerce', ',', 'washington', '25', ',', 'd.c.', '.']]\n"
     ]
    }
   ],
   "source": [
    "corpus = [[word.lower() for word in sent] for sent in corpus_sentence] # Cool list comprehension trick!\n",
    "print(corpus[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3032"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the sentences in corpus\n",
    "len(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['localities', 'ward', 'honesty', 'accomplishing', 'charles']"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Flatten and get Unique words\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocab = list(set(flatten(corpus)))\n",
    "vocab[0:5]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numericalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "739\n",
      "car\n"
     ]
    }
   ],
   "source": [
    "# Word2index and Index2word\n",
    "\n",
    "# assign id to those vocabs\n",
    "word2index = dict()\n",
    "word2index.update({\"<UNK>\":  0})\n",
    "for idx, v in enumerate(vocab):\n",
    "        word2index.update({v:  idx + 1})\n",
    "\n",
    "#add <UNK>, which is a very normal token exists in the world\n",
    "vocab.append('<UNK>') #chaky, can it be ##UNK, or UNKKKKKK, or anything\n",
    "\n",
    "# Testing\n",
    "print(word2index['car'])\n",
    "\n",
    "# index2word\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "print(index2word[word2index['car']])\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Glove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "# Build co-occurance matrix\n",
    "from collections import Counter\n",
    "X_i = Counter(flatten(corpus)) # X_i\n",
    "print(X_i['car'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this function to create skipgram, basically fix a bit from\n",
    "# generated batch function\n",
    "def skip_grams_generated(window_size=1):\n",
    "\n",
    "# I fix a little from Chaky so we can modify the window_size\n",
    "    \n",
    "    # Make skip gram of one size window\n",
    "    skip_grams = []\n",
    "    # loop each word sequence\n",
    "    # we starts from 1 because 0 has no context\n",
    "    # we stop at second last for the same reason\n",
    "    for sent in corpus:\n",
    "        for i in range(1, len(sent) - 1): # So we can modify the window size\n",
    "            target = sent[i]\n",
    "            \n",
    "            context = list()\n",
    "            # ['a', 'b', 'c', 'd', 'e'] if window size = 2 and target is c\n",
    "            # this is basically append 'b', 'd', 'a', 'e' into context\n",
    "            \n",
    "            for j in range(window_size):\n",
    "                \n",
    "                if i - (j + 1) >= 0: # Check if it outside of range from the left of list\n",
    "                    context.append(sent[i - (j + 1)])\n",
    "                    #context.append(word2index[sent[i - (j + 1)]])\n",
    "                \n",
    "                if i + (j + 1) < len(sent): # Check if it outside of range from the right of list\n",
    "                    context.append(sent[i + (j + 1)])\n",
    "\n",
    "            for w in context:\n",
    "                skip_grams.append((target, w)) # Return tuple instead of list because we want to use count\n",
    "                                                # function later\n",
    "    \n",
    "    return skip_grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({('office', 'the'): 14,\n",
       "         ('office', 'of'): 14,\n",
       "         ('office', 'business'): 2,\n",
       "         ('of', 'office'): 14,\n",
       "         ('of', 'business'): 26,\n",
       "         ('of', 'the'): 1900,\n",
       "         ('of', 'economics'): 2,\n",
       "         ('business', 'of'): 26,\n",
       "         ('business', 'economics'): 2,\n",
       "         ('business', 'office'): 2,\n",
       "         ('business', '('): 2,\n",
       "         ('economics', 'business'): 2,\n",
       "         ('economics', '('): 1,\n",
       "         ('economics', 'of'): 2,\n",
       "         ('economics', 'obe'): 1,\n",
       "         ('(', 'economics'): 1,\n",
       "         ('(', 'obe'): 1,\n",
       "         ('(', 'business'): 2,\n",
       "         ('(', ')'): 141,\n",
       "         ('obe', '('): 1,\n",
       "         ('obe', ')'): 1,\n",
       "         ('obe', 'economics'): 1,\n",
       "         ('obe', 'of'): 1,\n",
       "         (')', 'obe'): 1,\n",
       "         (')', 'of'): 32,\n",
       "         (')', '('): 129,\n",
       "         (')', 'the'): 30,\n",
       "         ('of', ')'): 33,\n",
       "         ('of', 'obe'): 1,\n",
       "         ('of', 'u.s.'): 10,\n",
       "         ('the', 'of'): 1804,\n",
       "         ('the', 'u.s.'): 9,\n",
       "         ('the', ')'): 30,\n",
       "         ('the', 'department'): 64,\n",
       "         ('u.s.', 'the'): 9,\n",
       "         ('u.s.', 'department'): 5,\n",
       "         ('u.s.', 'of'): 10,\n",
       "         ('department', 'u.s.'): 5,\n",
       "         ('department', 'of'): 68,\n",
       "         ('department', 'the'): 65,\n",
       "         ('department', 'commerce'): 4,\n",
       "         ('of', 'department'): 68,\n",
       "         ('of', 'commerce'): 13,\n",
       "         ('of', 'provides'): 1,\n",
       "         ('commerce', 'of'): 13,\n",
       "         ('commerce', 'provides'): 1,\n",
       "         ('commerce', 'department'): 4,\n",
       "         ('commerce', 'basic'): 1,\n",
       "         ('provides', 'commerce'): 1,\n",
       "         ('provides', 'basic'): 1,\n",
       "         ('provides', 'of'): 1,\n",
       "         ('provides', 'measures'): 1,\n",
       "         ('basic', 'provides'): 1,\n",
       "         ('basic', 'measures'): 1,\n",
       "         ('basic', 'commerce'): 1,\n",
       "         ('basic', 'of'): 9,\n",
       "         ('measures', 'basic'): 1,\n",
       "         ('measures', 'of'): 2,\n",
       "         ('measures', 'provides'): 1,\n",
       "         ('measures', 'the'): 1,\n",
       "         ('of', 'measures'): 2,\n",
       "         ('of', 'basic'): 9,\n",
       "         ('of', 'national'): 21,\n",
       "         ('the', 'national'): 30,\n",
       "         ('the', 'measures'): 1,\n",
       "         ('the', 'economy'): 11,\n",
       "         ('national', 'the'): 31,\n",
       "         ('national', 'economy'): 1,\n",
       "         ('national', 'of'): 21,\n",
       "         ('national', 'and'): 20,\n",
       "         ('economy', 'national'): 1,\n",
       "         ('economy', 'and'): 3,\n",
       "         ('economy', 'the'): 11,\n",
       "         ('economy', 'current'): 1,\n",
       "         ('and', 'economy'): 3,\n",
       "         ('and', 'current'): 6,\n",
       "         ('and', 'national'): 20,\n",
       "         ('and', 'analysis'): 2,\n",
       "         ('current', 'and'): 6,\n",
       "         ('current', 'analysis'): 1,\n",
       "         ('current', 'economy'): 1,\n",
       "         ('current', 'of'): 10,\n",
       "         ('analysis', 'current'): 1,\n",
       "         ('analysis', 'of'): 5,\n",
       "         ('analysis', 'and'): 2,\n",
       "         ('analysis', 'short-run'): 1,\n",
       "         ('of', 'analysis'): 5,\n",
       "         ('of', 'short-run'): 1,\n",
       "         ('of', 'current'): 10,\n",
       "         ('of', 'changes'): 4,\n",
       "         ('short-run', 'of'): 1,\n",
       "         ('short-run', 'changes'): 1,\n",
       "         ('short-run', 'analysis'): 1,\n",
       "         ('short-run', 'in'): 1,\n",
       "         ('changes', 'short-run'): 1,\n",
       "         ('changes', 'in'): 6,\n",
       "         ('changes', 'of'): 4,\n",
       "         ('changes', 'the'): 5,\n",
       "         ('in', 'changes'): 6,\n",
       "         ('in', 'the'): 438,\n",
       "         ('in', 'short-run'): 1,\n",
       "         ('in', 'economic'): 2,\n",
       "         ('the', 'in'): 466,\n",
       "         ('the', 'economic'): 3,\n",
       "         ('the', 'changes'): 5,\n",
       "         ('the', 'situation'): 7,\n",
       "         ('economic', 'the'): 3,\n",
       "         ('economic', 'situation'): 1,\n",
       "         ('economic', 'in'): 2,\n",
       "         ('economic', 'and'): 10,\n",
       "         ('situation', 'economic'): 1,\n",
       "         ('situation', 'and'): 1,\n",
       "         ('situation', 'the'): 7,\n",
       "         ('situation', 'business'): 1,\n",
       "         ('and', 'situation'): 1,\n",
       "         ('and', 'business'): 20,\n",
       "         ('and', 'economic'): 10,\n",
       "         ('and', 'outlook'): 1,\n",
       "         ('business', 'and'): 21,\n",
       "         ('business', 'outlook'): 2,\n",
       "         ('business', 'situation'): 1,\n",
       "         ('business', '.'): 17,\n",
       "         ('outlook', 'business'): 2,\n",
       "         ('outlook', '.'): 2,\n",
       "         ('outlook', 'and'): 1,\n",
       "         ('develops', 'it'): 1,\n",
       "         ('develops', 'and'): 1,\n",
       "         ('develops', 'analyzes'): 1,\n",
       "         ('and', 'develops'): 1,\n",
       "         ('and', 'analyzes'): 1,\n",
       "         ('and', 'it'): 14,\n",
       "         ('and', 'the'): 364,\n",
       "         ('analyzes', 'and'): 1,\n",
       "         ('analyzes', 'the'): 1,\n",
       "         ('analyzes', 'develops'): 1,\n",
       "         ('analyzes', 'national'): 1,\n",
       "         ('the', 'analyzes'): 1,\n",
       "         ('the', 'and'): 362,\n",
       "         ('the', 'income'): 12,\n",
       "         ('national', 'income'): 1,\n",
       "         ('national', 'analyzes'): 1,\n",
       "         ('national', ','): 10,\n",
       "         ('income', 'national'): 1,\n",
       "         ('income', ','): 4,\n",
       "         ('income', 'the'): 12,\n",
       "         ('income', 'balance'): 1,\n",
       "         (',', 'income'): 4,\n",
       "         (',', 'balance'): 2,\n",
       "         (',', 'national'): 11,\n",
       "         (',', 'of'): 275,\n",
       "         ('balance', ','): 2,\n",
       "         ('balance', 'of'): 5,\n",
       "         ('balance', 'income'): 1,\n",
       "         ('balance', 'international'): 1,\n",
       "         ('of', 'balance'): 5,\n",
       "         ('of', 'international'): 8,\n",
       "         ('of', ','): 274,\n",
       "         ('of', 'payments'): 9,\n",
       "         ('international', 'of'): 8,\n",
       "         ('international', 'payments'): 1,\n",
       "         ('international', 'balance'): 1,\n",
       "         ('international', ','): 11,\n",
       "         ('payments', 'international'): 1,\n",
       "         ('payments', ','): 4,\n",
       "         ('payments', 'of'): 9,\n",
       "         ('payments', 'and'): 5,\n",
       "         (',', 'payments'): 4,\n",
       "         (',', 'and'): 632,\n",
       "         (',', 'international'): 11,\n",
       "         (',', 'many'): 20,\n",
       "         ('and', ','): 616,\n",
       "         ('and', 'many'): 7,\n",
       "         ('and', 'payments'): 5,\n",
       "         ('and', 'other'): 55,\n",
       "         ('many', 'and'): 7,\n",
       "         ('many', 'other'): 6,\n",
       "         ('many', ','): 20,\n",
       "         ('many', 'business'): 1,\n",
       "         ('other', 'many'): 6,\n",
       "         ('other', 'business'): 1,\n",
       "         ('other', 'and'): 56,\n",
       "         ('other', 'indicators'): 1,\n",
       "         ('business', 'other'): 1,\n",
       "         ('business', 'indicators'): 1,\n",
       "         ('business', 'many'): 1,\n",
       "         ('indicators', 'business'): 1,\n",
       "         ('indicators', '.'): 1,\n",
       "         ('indicators', 'other'): 1,\n",
       "         ('measures', 'such'): 2,\n",
       "         ('measures', 'are'): 2,\n",
       "         ('measures', 'essential'): 1,\n",
       "         ('are', 'measures'): 2,\n",
       "         ('are', 'essential'): 4,\n",
       "         ('are', 'such'): 9,\n",
       "         ('are', 'to'): 55,\n",
       "         ('essential', 'are'): 4,\n",
       "         ('essential', 'to'): 4,\n",
       "         ('essential', 'measures'): 1,\n",
       "         ('essential', 'its'): 1,\n",
       "         ('to', 'essential'): 4,\n",
       "         ('to', 'its'): 21,\n",
       "         ('to', 'are'): 55,\n",
       "         ('to', 'job'): 2,\n",
       "         ('its', 'to'): 21,\n",
       "         ('its', 'job'): 1,\n",
       "         ('its', 'essential'): 1,\n",
       "         ('its', 'of'): 30,\n",
       "         ('job', 'its'): 1,\n",
       "         ('job', 'of'): 2,\n",
       "         ('job', 'to'): 2,\n",
       "         ('job', 'presenting'): 1,\n",
       "         ('of', 'job'): 2,\n",
       "         ('of', 'presenting'): 1,\n",
       "         ('of', 'its'): 30,\n",
       "         ('presenting', 'of'): 1,\n",
       "         ('presenting', 'business'): 1,\n",
       "         ('presenting', 'job'): 1,\n",
       "         ('presenting', 'and'): 1,\n",
       "         ('business', 'presenting'): 1,\n",
       "         ('business', 'government'): 2,\n",
       "         ('and', 'government'): 9,\n",
       "         ('and', 'presenting'): 1,\n",
       "         ('and', 'with'): 31,\n",
       "         ('government', 'and'): 9,\n",
       "         ('government', 'with'): 4,\n",
       "         ('government', 'business'): 2,\n",
       "         ('government', 'the'): 99,\n",
       "         ('with', 'government'): 4,\n",
       "         ('with', 'the'): 112,\n",
       "         ('with', 'and'): 30,\n",
       "         ('with', 'facts'): 1,\n",
       "         ('the', 'with'): 115,\n",
       "         ('the', 'facts'): 6,\n",
       "         ('the', 'government'): 88,\n",
       "         ('the', 'required'): 7,\n",
       "         ('facts', 'the'): 6,\n",
       "         ('facts', 'required'): 1,\n",
       "         ('facts', 'with'): 1,\n",
       "         ('facts', 'to'): 1,\n",
       "         ('required', 'facts'): 1,\n",
       "         ('required', 'to'): 19,\n",
       "         ('required', 'the'): 7,\n",
       "         ('required', 'meet'): 1,\n",
       "         ('to', 'required'): 19,\n",
       "         ('to', 'meet'): 11,\n",
       "         ('to', 'facts'): 1,\n",
       "         ('to', 'the'): 568,\n",
       "         ('meet', 'to'): 12,\n",
       "         ('meet', 'the'): 5,\n",
       "         ('meet', 'required'): 1,\n",
       "         ('meet', 'objective'): 1,\n",
       "         ('the', 'meet'): 5,\n",
       "         ('the', 'objective'): 6,\n",
       "         ('the', 'to'): 571,\n",
       "         ('objective', 'the'): 12,\n",
       "         ('objective', 'of'): 7,\n",
       "         ('objective', 'meet'): 1,\n",
       "         ('objective', 'expanding'): 1,\n",
       "         ('of', 'objective'): 7,\n",
       "         ('of', 'expanding'): 1,\n",
       "         ('expanding', 'of'): 1,\n",
       "         ('expanding', 'business'): 1,\n",
       "         ('expanding', 'objective'): 1,\n",
       "         ('expanding', 'and'): 1,\n",
       "         ('business', 'expanding'): 1,\n",
       "         ('business', 'improving'): 1,\n",
       "         ('and', 'improving'): 1,\n",
       "         ('and', 'expanding'): 1,\n",
       "         ('improving', 'and'): 1,\n",
       "         ('improving', 'the'): 3,\n",
       "         ('improving', 'business'): 1,\n",
       "         ('improving', 'operation'): 1,\n",
       "         ('the', 'improving'): 3,\n",
       "         ('the', 'operation'): 15,\n",
       "         ('operation', 'the'): 15,\n",
       "         ('operation', 'of'): 9,\n",
       "         ('operation', 'improving'): 1,\n",
       "         ('of', 'operation'): 9,\n",
       "         ('of', 'economy'): 7,\n",
       "         ('the', '.'): 176,\n",
       "         ('economy', '.'): 3,\n",
       "         ('economy', 'of'): 7,\n",
       "         ('further', 'for'): 7,\n",
       "         ('further', 'information'): 4,\n",
       "         ('further', 'contact'): 1,\n",
       "         ('information', 'further'): 4,\n",
       "         ('information', 'contact'): 4,\n",
       "         ('information', 'for'): 7,\n",
       "         ('information', 'director'): 1,\n",
       "         ('contact', 'information'): 4,\n",
       "         ('contact', 'director'): 1,\n",
       "         ('contact', 'further'): 1,\n",
       "         ('contact', ','): 4,\n",
       "         ('director', 'contact'): 1,\n",
       "         ('director', ','): 6,\n",
       "         ('director', 'information'): 1,\n",
       "         ('director', 'office'): 1,\n",
       "         (',', 'director'): 6,\n",
       "         (',', 'office'): 8,\n",
       "         (',', 'contact'): 4,\n",
       "         ('office', ','): 8,\n",
       "         ('office', 'director'): 1,\n",
       "         ('business', ','): 28,\n",
       "         ('economics', ','): 2,\n",
       "         ('economics', 'u.s.'): 1,\n",
       "         (',', 'economics'): 2,\n",
       "         (',', 'u.s.'): 17,\n",
       "         (',', 'business'): 28,\n",
       "         (',', 'department'): 17,\n",
       "         ('u.s.', ','): 16,\n",
       "         ('u.s.', 'economics'): 1,\n",
       "         ('department', ','): 17,\n",
       "         ('commerce', ','): 8,\n",
       "         ('commerce', 'washington'): 1,\n",
       "         (',', 'commerce'): 8,\n",
       "         (',', 'washington'): 32,\n",
       "         (',', '25'): 25,\n",
       "         ('washington', ','): 32,\n",
       "         ('washington', '25'): 9,\n",
       "         ('washington', 'commerce'): 1,\n",
       "         ('25', 'washington'): 9,\n",
       "         ('25', ','): 25,\n",
       "         ('25', 'd.c.'): 9,\n",
       "         (',', 'd.c.'): 16,\n",
       "         (',', '.'): 42,\n",
       "         ('d.c.', ','): 16,\n",
       "         ('d.c.', '.'): 7,\n",
       "         ('d.c.', '25'): 9,\n",
       "         ('information', 'economic'): 1,\n",
       "         ('information', 'is'): 2,\n",
       "         ('information', 'made'): 1,\n",
       "         ('is', 'information'): 2,\n",
       "         ('is', 'made'): 7,\n",
       "         ('is', 'economic'): 2,\n",
       "         ('is', 'available'): 5,\n",
       "         ('made', 'is'): 7,\n",
       "         ('made', 'available'): 5,\n",
       "         ('made', 'information'): 1,\n",
       "         ('made', 'to'): 39,\n",
       "         ('available', 'made'): 5,\n",
       "         ('available', 'to'): 14,\n",
       "         ('available', 'is'): 5,\n",
       "         ('available', 'businessmen'): 1,\n",
       "         ('to', 'available'): 14,\n",
       "         ('to', 'businessmen'): 1,\n",
       "         ('to', 'made'): 39,\n",
       "         ('to', 'and'): 119,\n",
       "         ('businessmen', 'to'): 1,\n",
       "         ('businessmen', 'and'): 2,\n",
       "         ('businessmen', 'available'): 1,\n",
       "         ('businessmen', 'economists'): 1,\n",
       "         ('and', 'businessmen'): 2,\n",
       "         ('and', 'economists'): 2,\n",
       "         ('and', 'to'): 118,\n",
       "         ('and', 'promptly'): 1,\n",
       "         ('economists', 'and'): 2,\n",
       "         ('economists', 'promptly'): 1,\n",
       "         ('economists', 'businessmen'): 1,\n",
       "         ('economists', 'through'): 1,\n",
       "         ('promptly', 'economists'): 1,\n",
       "         ('promptly', 'through'): 1,\n",
       "         ('promptly', 'and'): 1,\n",
       "         ('promptly', 'the'): 1,\n",
       "         ('through', 'promptly'): 1,\n",
       "         ('through', 'the'): 22,\n",
       "         ('through', 'economists'): 1,\n",
       "         ('through', 'monthly'): 1,\n",
       "         ('the', 'through'): 24,\n",
       "         ('the', 'monthly'): 1,\n",
       "         ('the', 'promptly'): 1,\n",
       "         ('the', 'survey'): 1,\n",
       "         ('monthly', 'the'): 1,\n",
       "         ('monthly', 'survey'): 1,\n",
       "         ('monthly', 'through'): 1,\n",
       "         ('monthly', 'of'): 1,\n",
       "         ('survey', 'monthly'): 1,\n",
       "         ('survey', 'of'): 2,\n",
       "         ('survey', 'the'): 1,\n",
       "         ('survey', 'current'): 1,\n",
       "         ('of', 'survey'): 2,\n",
       "         ('of', 'monthly'): 1,\n",
       "         ('current', 'business'): 1,\n",
       "         ('current', 'survey'): 1,\n",
       "         ('business', 'current'): 1,\n",
       "         ('business', 'its'): 1,\n",
       "         ('and', 'its'): 17,\n",
       "         ('and', 'weekly'): 1,\n",
       "         ('its', 'and'): 17,\n",
       "         ('its', 'weekly'): 1,\n",
       "         ('its', 'business'): 1,\n",
       "         ('its', 'supplement'): 1,\n",
       "         ('weekly', 'its'): 1,\n",
       "         ('weekly', 'supplement'): 1,\n",
       "         ('weekly', 'and'): 1,\n",
       "         ('weekly', '.'): 2,\n",
       "         ('supplement', 'weekly'): 1,\n",
       "         ('supplement', '.'): 1,\n",
       "         ('supplement', 'its'): 2,\n",
       "         ('periodical', 'this'): 1,\n",
       "         ('periodical', ','): 2,\n",
       "         ('periodical', 'including'): 1,\n",
       "         (',', 'periodical'): 2,\n",
       "         (',', 'including'): 21,\n",
       "         (',', 'this'): 88,\n",
       "         (',', 'weekly'): 1,\n",
       "         ('including', ','): 21,\n",
       "         ('including', 'weekly'): 1,\n",
       "         ('including', 'periodical'): 1,\n",
       "         ('including', 'statistical'): 1,\n",
       "         ('weekly', 'including'): 1,\n",
       "         ('weekly', 'statistical'): 1,\n",
       "         ('weekly', ','): 1,\n",
       "         ('weekly', 'supplements'): 1,\n",
       "         ('statistical', 'weekly'): 1,\n",
       "         ('statistical', 'supplements'): 1,\n",
       "         ('statistical', 'including'): 1,\n",
       "         ('statistical', ','): 1,\n",
       "         ('supplements', 'statistical'): 1,\n",
       "         ('supplements', ','): 1,\n",
       "         ('supplements', 'weekly'): 1,\n",
       "         ('supplements', 'is'): 1,\n",
       "         (',', 'supplements'): 1,\n",
       "         (',', 'is'): 93,\n",
       "         (',', 'statistical'): 1,\n",
       "         (',', 'available'): 8,\n",
       "         ('is', ','): 93,\n",
       "         ('is', 'supplements'): 1,\n",
       "         ('is', 'for'): 23,\n",
       "         ('available', 'for'): 10,\n",
       "         ('available', ','): 7,\n",
       "         ('available', '$4'): 1,\n",
       "         ('for', 'available'): 10,\n",
       "         ('for', '$4'): 2,\n",
       "         ('for', 'is'): 22,\n",
       "         ('for', 'per'): 4,\n",
       "         ('$4', 'for'): 2,\n",
       "         ('$4', 'per'): 1,\n",
       "         ('$4', 'available'): 1,\n",
       "         ('$4', 'year'): 1,\n",
       "         ('per', '$4'): 1,\n",
       "         ('per', 'year'): 6,\n",
       "         ('per', 'for'): 4,\n",
       "         ('per', 'from'): 1,\n",
       "         ('year', 'per'): 6,\n",
       "         ('year', 'from'): 1,\n",
       "         ('year', '$4'): 1,\n",
       "         ('year', 'commerce'): 1,\n",
       "         ('from', 'year'): 1,\n",
       "         ('from', 'commerce'): 1,\n",
       "         ('from', 'per'): 1,\n",
       "         ('from', 'field'): 1,\n",
       "         ('commerce', 'from'): 1,\n",
       "         ('commerce', 'field'): 1,\n",
       "         ('commerce', 'year'): 1,\n",
       "         ('commerce', 'offices'): 1,\n",
       "         ('field', 'commerce'): 1,\n",
       "         ('field', 'offices'): 1,\n",
       "         ('field', 'from'): 1,\n",
       "         ('field', 'or'): 1,\n",
       "         ('offices', 'field'): 1,\n",
       "         ('offices', 'or'): 1,\n",
       "         ('offices', 'commerce'): 1,\n",
       "         ('offices', 'superintendent'): 1,\n",
       "         ('or', 'offices'): 1,\n",
       "         ('or', 'superintendent'): 1,\n",
       "         ('or', 'field'): 1,\n",
       "         ('or', 'of'): 41,\n",
       "         ('superintendent', 'or'): 1,\n",
       "         ('superintendent', 'of'): 3,\n",
       "         ('superintendent', 'offices'): 1,\n",
       "         ('superintendent', 'documents'): 2,\n",
       "         ('of', 'superintendent'): 3,\n",
       "         ('of', 'documents'): 3,\n",
       "         ('of', 'or'): 41,\n",
       "         ('documents', 'of'): 3,\n",
       "         ('documents', ','): 3,\n",
       "         ('documents', 'superintendent'): 2,\n",
       "         ('documents', 'u.s.'): 2,\n",
       "         (',', 'documents'): 3,\n",
       "         (',', 'government'): 12,\n",
       "         ('u.s.', 'government'): 3,\n",
       "         ('u.s.', 'documents'): 2,\n",
       "         ('u.s.', 'printing'): 2,\n",
       "         ('government', 'u.s.'): 4,\n",
       "         ('government', 'printing'): 2,\n",
       "         ('government', ','): 12,\n",
       "         ('government', 'office'): 2,\n",
       "         ('printing', 'government'): 2,\n",
       "         ('printing', 'office'): 2,\n",
       "         ('printing', 'u.s.'): 2,\n",
       "         ('printing', ','): 5,\n",
       "         ('office', 'printing'): 2,\n",
       "         ('office', 'government'): 2,\n",
       "         ('office', 'washington'): 2,\n",
       "         (',', 'printing'): 5,\n",
       "         ('washington', 'office'): 2,\n",
       "         ('assistance', 'technical'): 4,\n",
       "         ('assistance', 'to'): 7,\n",
       "         ('assistance', 'small'): 1,\n",
       "         ('to', 'assistance'): 7,\n",
       "         ('to', 'small'): 14,\n",
       "         ('to', 'technical'): 3,\n",
       "         ('to', 'business'): 9,\n",
       "         ('small', 'to'): 15,\n",
       "         ('small', 'business'): 36,\n",
       "         ('small', 'assistance'): 1,\n",
       "         ('small', 'community'): 1,\n",
       "         ('business', 'small'): 37,\n",
       "         ('business', 'community'): 2,\n",
       "         ('business', 'to'): 8,\n",
       "         ('small', 'the'): 13,\n",
       "         ('small', 'administration'): 7,\n",
       "         ('business', 'administration'): 9,\n",
       "         ('business', 'the'): 26,\n",
       "         ('administration', 'business'): 9,\n",
       "         ('administration', '('): 1,\n",
       "         ('administration', 'small'): 9,\n",
       "         ('administration', 'sba'): 1,\n",
       "         ('(', 'administration'): 1,\n",
       "         ('(', 'sba'): 1,\n",
       "         ('sba', '('): 1,\n",
       "         ('sba', ')'): 1,\n",
       "         ('sba', 'administration'): 1,\n",
       "         ('sba', 'provides'): 1,\n",
       "         (')', 'sba'): 1,\n",
       "         (')', 'provides'): 1,\n",
       "         (')', 'guidance'): 1,\n",
       "         ('provides', ')'): 1,\n",
       "         ('provides', 'guidance'): 1,\n",
       "         ('provides', 'sba'): 1,\n",
       "         ('provides', 'and'): 2,\n",
       "         ('guidance', 'provides'): 1,\n",
       "         ('guidance', 'and'): 2,\n",
       "         ('guidance', ')'): 1,\n",
       "         ('guidance', 'advice'): 1,\n",
       "         ('and', 'guidance'): 3,\n",
       "         ('and', 'advice'): 2,\n",
       "         ('and', 'provides'): 2,\n",
       "         ('and', 'on'): 28,\n",
       "         ('advice', 'and'): 2,\n",
       "         ('advice', 'on'): 2,\n",
       "         ('advice', 'guidance'): 2,\n",
       "         ('advice', 'sources'): 1,\n",
       "         ('on', 'advice'): 2,\n",
       "         ('on', 'sources'): 4,\n",
       "         ('on', 'and'): 30,\n",
       "         ('on', 'of'): 39,\n",
       "         ('sources', 'on'): 4,\n",
       "         ('sources', 'of'): 8,\n",
       "         ('sources', 'advice'): 1,\n",
       "         ('sources', 'technical'): 1,\n",
       "         ('of', 'sources'): 8,\n",
       "         ('of', 'technical'): 3,\n",
       "         ('of', 'on'): 40,\n",
       "         ('of', 'information'): 9,\n",
       "         ('technical', 'of'): 3,\n",
       "         ('technical', 'information'): 2,\n",
       "         ('technical', 'sources'): 1,\n",
       "         ('technical', 'relating'): 1,\n",
       "         ('information', 'technical'): 2,\n",
       "         ('information', 'relating'): 2,\n",
       "         ('information', 'of'): 9,\n",
       "         ('information', 'to'): 7,\n",
       "         ('relating', 'information'): 2,\n",
       "         ('relating', 'to'): 8,\n",
       "         ('relating', 'technical'): 1,\n",
       "         ('relating', 'small'): 1,\n",
       "         ('to', 'relating'): 8,\n",
       "         ('to', 'information'): 7,\n",
       "         ('small', 'relating'): 1,\n",
       "         ('small', 'management'): 1,\n",
       "         ('business', 'management'): 4,\n",
       "         ('management', 'business'): 3,\n",
       "         ('management', 'and'): 12,\n",
       "         ('management', 'small'): 1,\n",
       "         ('management', 'research'): 1,\n",
       "         ('and', 'management'): 12,\n",
       "         ('and', 'research'): 32,\n",
       "         ('and', 'and'): 50,\n",
       "         ('research', 'and'): 29,\n",
       "         ('research', 'management'): 1,\n",
       "         ('research', 'development'): 12,\n",
       "         ('and', 'development'): 42,\n",
       "         ('and', 'of'): 268,\n",
       "         ('development', 'and'): 40,\n",
       "         ('development', 'of'): 45,\n",
       "         ('development', 'research'): 14,\n",
       "         ('development', 'products'): 1,\n",
       "         ('of', 'development'): 45,\n",
       "         ('of', 'products'): 7,\n",
       "         ('of', 'and'): 269,\n",
       "         ('of', '.'): 132,\n",
       "         ('products', 'of'): 7,\n",
       "         ('products', '.'): 4,\n",
       "         ('products', 'development'): 1,\n",
       "         ('management', 'practical'): 1,\n",
       "         ('management', 'problems'): 1,\n",
       "         ('problems', 'management'): 1,\n",
       "         ('problems', 'and'): 6,\n",
       "         ('problems', 'practical'): 1,\n",
       "         ('problems', 'their'): 2,\n",
       "         ('and', 'problems'): 6,\n",
       "         ('and', 'their'): 21,\n",
       "         ('and', 'suggested'): 1,\n",
       "         ('their', 'and'): 22,\n",
       "         ('their', 'suggested'): 1,\n",
       "         ('their', 'problems'): 2,\n",
       "         ('their', 'solutions'): 1,\n",
       "         ('suggested', 'their'): 1,\n",
       "         ('suggested', 'solutions'): 1,\n",
       "         ('suggested', 'and'): 1,\n",
       "         ('suggested', 'are'): 1,\n",
       "         ('solutions', 'suggested'): 1,\n",
       "         ('solutions', 'are'): 1,\n",
       "         ('solutions', 'their'): 1,\n",
       "         ('solutions', 'dealt'): 1,\n",
       "         ('are', 'solutions'): 1,\n",
       "         ('are', 'dealt'): 1,\n",
       "         ('are', 'suggested'): 1,\n",
       "         ('are', 'with'): 5,\n",
       "         ('dealt', 'are'): 1,\n",
       "         ('dealt', 'with'): 2,\n",
       "         ('dealt', 'solutions'): 1,\n",
       "         ('dealt', 'in'): 1,\n",
       "         ('with', 'dealt'): 2,\n",
       "         ('with', 'in'): 35,\n",
       "         ('with', 'are'): 5,\n",
       "         ('with', 'a'): 26,\n",
       "         ('in', 'with'): 29,\n",
       "         ('in', 'a'): 65,\n",
       "         ('in', 'dealt'): 1,\n",
       "         ('in', 'series'): 2,\n",
       "         ('a', 'in'): 69,\n",
       "         ('a', 'series'): 7,\n",
       "         ('a', 'with'): 27,\n",
       "         ('a', 'of'): 226,\n",
       "         ('series', 'a'): 7,\n",
       "         ('series', 'of'): 8,\n",
       "         ('series', 'in'): 2,\n",
       "         ('series', 'sba'): 1,\n",
       "         ('of', 'series'): 8,\n",
       "         ('of', 'sba'): 1,\n",
       "         ('of', 'a'): 234,\n",
       "         ('of', 'publications'): 3,\n",
       "         ('sba', 'of'): 1,\n",
       "         ('sba', 'publications'): 1,\n",
       "         ('sba', 'series'): 1,\n",
       "         ('sba', '.'): 2,\n",
       "         ('publications', 'sba'): 1,\n",
       "         ('publications', '.'): 5,\n",
       "         ('publications', 'of'): 3,\n",
       "         ('publications', 'these'): 3,\n",
       "         ('publications', ','): 2,\n",
       "         ('publications', 'written'): 1,\n",
       "         (',', 'publications'): 2,\n",
       "         (',', 'written'): 1,\n",
       "         (',', 'these'): 30,\n",
       "         (',', 'especially'): 3,\n",
       "         ('written', ','): 1,\n",
       "         ('written', 'especially'): 1,\n",
       "         ('written', 'publications'): 1,\n",
       "         ('written', 'for'): 1,\n",
       "         ('especially', 'written'): 1,\n",
       "         ('especially', 'for'): 2,\n",
       "         ('especially', ','): 3,\n",
       "         ('especially', 'the'): 1,\n",
       "         ('for', 'especially'): 2,\n",
       "         ('for', 'the'): 261,\n",
       "         ('for', 'written'): 1,\n",
       "         ('for', 'managers'): 1,\n",
       "         ('the', 'for'): 262,\n",
       "         ('the', 'managers'): 1,\n",
       "         ('the', 'especially'): 1,\n",
       "         ('the', 'or'): 66,\n",
       "         ('managers', 'the'): 1,\n",
       "         ('managers', 'or'): 1,\n",
       "         ('managers', 'for'): 1,\n",
       "         ('managers', 'owners'): 1,\n",
       "         ('or', 'managers'): 1,\n",
       "         ('or', 'owners'): 1,\n",
       "         ('or', 'the'): 70,\n",
       "         ('owners', 'or'): 1,\n",
       "         ('owners', 'of'): 2,\n",
       "         ('owners', 'managers'): 1,\n",
       "         ('owners', 'small'): 2,\n",
       "         ('of', 'owners'): 2,\n",
       "         ('of', 'small'): 12,\n",
       "         ('of', 'businesses'): 2,\n",
       "         ('small', 'of'): 12,\n",
       "         ('small', 'businesses'): 2,\n",
       "         ('small', 'owners'): 2,\n",
       "         ('small', ','): 12,\n",
       "         ('businesses', 'small'): 2,\n",
       "         ('businesses', ','): 2,\n",
       "         ('businesses', 'of'): 2,\n",
       "         ('businesses', 'indirectly'): 1,\n",
       "         (',', 'businesses'): 2,\n",
       "         (',', 'indirectly'): 2,\n",
       "         (',', 'small'): 12,\n",
       "         (',', 'aid'): 6,\n",
       "         ('indirectly', ','): 2,\n",
       "         ('indirectly', 'aid'): 1,\n",
       "         ('indirectly', 'businesses'): 1,\n",
       "         ('indirectly', 'in'): 1,\n",
       "         ('aid', 'indirectly'): 1,\n",
       "         ('aid', 'in'): 5,\n",
       "         ('aid', ','): 6,\n",
       "         ('aid', 'community'): 2,\n",
       "         ('in', 'aid'): 5,\n",
       "         ('in', 'community'): 3,\n",
       "         ('in', 'indirectly'): 1,\n",
       "         ('in', 'development'): 15,\n",
       "         ('community', 'in'): 3,\n",
       "         ('community', 'development'): 4,\n",
       "         ('community', 'aid'): 2,\n",
       "         ('community', 'programs'): 2,\n",
       "         ('development', 'community'): 4,\n",
       "         ('development', 'programs'): 7,\n",
       "         ('development', 'in'): 15,\n",
       "         ('development', '.'): 15,\n",
       "         ('programs', 'development'): 7,\n",
       "         ('programs', '.'): 8,\n",
       "         ('programs', 'community'): 2,\n",
       "         ('are', 'they'): 17,\n",
       "         ('are', 'written'): 1,\n",
       "         ('are', 'by'): 20,\n",
       "         ('written', 'are'): 1,\n",
       "         ('written', 'by'): 1,\n",
       "         ('written', 'they'): 1,\n",
       "         ('written', 'specialists'): 1,\n",
       "         ('by', 'written'): 1,\n",
       "         ('by', 'specialists'): 1,\n",
       "         ('by', 'are'): 20,\n",
       "         ('by', 'in'): 13,\n",
       "         ('specialists', 'by'): 1,\n",
       "         ('specialists', 'in'): 3,\n",
       "         ('specialists', 'written'): 1,\n",
       "         ('specialists', 'numerous'): 1,\n",
       "         ('in', 'specialists'): 3,\n",
       "         ('in', 'numerous'): 1,\n",
       "         ('in', 'by'): 13,\n",
       "         ('in', 'types'): 1,\n",
       "         ('numerous', 'in'): 1,\n",
       "         ('numerous', 'types'): 1,\n",
       "         ('numerous', 'specialists'): 1,\n",
       "         ('numerous', 'of'): 2,\n",
       "         ('types', 'numerous'): 1,\n",
       "         ('types', 'of'): 14,\n",
       "         ('types', 'in'): 1,\n",
       "         ('types', 'business'): 1,\n",
       "         ('of', 'types'): 16,\n",
       "         ('of', 'numerous'): 2,\n",
       "         ('of', 'enterprises'): 1,\n",
       "         ('business', 'enterprises'): 2,\n",
       "         ('business', 'types'): 1,\n",
       "         ('enterprises', 'business'): 2,\n",
       "         ('enterprises', ','): 1,\n",
       "         ('enterprises', 'of'): 1,\n",
       "         ('enterprises', 'cover'): 1,\n",
       "         (',', 'enterprises'): 1,\n",
       "         (',', 'cover'): 3,\n",
       "         (',', 'a'): 141,\n",
       "         ('cover', ','): 3,\n",
       "         ('cover', 'a'): 1,\n",
       "         ('cover', 'enterprises'): 1,\n",
       "         ('cover', 'wide'): 1,\n",
       "         ('a', 'cover'): 1,\n",
       "         ('a', 'wide'): 5,\n",
       "         ('a', ','): 140,\n",
       "         ('a', 'range'): 2,\n",
       "         ('wide', 'a'): 5,\n",
       "         ('wide', 'range'): 1,\n",
       "         ('wide', 'cover'): 1,\n",
       "         ('wide', 'of'): 5,\n",
       "         ('range', 'wide'): 1,\n",
       "         ('range', 'of'): 8,\n",
       "         ('range', 'a'): 2,\n",
       "         ('range', 'subjects'): 1,\n",
       "         ('of', 'range'): 8,\n",
       "         ('of', 'subjects'): 4,\n",
       "         ('of', 'wide'): 5,\n",
       "         ('subjects', 'of'): 4,\n",
       "         ('subjects', ','): 1,\n",
       "         ('subjects', 'range'): 1,\n",
       "         ('subjects', 'and'): 3,\n",
       "         (',', 'subjects'): 1,\n",
       "         (',', 'are'): 67,\n",
       "         ('and', 'are'): 33,\n",
       "         ('and', 'subjects'): 3,\n",
       "         ('and', 'directed'): 8,\n",
       "         ('are', 'and'): 34,\n",
       "         ('are', 'directed'): 2,\n",
       "         ('are', ','): 67,\n",
       "         ('directed', 'are'): 2,\n",
       "         ('directed', 'to'): 9,\n",
       "         ('directed', 'and'): 8,\n",
       "         ('directed', 'the'): 5,\n",
       "         ('to', 'directed'): 9,\n",
       "         ('to', 'needs'): 3,\n",
       "         ('the', 'needs'): 7,\n",
       "         ('the', 'directed'): 5,\n",
       "         ('needs', 'the'): 7,\n",
       "         ('needs', 'and'): 3,\n",
       "         ('needs', 'to'): 3,\n",
       "         ('needs', 'interests'): 1,\n",
       "         ('and', 'needs'): 3,\n",
       "         ('and', 'interests'): 3,\n",
       "         ('interests', 'and'): 3,\n",
       "         ('interests', 'of'): 8,\n",
       "         ('interests', 'needs'): 1,\n",
       "         ('interests', 'the'): 8,\n",
       "         ('of', 'interests'): 8,\n",
       "         ('the', 'small'): 12,\n",
       "         ('the', 'interests'): 8,\n",
       "         ('the', 'firm'): 4,\n",
       "         ('small', 'firm'): 1,\n",
       "         ('small', '.'): 5,\n",
       "         ('firm', 'small'): 1,\n",
       "         ('firm', '.'): 1,\n",
       "         ('firm', 'the'): 4,\n",
       "         ('offers', 'sba'): 1,\n",
       "         ('offers', 'administrative'): 1,\n",
       "         ('offers', 'management'): 1,\n",
       "         ('administrative', 'offers'): 1,\n",
       "         ('administrative', 'management'): 1,\n",
       "         ('administrative', 'sba'): 1,\n",
       "         ('administrative', 'courses'): 1,\n",
       "         ('management', 'administrative'): 1,\n",
       "         ('management', 'courses'): 1,\n",
       "         ('management', 'offers'): 1,\n",
       "         ('management', ','): 10,\n",
       "         ('courses', 'management'): 1,\n",
       "         ('courses', ','): 4,\n",
       "         ('courses', 'administrative'): 1,\n",
       "         ('courses', 'which'): 1,\n",
       "         (',', 'courses'): 4,\n",
       "         (',', 'which'): 54,\n",
       "         (',', 'management'): 10,\n",
       "         ('which', ','): 54,\n",
       "         ('which', 'are'): 23,\n",
       "         ('which', 'courses'): 1,\n",
       "         ('which', 'designed'): 2,\n",
       "         ('are', 'which'): 23,\n",
       "         ('are', 'designed'): 5,\n",
       "         ('designed', 'are'): 5,\n",
       "         ('designed', 'to'): 17,\n",
       "         ('designed', 'which'): 2,\n",
       "         ('designed', 'improve'): 1,\n",
       "         ('to', 'designed'): 17,\n",
       "         ('to', 'improve'): 2,\n",
       "         ('improve', 'to'): 2,\n",
       "         ('improve', 'the'): 2,\n",
       "         ('improve', 'designed'): 1,\n",
       "         ('improve', 'management'): 1,\n",
       "         ('the', 'improve'): 2,\n",
       "         ('the', 'management'): 8,\n",
       "         ('the', 'efficiency'): 1,\n",
       "         ('management', 'the'): 9,\n",
       "         ('management', 'efficiency'): 1,\n",
       "         ('management', 'improve'): 1,\n",
       "         ('efficiency', 'management'): 1,\n",
       "         ('efficiency', 'and'): 3,\n",
       "         ('efficiency', 'the'): 1,\n",
       "         ('efficiency', '``'): 1,\n",
       "         ('and', 'efficiency'): 3,\n",
       "         ('and', '``'): 17,\n",
       "         ('and', 'know-how'): 1,\n",
       "         ('``', 'and'): 17,\n",
       "         ('``', 'know-how'): 1,\n",
       "         ('``', 'efficiency'): 1,\n",
       "         ('``', \"''\"): 37,\n",
       "         ('know-how', '``'): 1,\n",
       "         ('know-how', \"''\"): 1,\n",
       "         ('know-how', 'and'): 1,\n",
       "         ('know-how', 'of'): 1,\n",
       "         (\"''\", 'know-how'): 1,\n",
       "         (\"''\", 'of'): 7,\n",
       "         (\"''\", '``'): 37,\n",
       "         (\"''\", 'small'): 1,\n",
       "         ('of', \"''\"): 7,\n",
       "         ('of', 'know-how'): 1,\n",
       "         ('small', \"''\"): 1,\n",
       "         ('small', 'concerns'): 13,\n",
       "         ('business', 'concerns'): 13,\n",
       "         ('business', 'within'): 1,\n",
       "         ('concerns', 'business'): 13,\n",
       "         ('concerns', 'within'): 1,\n",
       "         ('concerns', 'small'): 13,\n",
       "         ('concerns', 'a'): 1,\n",
       "         ('within', 'concerns'): 1,\n",
       "         ('within', 'a'): 6,\n",
       "         ('within', 'business'): 1,\n",
       "         ('within', 'community'): 1,\n",
       "         ('a', 'within'): 8,\n",
       "         ('a', 'community'): 2,\n",
       "         ('a', 'concerns'): 1,\n",
       "         ('a', '.'): 19,\n",
       "         ('community', 'a'): 2,\n",
       "         ('community', '.'): 6,\n",
       "         ('community', 'within'): 1,\n",
       "         ('cosponsors', 'sba'): 1,\n",
       "         ('cosponsors', 'these'): 1,\n",
       "         ('cosponsors', 'courses'): 1,\n",
       "         ('these', 'cosponsors'): 1,\n",
       "         ('these', 'courses'): 1,\n",
       "         ('these', 'sba'): 1,\n",
       "         ('these', 'with'): 5,\n",
       "         ('courses', 'these'): 1,\n",
       "         ('courses', 'with'): 1,\n",
       "         ('courses', 'cosponsors'): 1,\n",
       "         ('courses', 'educational'): 1,\n",
       "         ('with', 'courses'): 1,\n",
       "         ('with', 'educational'): 2,\n",
       "         ('with', 'these'): 4,\n",
       "         ('with', 'institutions'): 2,\n",
       "         ('educational', 'with'): 2,\n",
       "         ('educational', 'institutions'): 2,\n",
       "         ('educational', 'courses'): 1,\n",
       "         ('educational', 'and'): 6,\n",
       "         ('institutions', 'educational'): 2,\n",
       "         ('institutions', 'and'): 8,\n",
       "         ('institutions', 'with'): 2,\n",
       "         ('institutions', 'community'): 1,\n",
       "         ('and', 'institutions'): 8,\n",
       "         ('and', 'community'): 3,\n",
       "         ('and', 'educational'): 6,\n",
       "         ('and', 'groups'): 3,\n",
       "         ('community', 'and'): 3,\n",
       "         ('community', 'groups'): 1,\n",
       "         ('community', 'institutions'): 1,\n",
       "         ('groups', 'community'): 1,\n",
       "         ('groups', '.'): 3,\n",
       "         ('groups', 'and'): 3,\n",
       "         ('the', \"sba's\"): 1,\n",
       "         (\"sba's\", 'the'): 1,\n",
       "         (\"sba's\", 'management'): 1,\n",
       "         (\"sba's\", 'through'): 1,\n",
       "         (\"sba's\", 'counseling'): 1,\n",
       "         ('management', \"sba's\"): 1,\n",
       "         ('management', 'counseling'): 1,\n",
       "         ('management', 'program'): 1,\n",
       "         ('counseling', 'management'): 1,\n",
       "         ('counseling', 'program'): 1,\n",
       "         ('counseling', \"sba's\"): 1,\n",
       "         ('counseling', ','): 3,\n",
       "         ('program', 'counseling'): 1,\n",
       "         ('program', ','): 20,\n",
       "         ('program', 'management'): 1,\n",
       "         ('program', 'practical'): 1,\n",
       "         (',', 'program'): 20,\n",
       "         (',', 'practical'): 2,\n",
       "         (',', 'counseling'): 3,\n",
       "         (',', ','): 840,\n",
       "         ('practical', ','): 2,\n",
       "         ('practical', 'program'): 1,\n",
       "         ('practical', 'personalized'): 1,\n",
       "         (',', 'personalized'): 1,\n",
       "         (',', 'advice'): 3,\n",
       "         ('personalized', ','): 1,\n",
       "         ('personalized', 'advice'): 1,\n",
       "         ('personalized', 'practical'): 1,\n",
       "         ('personalized', 'on'): 1,\n",
       "         ('advice', 'personalized'): 1,\n",
       "         ('advice', ','): 3,\n",
       "         ('advice', 'sound'): 1,\n",
       "         ('on', 'sound'): 3,\n",
       "         ('on', 'personalized'): 1,\n",
       "         ('on', 'management'): 1,\n",
       "         ('sound', 'on'): 3,\n",
       "         ('sound', 'management'): 1,\n",
       "         ('sound', 'advice'): 1,\n",
       "         ('sound', 'principles'): 1,\n",
       "         ('management', 'sound'): 1,\n",
       "         ('management', 'principles'): 1,\n",
       "         ('management', 'on'): 1,\n",
       "         ('management', 'is'): 3,\n",
       "         ('principles', 'management'): 1,\n",
       "         ('principles', 'is'): 1,\n",
       "         ('principles', 'sound'): 1,\n",
       "         ('principles', 'available'): 1,\n",
       "         ('is', 'principles'): 1,\n",
       "         ('is', 'management'): 3,\n",
       "         ('is', 'upon'): 1,\n",
       "         ('available', 'upon'): 1,\n",
       "         ('available', 'principles'): 1,\n",
       "         ('available', 'request'): 1,\n",
       "         ('upon', 'available'): 1,\n",
       "         ('upon', 'request'): 4,\n",
       "         ('upon', 'is'): 1,\n",
       "         ('upon', 'to'): 4,\n",
       "         ('request', 'upon'): 4,\n",
       "         ('request', 'to'): 2,\n",
       "         ('request', 'available'): 1,\n",
       "         ('request', 'both'): 1,\n",
       "         ('to', 'request'): 2,\n",
       "         ('to', 'both'): 3,\n",
       "         ('to', 'upon'): 4,\n",
       "         ('to', 'prospective'): 2,\n",
       "         ('both', 'to'): 3,\n",
       "         ('both', 'prospective'): 1,\n",
       "         ...})"
      ]
     },
     "execution_count": 247,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find co-occurance in skip_grams with window of 2\n",
    "X_ik_skipgram = Counter(skip_grams_generated(window_size=2))\n",
    "\n",
    "X_ik_skipgram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Weight function\n",
    "\n",
    "#simply a normalized function...don't worry too much\n",
    "def weighting(w_i, w_j, X_ik):\n",
    "        \n",
    "    #check whether the co-occurrences exist between these two words\n",
    "    try:\n",
    "        x_ij = X_ik[(w_i, w_j)]\n",
    "    except:\n",
    "        x_ij = 1  #if does not exist, set it to 1, basically smoothing technique\n",
    "                \n",
    "    x_max = 100 #100 # fixed in paper  #cannot exceed 100 counts\n",
    "    alpha = 0.75 # Followed Chaky way!\n",
    "    \n",
    "    #if co-occurrence does not exceed 100, scale it based on some alpha\n",
    "    if x_ij < x_max:\n",
    "        result = (x_ij/x_max)**alpha  #scale it\n",
    "    else:\n",
    "        result = 1  #if is greater than max, set it to 1 maximum\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from itertools import combinations_with_replacement\n",
    "from tqdm import tqdm\n",
    "\n",
    "X_ik = {}  #for keeping the co-occurences\n",
    "weighting_dic = {} #scaling the percentage of sampling\n",
    "# Use tqdm as amanda recommend!\n",
    "for bigram in tqdm(combinations_with_replacement(vocab, 2)):\n",
    "    if X_ik_skipgram.get(bigram) is not None:  #matches \n",
    "        co_occer = X_ik_skipgram[bigram]  #get the count from what we already counted\n",
    "        X_ik[bigram] = co_occer + 1 # + 1 for stability issue\n",
    "        X_ik[(bigram[1],bigram[0])] = co_occer+1   #count also for the opposite\n",
    "    else:\n",
    "        pass\n",
    "        \n",
    "    weighting_dic[bigram] = weighting(bigram[0], bigram[1], X_ik)\n",
    "    weighting_dic[(bigram[1], bigram[0])] = weighting(bigram[1], bigram[0], X_ik)\n",
    "\n",
    "# Do not print if you have large data, otherwise your pc will froze.\n",
    "#print(f\"{X_ik=}\")\n",
    "#print(f\"{weighting_dic=}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['the',\n",
       "  'office',\n",
       "  'of',\n",
       "  'business',\n",
       "  'economics',\n",
       "  '(',\n",
       "  'obe',\n",
       "  ')',\n",
       "  'of',\n",
       "  'the',\n",
       "  'u.s.',\n",
       "  'department',\n",
       "  'of',\n",
       "  'commerce',\n",
       "  'provides',\n",
       "  'basic',\n",
       "  'measures',\n",
       "  'of',\n",
       "  'the',\n",
       "  'national',\n",
       "  'economy',\n",
       "  'and',\n",
       "  'current',\n",
       "  'analysis',\n",
       "  'of',\n",
       "  'short-run',\n",
       "  'changes',\n",
       "  'in',\n",
       "  'the',\n",
       "  'economic',\n",
       "  'situation',\n",
       "  'and',\n",
       "  'business',\n",
       "  'outlook',\n",
       "  '.'],\n",
       " ['it',\n",
       "  'develops',\n",
       "  'and',\n",
       "  'analyzes',\n",
       "  'the',\n",
       "  'national',\n",
       "  'income',\n",
       "  ',',\n",
       "  'balance',\n",
       "  'of',\n",
       "  'international',\n",
       "  'payments',\n",
       "  ',',\n",
       "  'and',\n",
       "  'many',\n",
       "  'other',\n",
       "  'business',\n",
       "  'indicators',\n",
       "  '.'],\n",
       " ['such',\n",
       "  'measures',\n",
       "  'are',\n",
       "  'essential',\n",
       "  'to',\n",
       "  'its',\n",
       "  'job',\n",
       "  'of',\n",
       "  'presenting',\n",
       "  'business',\n",
       "  'and',\n",
       "  'government',\n",
       "  'with',\n",
       "  'the',\n",
       "  'facts',\n",
       "  'required',\n",
       "  'to',\n",
       "  'meet',\n",
       "  'the',\n",
       "  'objective',\n",
       "  'of',\n",
       "  'expanding',\n",
       "  'business',\n",
       "  'and',\n",
       "  'improving',\n",
       "  'the',\n",
       "  'operation',\n",
       "  'of',\n",
       "  'the',\n",
       "  'economy',\n",
       "  '.'],\n",
       " ['contact'],\n",
       " ['for',\n",
       "  'further',\n",
       "  'information',\n",
       "  'contact',\n",
       "  'director',\n",
       "  ',',\n",
       "  'office',\n",
       "  'of',\n",
       "  'business',\n",
       "  'economics',\n",
       "  ',',\n",
       "  'u.s.',\n",
       "  'department',\n",
       "  'of',\n",
       "  'commerce',\n",
       "  ',',\n",
       "  'washington',\n",
       "  '25',\n",
       "  ',',\n",
       "  'd.c.',\n",
       "  '.']]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def random_batch(batch_size, word_sequence, skip_grams, X_ik, weighting_dic):\n",
    "    \n",
    "    #convert to id since our skip_grams is word, not yet id\n",
    "    skip_grams_id = [(word2index[skip_gram[0]], word2index[skip_gram[1]]) for skip_gram in skip_grams]\n",
    "    \n",
    "    random_inputs = []\n",
    "    random_labels = []\n",
    "    random_coocs  = []\n",
    "    random_weightings = []\n",
    "    random_index = np.random.choice(range(len(skip_grams_id)), batch_size, replace=False) #randomly pick without replacement\n",
    "        \n",
    "    for i in random_index:\n",
    "        random_inputs.append([skip_grams_id[i][0]])  # target, e.g., 2\n",
    "        random_labels.append([skip_grams_id[i][1]])  # context word, e.g., 3\n",
    "        \n",
    "        #get cooc\n",
    "        pair = skip_grams[i]\n",
    "        try:\n",
    "            cooc = X_ik[pair]\n",
    "        except:\n",
    "            cooc = 1\n",
    "        random_coocs.append([math.log(cooc)])\n",
    "        \n",
    "        #get weighting\n",
    "        weighting = weighting_dic[pair]\n",
    "        random_weightings.append([weighting])\n",
    "                    \n",
    "    return np.array(random_inputs), np.array(random_labels), np.array(random_coocs), np.array(random_weightings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  [[7220]\n",
      " [6415]]\n",
      "Target:  [[3379]\n",
      " [2377]]\n",
      "Cooc:  [[0.69314718]\n",
      " [1.60943791]]\n",
      "Weighting:  [[0.05318296]\n",
      " [0.10573713]]\n"
     ]
    }
   ],
   "source": [
    "#testing the method\n",
    "batch_size = 2 # mini-batch size\n",
    "skip_grams = skip_grams_generated(window_size=2)\n",
    "input_batch, target_batch, cooc_batch, weighting_batch = random_batch(batch_size, corpus, skip_grams, X_ik, weighting_dic)\n",
    "\n",
    "print(\"Input: \", input_batch)\n",
    "print(\"Target: \", target_batch)\n",
    "print(\"Cooc: \", cooc_batch)\n",
    "print(\"Weighting: \", weighting_batch)\n",
    "\n",
    "#we will convert them to tensor during training, so don't worry..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GloVe(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size,embed_size):\n",
    "        super(GloVe,self).__init__()\n",
    "        self.embedding_v = nn.Embedding(vocab_size, embed_size) # center embedding\n",
    "        self.embedding_u = nn.Embedding(vocab_size, embed_size) # out embedding\n",
    "        \n",
    "        self.v_bias = nn.Embedding(vocab_size, 1)\n",
    "        self.u_bias = nn.Embedding(vocab_size, 1)\n",
    "        \n",
    "    def forward(self, center_words, target_words, coocs, weighting):\n",
    "        center_embeds = self.embedding_v(center_words) # [batch_size, 1, emb_size]\n",
    "        target_embeds = self.embedding_u(target_words) # [batch_size, 1, emb_size]\n",
    "        \n",
    "        center_bias = self.v_bias(center_words).squeeze(1)\n",
    "        target_bias = self.u_bias(target_words).squeeze(1)\n",
    "        \n",
    "        inner_product = target_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, 1, emb_size] @ [batch_size, emb_size, 1] = [batch_size, 1, 1] = [batch_size, 1]\n",
    "        \n",
    "        #note that coocs already got log\n",
    "        loss = weighting*torch.pow(inner_product +center_bias + target_bias - coocs, 2)\n",
    "        \n",
    "        return torch.sum(loss)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Glove Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "# We are nvidia fanboys, so CUDA!!!\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Declare parameters\n",
    "voc_size = len(vocab)\n",
    "batch_size     = 10 # mini-batch size\n",
    "embedding_size = 100 #so we can later plot\n",
    "model          = GloVe(voc_size, embedding_size)\n",
    "model = model.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 | cost: 245.649582 | time: 0m 0s\n",
      "Epoch: 200 | cost: 332.947571 | time: 0m 0s\n",
      "Epoch: 300 | cost: 85.693680 | time: 0m 0s\n",
      "Epoch: 400 | cost: 132.173416 | time: 0m 0s\n",
      "Epoch: 500 | cost: 77.435997 | time: 0m 0s\n",
      "Total time: 3m 57s\")\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "start_training = time.time()\n",
    "# Training\n",
    "num_epochs = 500\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    input_batch, target_batch, cooc_batch, weighting_batch = random_batch(batch_size, corpus, skip_grams, X_ik, weighting_dic)\n",
    "    input_batch  = torch.LongTensor(input_batch).to(device)         #[batch_size, 1]\n",
    "    target_batch = torch.LongTensor(target_batch).to(device)        #[batch_size, 1]\n",
    "    cooc_batch   = torch.FloatTensor(cooc_batch).to(device)         #[batch_size, 1]\n",
    "    weighting_batch = torch.FloatTensor(weighting_batch).to(device) #[batch_size, 1]\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    loss = model(input_batch, target_batch, cooc_batch, weighting_batch)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "\n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        print(f\"Epoch: {epoch + 1} | cost: {loss:.6f} | time: {epoch_mins}m {epoch_secs}s\")\n",
    "\n",
    "end_training = time.time()\n",
    "start_min, end_min = epoch_time(start_training, end_training)\n",
    "print(f'Total time: {start_min}m {end_min}s\")')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/Glove_500.pth'\n",
    "torch.save(model.state_dict(), path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Skipgrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size, word_sequence, window_size=1):\n",
    "\n",
    "# I fix a little from Chaky so we can modify the window_size\n",
    "    \n",
    "    # Make skip gram of one size window\n",
    "    skip_grams = []\n",
    "    # loop each word sequence\n",
    "    # we starts from 1 because 0 has no context\n",
    "    # we stop at second last for the same reason\n",
    "    for sent in corpus:\n",
    "        for i in range(1, len(sent) - 1): # So we can modify the window size\n",
    "            target = word2index[sent[i]]\n",
    "            \n",
    "            context = list()\n",
    "            # ['a', 'b', 'c', 'd', 'e'] if window size = 2 and target is c\n",
    "            # this is basically append 'b', 'd', 'a', 'e' into context\n",
    "            \n",
    "            for j in range(window_size):\n",
    "                \n",
    "                if i - (j + 1) >= 0: # Check if it outside of range from the left of list\n",
    "                    context.append(word2index[sent[i - (j + 1)]])\n",
    "                \n",
    "                if i + (j + 1) < len(sent): # Check if it outside of range from the right of list\n",
    "                    context.append(word2index[sent[i + (j + 1)]])\n",
    "            \n",
    "            #context = [word2index[sent[i - 1]], word2index[sent[i + 1]]]\n",
    "            for w in context:\n",
    "                skip_grams.append([target, w])\n",
    "    \n",
    "    random_inputs = []\n",
    "    random_labels = []\n",
    "    random_index = np.random.choice(range(len(skip_grams)), batch_size, replace=False) #randomly pick without replacement\n",
    "        \n",
    "    for i in random_index:\n",
    "        random_inputs.append([skip_grams[i][0]])  # target, e.g., 2\n",
    "        random_labels.append([skip_grams[i][1]])  # context word, e.g., 3\n",
    "            \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  [[4243]\n",
      " [6386]]\n",
      "Target:  [[4143]\n",
      " [6137]]\n"
     ]
    }
   ],
   "source": [
    "#testing the method\n",
    "batch_size = 2 # mini-batch size\n",
    "input_batch, target_batch = random_batch(batch_size, corpus, 2)\n",
    "\n",
    "print(\"Input: \", input_batch)\n",
    "print(\"Target: \", target_batch)\n",
    "#we will convert them to tensor during training, so don't worry..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Skipgram(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, emb_size):\n",
    "        super(Skipgram,self).__init__()\n",
    "        self.embedding_v = nn.Embedding(vocab_size, emb_size)\n",
    "        self.embedding_u = nn.Embedding(vocab_size, emb_size)\n",
    "    \n",
    "    def forward(self, center_words, target_words, all_vocabs):\n",
    "        center_embeds = self.embedding_v(center_words) # [batch_size, 1, emb_size]\n",
    "        target_embeds = self.embedding_u(target_words) # [batch_size, 1, emb_size]\n",
    "        all_embeds    = self.embedding_u(all_vocabs) #   [batch_size, voc_size, emb_size]\n",
    "        \n",
    "        scores      = target_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, 1, emb_size] @ [batch_size, emb_size, 1] = [batch_size, 1, 1] = [batch_size, 1]\n",
    "\n",
    "        norm_scores = all_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, voc_size, emb_size] @ [batch_size, emb_size, 1] = [batch_size, voc_size, 1] = [batch_size, voc_size]\n",
    "\n",
    "        nll = -torch.mean(torch.log(torch.exp(scores)/torch.sum(torch.exp(norm_scores), 1).unsqueeze(1))) # log-softmax\n",
    "        # scalar (loss must be scalar)    \n",
    "            \n",
    "        return nll # negative log likelihood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size     = 10 # mini-batch size\n",
    "embedding_size = 100 #so we can later plot\n",
    "model          = Skipgram(voc_size, embedding_size)\n",
    "model = model.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 7362])\n"
     ]
    }
   ],
   "source": [
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)\n",
    "\n",
    "#use for the normalized term in the probability calculation\n",
    "all_vocabs = prepare_sequence(list(vocab), word2index).expand(batch_size, len(vocab))  # [batch_size, voc_size]\n",
    "print(all_vocabs.shape)\n",
    "all_vocabs = all_vocabs.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 | cost: 28.747135 | time: 0m 52s\n",
      "Epoch: 200 | cost: 36.921993 | time: 0m 43s\n",
      "Epoch: 300 | cost: 39.399998 | time: 0m 44s\n",
      "Epoch: 400 | cost: 37.276257 | time: 0m 43s\n",
      "Epoch: 500 | cost: 33.657913 | time: 0m 45s\n",
      "Total time use in skipgram with window size of 2 3 miniute(s) 49 second\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Training\n",
    "start_train_time = time.time()\n",
    "num_epochs = 500 # At first I intend to use 5,000 but it's too much for my PC\n",
    "start = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    input_batch, target_batch = random_batch(batch_size, corpus, window_size=2)\n",
    "    input_batch  = torch.LongTensor(input_batch).to(device)  #[batch_size, 1]\n",
    "    target_batch = torch.LongTensor(target_batch).to(device) #[batch_size, 1]\n",
    "\n",
    "    optimizer.zero_grad()\n",
    "    loss = model(input_batch, target_batch, all_vocabs)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        end = time.time()\n",
    "        epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "        print(f\"Epoch: {epoch + 1} | cost: {loss:.6f} | time: {epoch_mins}m {epoch_secs}s\")\n",
    "        start = time.time()\n",
    "end_train_time = time.time()\n",
    "train_time_mins, train_time_secs = epoch_time(start_train_time, end_train_time)\n",
    "print(f'Total time use in skipgram with window size of 2 {train_time_mins} miniute(s) {train_time_secs} second')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/Skipgrams_500.pth'\n",
    "torch.save(model.state_dict(), path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CBOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random batch for cbow\n",
    "\n",
    "def random_batch_cbow(batch_size, word_sequence, window_size=1):\n",
    "\n",
    "    cbow = []\n",
    "\n",
    "    for sent in corpus:\n",
    "        for i in range(1, len(sent) - 1): # So we can modify the window size\n",
    "            target = word2index[sent[i]]\n",
    "            context = list()\n",
    "            \n",
    "            for j in range(window_size):\n",
    "                \n",
    "                if i - (j + 1) >= 0: # Check if it outside of range from the left of list\n",
    "                    context.append(word2index[sent[i - (j + 1)]])\n",
    "                \n",
    "                if i + (j + 1) < len(sent): # Check if it outside of range from the right of list\n",
    "                    context.append(word2index[sent[i + (j + 1)]])\n",
    "            \n",
    "            # This part is different from skipgram\n",
    "            # Now we use all context as input and target as label\n",
    "            for w in context:\n",
    "                cbow.append([context, target])\n",
    "    \n",
    "    random_inputs = []\n",
    "    random_labels = []\n",
    "    random_index = np.random.choice(range(len(cbow)), batch_size, replace=False) #randomly pick without replacement\n",
    "    \n",
    "    for i in random_index:\n",
    "        random_inputs.append(cbow[i][0])  # Context word that we want as input\n",
    "        random_labels.append([cbow[i][1]])  # Target word that we want as label\n",
    "    \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  [[3228 5876 2003 2377]\n",
      " [6398 6415 7069 5321]]\n",
      "Target:  [[4700]\n",
      " [7106]]\n"
     ]
    }
   ],
   "source": [
    "#testing the method\n",
    "batch_size = 2 # mini-batch size\n",
    "input_batch, target_batch = random_batch_cbow(batch_size, corpus, 2)\n",
    "\n",
    "print(\"Input: \", input_batch)\n",
    "print(\"Target: \", target_batch)\n",
    "#we will convert them to tensor during training, so don't worry..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Cbow(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, emb_size):\n",
    "        super(Cbow,self).__init__() # Not sure why we super(Cbow) or super(Skipgram)?\n",
    "        self.embedding_v = nn.Embedding(vocab_size, emb_size)\n",
    "        self.embedding_u = nn.Embedding(vocab_size, emb_size)\n",
    "    \n",
    "    def forward(self, center_words, target_words, all_vocabs):\n",
    "        center_embeds = self.embedding_v(center_words) # [batch_size, 1, emb_size]\n",
    "        target_embeds = self.embedding_u(target_words) # [batch_size, 1, emb_size]\n",
    "        all_embeds    = self.embedding_u(all_vocabs) #   [batch_size, voc_size, emb_size]\n",
    "        \n",
    "        scores      = target_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, 1, emb_size] @ [batch_size, emb_size, 1] = [batch_size, 1, 1] = [batch_size, 1]\n",
    "\n",
    "        norm_scores = all_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, voc_size, emb_size] @ [batch_size, emb_size, 1] = [batch_size, voc_size, 1] = [batch_size, voc_size]\n",
    "\n",
    "        nll = -torch.mean(torch.log(torch.exp(scores)/torch.sum(torch.exp(norm_scores), 1).unsqueeze(1))) # log-softmax\n",
    "        # scalar (loss must be scalar)    \n",
    "            \n",
    "        return nll # negative log likelihood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size     = 10 # mini-batch size\n",
    "embedding_size = 100 #so we can later plot\n",
    "model          = Cbow(voc_size, embedding_size)\n",
    "model = model.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 | cost: 38.277550 | time: 0m 35s\n",
      "Epoch: 200 | cost: 34.378628 | time: 0m 35s\n",
      "Epoch: 300 | cost: 30.358282 | time: 0m 39s\n",
      "Epoch: 400 | cost: 28.354191 | time: 0m 37s\n",
      "Epoch: 500 | cost: 30.310083 | time: 0m 41s\n"
     ]
    }
   ],
   "source": [
    "# Training\n",
    "import time\n",
    "num_epochs = 500\n",
    "start = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    input_batch, target_batch = random_batch_cbow(batch_size, corpus, 1)\n",
    "    input_batch  = torch.LongTensor(input_batch).to(device)  #[batch_size, 1]\n",
    "    target_batch = torch.LongTensor(target_batch).to(device) #[batch_size, 1]\n",
    "\n",
    "    optimizer.zero_grad()\n",
    "    loss = model(input_batch, target_batch, all_vocabs)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        end = time.time()\n",
    "        epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "        print(f\"Epoch: {epoch + 1} | cost: {loss:.6f} | time: {epoch_mins}m {epoch_secs}s\")\n",
    "        start = time.time()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/CBow_500.pth'\n",
    "torch.save(model.state_dict(), path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Skipgram with negative sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 70117)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_count = Counter(flatten(corpus))\n",
    "num_total_words = sum([c for w, c in word_count.items()])\n",
    "\n",
    "# Check if the counting work\n",
    "word_count['car'], num_total_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create unigram table\n",
    "Z = 0.001\n",
    "unigram_table = []\n",
    "\n",
    "for vo in vocab:\n",
    "    unigram_table.extend([vo] * int(((word_count[vo]/num_total_words)**0.75)/Z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'total': 3,\n",
       "         'records': 1,\n",
       "         'served': 1,\n",
       "         'example': 1,\n",
       "         'necessary': 4,\n",
       "         'considerable': 1,\n",
       "         'naval': 1,\n",
       "         'forth': 1,\n",
       "         'week': 2,\n",
       "         'shipments': 1,\n",
       "         'continuing': 1,\n",
       "         'proposed': 2,\n",
       "         'yarn': 1,\n",
       "         'final': 1,\n",
       "         'today': 2,\n",
       "         'approved': 1,\n",
       "         'federal': 4,\n",
       "         'approach': 1,\n",
       "         'come': 1,\n",
       "         'd.c.': 1,\n",
       "         'serious': 1,\n",
       "         'york': 3,\n",
       "         'sales': 3,\n",
       "         'officer': 2,\n",
       "         'persons': 1,\n",
       "         'keep': 1,\n",
       "         'often': 1,\n",
       "         'may': 11,\n",
       "         'feet': 1,\n",
       "         'january': 1,\n",
       "         'receive': 1,\n",
       "         'think': 1,\n",
       "         'movable': 1,\n",
       "         'values': 1,\n",
       "         'funds': 3,\n",
       "         'community': 2,\n",
       "         'export-import': 1,\n",
       "         'recognized': 1,\n",
       "         'concern': 1,\n",
       "         'john': 2,\n",
       "         'peace': 4,\n",
       "         'contributed': 1,\n",
       "         'small': 5,\n",
       "         'concerns': 2,\n",
       "         'compared': 1,\n",
       "         'about': 6,\n",
       "         'providence': 2,\n",
       "         'precision': 1,\n",
       "         'responsibility': 2,\n",
       "         'closely': 1,\n",
       "         'administration': 3,\n",
       "         \"''\": 8,\n",
       "         'show': 1,\n",
       "         'history': 1,\n",
       "         'aircraft': 1,\n",
       "         'within': 4,\n",
       "         'standard': 2,\n",
       "         'industry': 2,\n",
       "         'act': 6,\n",
       "         'by': 22,\n",
       "         '2': 6,\n",
       "         'interested': 2,\n",
       "         'lower': 1,\n",
       "         'surface': 1,\n",
       "         'believe': 1,\n",
       "         'view': 1,\n",
       "         'appointed': 1,\n",
       "         'system': 3,\n",
       "         'location': 1,\n",
       "         'promotion': 1,\n",
       "         'electronics': 1,\n",
       "         'personal': 3,\n",
       "         'did': 2,\n",
       "         'upon': 4,\n",
       "         'house': 2,\n",
       "         'payments': 2,\n",
       "         'trained': 1,\n",
       "         'reserve': 1,\n",
       "         ';': 22,\n",
       "         'management': 3,\n",
       "         'projects': 4,\n",
       "         'establishing': 1,\n",
       "         'citizens': 1,\n",
       "         'fund': 1,\n",
       "         'want': 1,\n",
       "         'added': 1,\n",
       "         'several': 3,\n",
       "         'j.': 1,\n",
       "         'it': 17,\n",
       "         'since': 3,\n",
       "         'station': 1,\n",
       "         'when': 5,\n",
       "         'boats': 2,\n",
       "         'government': 8,\n",
       "         'wages': 1,\n",
       "         'subsection': 1,\n",
       "         'indicated': 1,\n",
       "         'type': 1,\n",
       "         'though': 1,\n",
       "         'officers': 1,\n",
       "         'students': 2,\n",
       "         'domestic': 2,\n",
       "         'fire': 1,\n",
       "         'systems': 2,\n",
       "         'following': 2,\n",
       "         'conditions': 2,\n",
       "         'security': 2,\n",
       "         'say': 1,\n",
       "         'problem': 3,\n",
       "         'thus': 1,\n",
       "         'rehabilitation': 1,\n",
       "         'sale': 1,\n",
       "         'our': 10,\n",
       "         'plan': 3,\n",
       "         'while': 2,\n",
       "         'rates': 1,\n",
       "         'steps': 1,\n",
       "         'those': 5,\n",
       "         'even': 3,\n",
       "         'recent': 2,\n",
       "         'owner': 1,\n",
       "         'ground': 1,\n",
       "         'where': 4,\n",
       "         'he': 10,\n",
       "         'aspects': 1,\n",
       "         'end': 2,\n",
       "         'caused': 1,\n",
       "         'activities': 3,\n",
       "         'growth': 2,\n",
       "         'faculty': 3,\n",
       "         'launched': 1,\n",
       "         'internal': 1,\n",
       "         'that': 24,\n",
       "         'rather': 1,\n",
       "         'resources': 2,\n",
       "         'fall': 1,\n",
       "         'young': 1,\n",
       "         'student': 3,\n",
       "         'record': 1,\n",
       "         'areas': 2,\n",
       "         'armed': 2,\n",
       "         'made': 8,\n",
       "         'indicate': 1,\n",
       "         'my': 2,\n",
       "         'relatively': 1,\n",
       "         'never': 1,\n",
       "         'until': 2,\n",
       "         'cover': 1,\n",
       "         'past': 1,\n",
       "         'museum': 1,\n",
       "         'financial': 2,\n",
       "         'calendar': 2,\n",
       "         'second': 1,\n",
       "         'various': 2,\n",
       "         'might': 1,\n",
       "         'organization': 1,\n",
       "         'again': 1,\n",
       "         'were': 8,\n",
       "         'found': 2,\n",
       "         'account': 1,\n",
       "         'sam': 2,\n",
       "         'sba': 2,\n",
       "         'order': 2,\n",
       "         'program': 6,\n",
       "         \"officer's\": 1,\n",
       "         'conference': 1,\n",
       "         'directly': 1,\n",
       "         'experience': 1,\n",
       "         'o.': 1,\n",
       "         'into': 5,\n",
       "         'independence': 1,\n",
       "         'automatic': 1,\n",
       "         'they': 8,\n",
       "         'alaska': 1,\n",
       "         'turn': 1,\n",
       "         'issue': 1,\n",
       "         'manufacturing': 1,\n",
       "         'up': 3,\n",
       "         'few': 2,\n",
       "         'some': 5,\n",
       "         'half': 1,\n",
       "         'grant': 1,\n",
       "         'soon': 1,\n",
       "         'ships': 1,\n",
       "         'campus': 1,\n",
       "         'get': 1,\n",
       "         'policy': 4,\n",
       "         'million': 4,\n",
       "         'automobiles': 1,\n",
       "         'loan': 2,\n",
       "         'has': 10,\n",
       "         '-': 1,\n",
       "         'needed': 2,\n",
       "         'regular': 1,\n",
       "         'structure': 1,\n",
       "         'city': 1,\n",
       "         'range': 1,\n",
       "         'assure': 1,\n",
       "         'september': 1,\n",
       "         'consideration': 1,\n",
       "         'practice': 1,\n",
       "         'cases': 1,\n",
       "         'institute': 2,\n",
       "         '4': 3,\n",
       "         'held': 2,\n",
       "         'participate': 1,\n",
       "         'like': 2,\n",
       "         'series': 1,\n",
       "         'out': 4,\n",
       "         'improved': 1,\n",
       "         'review': 1,\n",
       "         'overseas': 1,\n",
       "         'justice': 1,\n",
       "         'month': 2,\n",
       "         'increasing': 1,\n",
       "         'find': 1,\n",
       "         'similar': 1,\n",
       "         'term': 1,\n",
       "         'maintain': 1,\n",
       "         'hours': 2,\n",
       "         'class': 2,\n",
       "         'science': 1,\n",
       "         'living': 1,\n",
       "         'take': 3,\n",
       "         'commission': 3,\n",
       "         'effect': 1,\n",
       "         'form': 3,\n",
       "         'later': 2,\n",
       "         'is': 30,\n",
       "         '20': 1,\n",
       "         'prior': 2,\n",
       "         'foundation': 1,\n",
       "         'delaware': 1,\n",
       "         'have': 13,\n",
       "         'determination': 1,\n",
       "         'state': 12,\n",
       "         'v.': 1,\n",
       "         '(': 18,\n",
       "         'stock': 2,\n",
       "         'both': 5,\n",
       "         'religious': 1,\n",
       "         'agreed': 1,\n",
       "         'report': 4,\n",
       "         'additional': 2,\n",
       "         'free': 1,\n",
       "         'nor': 1,\n",
       "         'cooperative': 1,\n",
       "         'large': 3,\n",
       "         'revenue': 1,\n",
       "         'working': 1,\n",
       "         'october': 1,\n",
       "         'strategic': 1,\n",
       "         'effort': 2,\n",
       "         'located': 1,\n",
       "         'received': 2,\n",
       "         'five': 1,\n",
       "         'difficult': 1,\n",
       "         'power': 1,\n",
       "         'engineering': 1,\n",
       "         'national': 5,\n",
       "         'contract': 1,\n",
       "         'news': 1,\n",
       "         'providing': 1,\n",
       "         '50': 1,\n",
       "         'production': 4,\n",
       "         'against': 2,\n",
       "         'atomic': 1,\n",
       "         'been': 8,\n",
       "         'country': 3,\n",
       "         'notte': 1,\n",
       "         'gross': 1,\n",
       "         'because': 3,\n",
       "         'hundred': 2,\n",
       "         'money': 1,\n",
       "         'island': 5,\n",
       "         'progress': 2,\n",
       "         'fact': 1,\n",
       "         'periods': 1,\n",
       "         'vapor': 1,\n",
       "         'offers': 1,\n",
       "         'civilian': 1,\n",
       "         'product': 1,\n",
       "         'further': 2,\n",
       "         'services': 4,\n",
       "         'include': 2,\n",
       "         'sec.': 1,\n",
       "         'leadership': 1,\n",
       "         'education': 1,\n",
       "         'hearing': 2,\n",
       "         'common': 1,\n",
       "         'civil': 1,\n",
       "         'attention': 2,\n",
       "         'enter': 1,\n",
       "         'manufacturers': 1,\n",
       "         'claim': 3,\n",
       "         'business': 8,\n",
       "         'field': 2,\n",
       "         'special': 2,\n",
       "         'making': 2,\n",
       "         'than': 8,\n",
       "         'long': 2,\n",
       "         'bring': 1,\n",
       "         'go': 1,\n",
       "         'cranston': 1,\n",
       "         'does': 1,\n",
       "         'cost': 3,\n",
       "         'same': 4,\n",
       "         'committee': 1,\n",
       "         'followed': 1,\n",
       "         'continues': 1,\n",
       "         'accuracy': 1,\n",
       "         'members': 3,\n",
       "         'tax': 6,\n",
       "         'best': 2,\n",
       "         'way': 2,\n",
       "         'four': 1,\n",
       "         'put': 1,\n",
       "         'court': 2,\n",
       "         '?': 5,\n",
       "         'april': 2,\n",
       "         'town': 1,\n",
       "         'illustration': 1,\n",
       "         'active': 1,\n",
       "         'force': 2,\n",
       "         'trustees': 1,\n",
       "         'arts': 1,\n",
       "         'bank': 2,\n",
       "         'representatives': 2,\n",
       "         'abroad': 2,\n",
       "         'i': 7,\n",
       "         'textile': 1,\n",
       "         'however': 3,\n",
       "         'difference': 1,\n",
       "         'importance': 1,\n",
       "         'greater': 2,\n",
       "         'devoted': 1,\n",
       "         'almost': 1,\n",
       "         'individual': 2,\n",
       "         'aid': 3,\n",
       "         'transportation': 1,\n",
       "         'regulations': 1,\n",
       "         'evidence': 1,\n",
       "         'authorized': 2,\n",
       "         'page': 1,\n",
       "         'available': 4,\n",
       "         'manufacture': 1,\n",
       "         'destructive': 1,\n",
       "         'notes': 1,\n",
       "         'method': 2,\n",
       "         'advice': 1,\n",
       "         'bulletin': 1,\n",
       "         'cities': 2,\n",
       "         'greatest': 1,\n",
       "         'studies': 2,\n",
       "         'cannot': 1,\n",
       "         'provision': 1,\n",
       "         'last': 2,\n",
       "         'times': 2,\n",
       "         'energy': 1,\n",
       "         'we': 9,\n",
       "         'ownership': 1,\n",
       "         'war': 1,\n",
       "         'work': 4,\n",
       "         'provisions': 2,\n",
       "         '1958': 1,\n",
       "         \"brown's\": 1,\n",
       "         'whose': 1,\n",
       "         'increased': 2,\n",
       "         'uniform': 2,\n",
       "         'complete': 1,\n",
       "         'foreign': 3,\n",
       "         'executive': 1,\n",
       "         'disease': 1,\n",
       "         'shall': 7,\n",
       "         'retired': 1,\n",
       "         'although': 1,\n",
       "         'approval': 1,\n",
       "         'lack': 1,\n",
       "         'pathology': 3,\n",
       "         'rate': 2,\n",
       "         'develop': 1,\n",
       "         'promote': 1,\n",
       "         'health': 1,\n",
       "         'institutions': 1,\n",
       "         'life': 1,\n",
       "         'item': 2,\n",
       "         'policies': 2,\n",
       "         'university': 2,\n",
       "         'commuter': 1,\n",
       "         'continue': 1,\n",
       "         'expansion': 1,\n",
       "         'adequate': 1,\n",
       "         'du': 3,\n",
       "         'private': 1,\n",
       "         'jr.': 2,\n",
       "         'loans': 2,\n",
       "         'c': 2,\n",
       "         'any': 9,\n",
       "         'matching': 1,\n",
       "         'agencies': 2,\n",
       "         'kind': 1,\n",
       "         'granted': 1,\n",
       "         'taken': 1,\n",
       "         'came': 1,\n",
       "         'outside': 1,\n",
       "         'institution': 1,\n",
       "         'channels': 1,\n",
       "         '.': 81,\n",
       "         'educational': 1,\n",
       "         'expenses': 1,\n",
       "         'will': 14,\n",
       "         'estimated': 2,\n",
       "         'issues': 1,\n",
       "         'produced': 1,\n",
       "         'corporation': 1,\n",
       "         'very': 2,\n",
       "         'recently': 1,\n",
       "         'brown': 4,\n",
       "         'change': 1,\n",
       "         'short': 1,\n",
       "         '1040': 1,\n",
       "         'congress': 3,\n",
       "         'use': 6,\n",
       "         'staff': 2,\n",
       "         'central': 2,\n",
       "         'purchasing': 1,\n",
       "         'increases': 1,\n",
       "         'sense': 1,\n",
       "         '11': 1,\n",
       "         'rayburn': 2,\n",
       "         'involved': 1,\n",
       "         'operations': 2,\n",
       "         'limited': 1,\n",
       "         'next': 1,\n",
       "         'situation': 1,\n",
       "         'who': 6,\n",
       "         'area': 2,\n",
       "         'normal': 1,\n",
       "         'political': 1,\n",
       "         'carleton': 2,\n",
       "         'began': 1,\n",
       "         'mind': 1,\n",
       "         'toward': 2,\n",
       "         'filed': 1,\n",
       "         'you': 6,\n",
       "         'original': 1,\n",
       "         'hand': 1,\n",
       "         'automobile': 1,\n",
       "         'for': 36,\n",
       "         'proclamation': 1,\n",
       "         'pressure': 1,\n",
       "         'materials': 1,\n",
       "         'reduction': 1,\n",
       "         'fuel': 1,\n",
       "         'establish': 1,\n",
       "         'forces': 3,\n",
       "         'ago': 1,\n",
       "         'acres': 2,\n",
       "         'due': 2,\n",
       "         'scale': 1,\n",
       "         'not': 12,\n",
       "         'below': 2,\n",
       "         'year': 11,\n",
       "         'reports': 1,\n",
       "         'july': 2,\n",
       "         'only': 5,\n",
       "         '100': 1,\n",
       "         'building': 1,\n",
       "         'required': 3,\n",
       "         'such': 12,\n",
       "         'practices': 1,\n",
       "         'establishment': 1,\n",
       "         'at': 14,\n",
       "         'all': 10,\n",
       "         'built': 1,\n",
       "         'concerning': 1,\n",
       "         'question': 1,\n",
       "         'losses': 1,\n",
       "         'market': 3,\n",
       "         'workshop': 1,\n",
       "         'blockade': 1,\n",
       "         'considered': 1,\n",
       "         'unit': 1,\n",
       "         'able': 1,\n",
       "         'portion': 1,\n",
       "         'trade': 2,\n",
       "         'countries': 3,\n",
       "         'basic': 2,\n",
       "         'sum': 1,\n",
       "         'transition': 1,\n",
       "         'producing': 1,\n",
       "         'nothing': 1,\n",
       "         'percent': 2,\n",
       "         'electrical': 1,\n",
       "         'above': 3,\n",
       "         'long-term': 1,\n",
       "         'ballet': 1,\n",
       "         'increase': 3,\n",
       "         'wildlife': 1,\n",
       "         'allowed': 1,\n",
       "         'own': 3,\n",
       "         'company': 2,\n",
       "         'look': 1,\n",
       "         'travel': 1,\n",
       "         'entire': 1,\n",
       "         'means': 2,\n",
       "         'much': 2,\n",
       "         'wide': 1,\n",
       "         'rupees': 1,\n",
       "         'treasury': 2,\n",
       "         'amount': 3,\n",
       "         'course': 1,\n",
       "         'agreement': 3,\n",
       "         'title': 2,\n",
       "         'on': 21,\n",
       "         'regional': 1,\n",
       "         'questions': 1,\n",
       "         'liquid': 1,\n",
       "         'each': 6,\n",
       "         'ohio': 1,\n",
       "         'universal': 1,\n",
       "         'investigation': 1,\n",
       "         'throughout': 2,\n",
       "         'constructed': 1,\n",
       "         'largely': 1,\n",
       "         'exhibits': 1,\n",
       "         'together': 1,\n",
       "         'help': 2,\n",
       "         'reason': 1,\n",
       "         'this': 22,\n",
       "         ':': 7,\n",
       "         'supply': 1,\n",
       "         'without': 2,\n",
       "         'freedom': 2,\n",
       "         'towns': 2,\n",
       "         'returns': 1,\n",
       "         'unadjusted': 1,\n",
       "         'now': 4,\n",
       "         'service': 6,\n",
       "         'standards': 2,\n",
       "         'rules': 1,\n",
       "         'islands': 1,\n",
       "         'serve': 2,\n",
       "         'once': 1,\n",
       "         'timber': 1,\n",
       "         'here': 2,\n",
       "         'temperature': 1,\n",
       "         'economic': 3,\n",
       "         'among': 2,\n",
       "         'present': 3,\n",
       "         'certain': 2,\n",
       "         'authority': 1,\n",
       "         'provided': 3,\n",
       "         'must': 7,\n",
       "         'puerto': 2,\n",
       "         'see': 3,\n",
       "         'capita': 1,\n",
       "         'public': 4,\n",
       "         'pursuant': 2,\n",
       "         'forests': 1,\n",
       "         'deduct': 1,\n",
       "         'future': 2,\n",
       "         'e': 1,\n",
       "         'denied': 1,\n",
       "         's.': 2,\n",
       "         'tangible': 1,\n",
       "         'more': 8,\n",
       "         'world': 4,\n",
       "         'training': 2,\n",
       "         'rico': 2,\n",
       "         'division': 3,\n",
       "         'whereof': 1,\n",
       "         'top': 1,\n",
       "         'between': 4,\n",
       "         'every': 2,\n",
       "         'light': 1,\n",
       "         'described': 1,\n",
       "         'agency': 2,\n",
       "         'men': 3,\n",
       "         'do': 4,\n",
       "         'source': 1,\n",
       "         'motors': 3,\n",
       "         'its': 9,\n",
       "         'time': 7,\n",
       "         'recreation': 1,\n",
       "         'operating': 2,\n",
       "         'strength': 1,\n",
       "         'long-range': 2,\n",
       "         'revenues': 1,\n",
       "         'mile': 1,\n",
       "         'back': 1,\n",
       "         '1959': 3,\n",
       "         'represent': 1,\n",
       "         'highway': 1,\n",
       "         'expected': 1,\n",
       "         'over': 4,\n",
       "         'scientific': 1,\n",
       "         'cents': 1,\n",
       "         'roads': 2,\n",
       "         'allowances': 2,\n",
       "         'approximately': 1,\n",
       "         'directed': 1,\n",
       "         'value': 2,\n",
       "         'strong': 1,\n",
       "         'assessment': 1,\n",
       "         'interference': 3,\n",
       "         'opportunity': 2,\n",
       "         'employees': 2,\n",
       "         'task': 1,\n",
       "         'pont': 2,\n",
       "         'know': 1,\n",
       "         'known': 1,\n",
       "         '15': 2,\n",
       "         'processes': 1,\n",
       "         'wall': 1,\n",
       "         'sponsored': 1,\n",
       "         'these': 10,\n",
       "         'includes': 1,\n",
       "         'major': 2,\n",
       "         'largest': 1,\n",
       "         'communities': 1,\n",
       "         'data': 2,\n",
       "         'trial': 1,\n",
       "         'us': 2,\n",
       "         'united': 10,\n",
       "         'many': 6,\n",
       "         'surplus': 1,\n",
       "         'procurement': 1,\n",
       "         'return': 3,\n",
       "         'after': 3,\n",
       "         'land': 2,\n",
       "         'filing': 1,\n",
       "         'give': 2,\n",
       "         \"foundation's\": 1,\n",
       "         '10': 2,\n",
       "         'vehicle': 1,\n",
       "         'material': 1,\n",
       "         'fully': 1,\n",
       "         'least': 2,\n",
       "         'copies': 1,\n",
       "         'share': 2,\n",
       "         'agricultural': 1,\n",
       "         'lord': 1,\n",
       "         'an': 13,\n",
       "         'add': 1,\n",
       "         'which': 15,\n",
       "         'units': 1,\n",
       "         'carrying': 1,\n",
       "         'budget': 2,\n",
       "         'laboratory': 1,\n",
       "         'prepared': 1,\n",
       "         'said': 2,\n",
       "         'months': 1,\n",
       "         'plans': 2,\n",
       "         'u.': 2,\n",
       "         'forest': 2,\n",
       "         'association': 2,\n",
       "         'allotments': 1,\n",
       "         'assignment': 1,\n",
       "         'had': 4,\n",
       "         'miles': 2,\n",
       "         'distributed': 1,\n",
       "         'three': 2,\n",
       "         'west': 1,\n",
       "         'night': 1,\n",
       "         'but': 8,\n",
       "         'appeal': 1,\n",
       "         'cuba': 1,\n",
       "         'blocks': 1,\n",
       "         'nations': 2,\n",
       "         'advisory': 1,\n",
       "         'operation': 3,\n",
       "         'him': 2,\n",
       "         'corps': 4,\n",
       "         'either': 1,\n",
       "         'concrete': 1,\n",
       "         'home': 2,\n",
       "         'sunday': 1,\n",
       "         'complex': 1,\n",
       "         'skilled': 1,\n",
       "         'taxes': 1,\n",
       "         'situated': 1,\n",
       "         \"state's\": 2,\n",
       "         'women': 1,\n",
       "         'responsible': 1,\n",
       "         'stated': 1,\n",
       "         'af': 2,\n",
       "         'machinery': 1,\n",
       "         'appropriate': 1,\n",
       "         'matters': 1,\n",
       "         'skywave': 3,\n",
       "         'consider': 1,\n",
       "         'christiana': 1,\n",
       "         'growing': 1,\n",
       "         'excluding': 1,\n",
       "         'military': 4,\n",
       "         'nation': 2,\n",
       "         'first': 5,\n",
       "         'person': 1,\n",
       "         'a.': 2,\n",
       "         'adjustments': 1,\n",
       "         'officials': 1,\n",
       "         'organizations': 2,\n",
       "         'right': 1,\n",
       "         'components': 1,\n",
       "         'different': 1,\n",
       "         'levels': 1,\n",
       "         'u.s.': 2,\n",
       "         'billion': 2,\n",
       "         'law': 2,\n",
       "         'otherwise': 2,\n",
       "         'itself': 1,\n",
       "         'calendars': 1,\n",
       "         'paid': 2,\n",
       "         '1962': 2,\n",
       "         'group': 1,\n",
       "         'cooperation': 1,\n",
       "         'pay': 3,\n",
       "         'payment': 3,\n",
       "         'principal': 2,\n",
       "         'direct': 1,\n",
       "         'concerned': 1,\n",
       "         'groups': 1,\n",
       "         'based': 2,\n",
       "         'addition': 2,\n",
       "         'be': 28,\n",
       "         'continued': 1,\n",
       "         'obtained': 2,\n",
       "         'results': 1,\n",
       "         ')': 18,\n",
       "         'tool': 1,\n",
       "         'income': 4,\n",
       "         'habitat': 1,\n",
       "         'costs': 2,\n",
       "         'initial': 1,\n",
       "         'partnership': 1,\n",
       "         'whether': 2,\n",
       "         '31': 2,\n",
       "         'circumstances': 1,\n",
       "         'neither': 1,\n",
       "         'day': 4,\n",
       "         '30': 2,\n",
       "         'specified': 1,\n",
       "         'purchases': 1,\n",
       "         '``': 8,\n",
       "         'the': 130,\n",
       "         'answer': 1,\n",
       "         'lands': 1,\n",
       "         'remarks': 1,\n",
       "         'assist': 1,\n",
       "         'applied': 1,\n",
       "         'helium': 1,\n",
       "         'india': 2,\n",
       "         'rights': 1,\n",
       "         'journal': 1,\n",
       "         'recommendation': 1,\n",
       "         'run': 1,\n",
       "         'nine': 1,\n",
       "         '1952': 1,\n",
       "         'stations': 4,\n",
       "         'should': 8,\n",
       "         'r.': 1,\n",
       "         'emergency': 1,\n",
       "         '7': 2,\n",
       "         'terms': 1,\n",
       "         'then': 1,\n",
       "         'carry': 1,\n",
       "         'paragraphs': 1,\n",
       "         'reduce': 1,\n",
       "         'minimum': 1,\n",
       "         'commodities': 1,\n",
       "         'november': 1,\n",
       "         'maximum': 1,\n",
       "         'refund': 1,\n",
       "         'including': 2,\n",
       "         'being': 3,\n",
       "         'manager': 1,\n",
       "         'relating': 1,\n",
       "         'set': 3,\n",
       "         'period': 4,\n",
       "         'well': 3,\n",
       "         'departments': 1,\n",
       "         'daytime': 1,\n",
       "         'understanding': 1,\n",
       "         '8': 1,\n",
       "         'testimony': 1,\n",
       "         '3': 4,\n",
       "         'speaker': 2,\n",
       "         'to': 65,\n",
       "         'another': 2,\n",
       "         'existing': 1,\n",
       "         'medical': 4,\n",
       "         'result': 2,\n",
       "         'adjusted': 1,\n",
       "         'significance': 1,\n",
       "         'date': 2,\n",
       "         'man': 1,\n",
       "         'goals': 1,\n",
       "         'entitled': 2,\n",
       "         'amounts': 1,\n",
       "         'completion': 1,\n",
       "         'affairs': 1,\n",
       "         'years': 7,\n",
       "         'family': 1,\n",
       "         'except': 1,\n",
       "         'application': 2,\n",
       "         'number': 3,\n",
       "         'b': 3,\n",
       "         'capacity': 1,\n",
       "         'state-owned': 1,\n",
       "         'radiation': 1,\n",
       "         'college': 3,\n",
       "         'entrance': 1,\n",
       "         'problems': 3,\n",
       "         'capital': 1,\n",
       "         'measures': 1,\n",
       "         'equipment': 3,\n",
       "         'improvement': 1,\n",
       "         'developments': 1,\n",
       "         'factors': 1,\n",
       "         'athletic': 1,\n",
       "         'joint': 1,\n",
       "         'formed': 1,\n",
       "         'project': 1,\n",
       "         'respect': 3,\n",
       "         'permitted': 1,\n",
       "         'developed': 3,\n",
       "         'substantial': 2,\n",
       "         'assistance': 4,\n",
       "         'plant': 3,\n",
       "         'washington': 2,\n",
       "         'case': 2,\n",
       "         'frequencies': 1,\n",
       "         'meeting': 1,\n",
       "         'connection': 1,\n",
       "         'club': 1,\n",
       "         'sets': 2,\n",
       "         'determine': 2,\n",
       "         'changes': 1,\n",
       "         'property': 5,\n",
       "         'nighttime': 1,\n",
       "         'days': 1,\n",
       "         'included': 1,\n",
       "         'needs': 1,\n",
       "         'spectra': 1,\n",
       "         'firms': 1,\n",
       "         '25': 2,\n",
       "         'what': 4,\n",
       "         'designed': 2,\n",
       "         'price': 1,\n",
       "         'conducted': 1,\n",
       "         'publications': 1,\n",
       "         'air': 2,\n",
       "         'could': 3,\n",
       "         'through': 5,\n",
       "         'success': 1,\n",
       "         'maintenance': 2,\n",
       "         'district': 2,\n",
       "         'section': 4,\n",
       "         'sound': 1,\n",
       "         'thousand': 1,\n",
       "         'self-help': 1,\n",
       "         'awards': 1,\n",
       "         'international': 2,\n",
       "         'published': 1,\n",
       "         'fallout': 1,\n",
       "         'particularly': 1,\n",
       "         'information': 3,\n",
       "         'trails': 1,\n",
       "         'phase': 1,\n",
       "         'demand': 1,\n",
       "         'interior': 1,\n",
       "         'credit': 2,\n",
       "         'planning': 3,\n",
       "         'space': 1,\n",
       "         'statement': 1,\n",
       "         'meet': 1,\n",
       "         'competitive': 1,\n",
       "         '5': 2,\n",
       "         'basis': 3,\n",
       "         'rhode': 5,\n",
       "         'forward': 1,\n",
       "         'interests': 1,\n",
       "         'banks': 1,\n",
       "         'cars': 2,\n",
       "         'provide': 4,\n",
       "         'hawaii': 1,\n",
       "         'thereof': 1,\n",
       "         'position': 2,\n",
       "         'hope': 1,\n",
       "         'methods': 1,\n",
       "         'close': 2,\n",
       "         'me': 1,\n",
       "         'american': 4,\n",
       "         'assessors': 2,\n",
       "         'decision': 1,\n",
       "         'techniques': 1,\n",
       "         'development': 8,\n",
       "         'markets': 1,\n",
       "         'average': 2,\n",
       "         'objectives': 1,\n",
       "         'animals': 1,\n",
       "         'bureau': 1,\n",
       "         'was': 13,\n",
       "         'still': 2,\n",
       "         'honor': 1,\n",
       "         'relationship': 1,\n",
       "         '--': 9,\n",
       "         'less': 3,\n",
       "         'there': 7,\n",
       "         '50%': 1,\n",
       "         'usually': 1,\n",
       "         'shelter': 4,\n",
       "         'office': 2,\n",
       "         'advantages': 1,\n",
       "         'during': 6,\n",
       "         'petitioner': 2,\n",
       "         'extension': 1,\n",
       "         'director': 2,\n",
       "         'milling': 1,\n",
       "         'motor': 1,\n",
       "         'lines': 1,\n",
       "         'deductions': 1,\n",
       "         'army': 1,\n",
       "         'substantially': 1,\n",
       "         'need': 3,\n",
       "         'expenditures': 2,\n",
       "         'finance': 1,\n",
       "         'parts': 1,\n",
       "         'placed': 1,\n",
       "         'important': 4,\n",
       "         'request': 1,\n",
       "         'annual': 2,\n",
       "         'evaluation': 1,\n",
       "         'social': 1,\n",
       "         'contribute': 1,\n",
       "         'if': 8,\n",
       "         'percentage': 1,\n",
       "         'quality': 1,\n",
       "         'cooperatives': 1,\n",
       "         'degree': 1,\n",
       "         'professional': 1,\n",
       "         'farm': 1,\n",
       "         'point': 2,\n",
       "         '104': 1,\n",
       "         \"pont's\": 1,\n",
       "         'used': 3,\n",
       "         'opinion': 1,\n",
       "         'shares': 1,\n",
       "         'june': 3,\n",
       "         'economy': 1,\n",
       "         'your': 4,\n",
       "         'vehicles': 3,\n",
       "         'programs': 3,\n",
       "         'electronic': 2,\n",
       "         'provides': 1,\n",
       "         'interest': 4,\n",
       "         'prevent': 1,\n",
       "         'base': 1,\n",
       "         'types': 2,\n",
       "         'others': 2,\n",
       "         'uses': 1,\n",
       "         'population': 1,\n",
       "         'along': 1,\n",
       "         'department': 6,\n",
       "         'shown': 2,\n",
       "         'reasons': 1,\n",
       "         'people': 2,\n",
       "         'vocational': 1,\n",
       "         'no': 5,\n",
       "         'd': 1,\n",
       "         'legislative': 1,\n",
       "         'alone': 1,\n",
       "         'fields': 1,\n",
       "         'so': 6,\n",
       "         'knowledge': 1,\n",
       "         'condition': 1,\n",
       "         'teaching': 1,\n",
       "         'stockholders': 1,\n",
       "         'specific': 1,\n",
       "         'items': 1,\n",
       "         'therefore': 2,\n",
       "         'encourage': 1,\n",
       "         'too': 1,\n",
       "         'products': 2,\n",
       "         'editor': 1,\n",
       "         'defense': 3,\n",
       "         'are': 21,\n",
       "         'president': 4,\n",
       "         'with': 19,\n",
       "         'fiscal': 7,\n",
       "         'am': 2,\n",
       "         'action': 3,\n",
       "         'suitable': 1,\n",
       "         'exchange': 1,\n",
       "         'decisions': 1,\n",
       "         'water': 2,\n",
       "         'control': 3,\n",
       "         '&': 4,\n",
       "         'called': 1,\n",
       "         ...})"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(unigram_table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# Does the same thing as above.\n",
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)\n",
    "\n",
    "# Pick values from the table that we create before.\n",
    "def negative_sampling(targets, unigram_table, k):\n",
    "    batch_size = targets.size(0)\n",
    "    neg_samples = []\n",
    "    for i in range(batch_size):\n",
    "        nsample = []\n",
    "        target_index = targets[i].item()\n",
    "        while len(nsample) < k: # num of sampling\n",
    "            neg = random.choice(unigram_table)\n",
    "            if word2index[neg] == target_index:\n",
    "                continue\n",
    "            nsample.append(neg)\n",
    "        neg_samples.append(prepare_sequence(nsample, word2index).view(1, -1))\n",
    "    \n",
    "    return torch.cat(neg_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 384, 3991, 3490],\n",
       "        [5876, 4921, 4700],\n",
       "        [6244, 1422, 7069],\n",
       "        [5271, 4046, 3943],\n",
       "        [3877, 6788, 1518],\n",
       "        [4700, 4281, 2161],\n",
       "        [2821, 1849, 7335],\n",
       "        [5811, 4700, 6398],\n",
       "        [2104,  330,  699],\n",
       "        [ 177, 5317, 1399]])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing\n",
    "num_neg = 3\n",
    "negative_sampling(target_batch, unigram_table, num_neg)\n",
    "\n",
    "#{'grapes': 0, 'apple': 1, 'animal': 2, 'cat': 3, 'ice': 4, 'orange': 5, 'dog': 6, 'monkey': 7, 'conda': 8, 'fruit': 9, 'banana': 10}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SkipgramNegSampling(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, emb_size):\n",
    "        super(SkipgramNegSampling, self).__init__()\n",
    "        self.embedding_v = nn.Embedding(vocab_size, emb_size) # center embedding\n",
    "        self.embedding_u = nn.Embedding(vocab_size, emb_size) # out embedding\n",
    "        self.logsigmoid = nn.LogSigmoid()\n",
    "                    \n",
    "    def forward(self, center_words, target_words, negative_words):\n",
    "        center_embeds = self.embedding_v(center_words) # [batch_size, 1, emb_size]\n",
    "        target_embeds = self.embedding_u(target_words) # [batch_size, 1, emb_size]\n",
    "        neg_embeds    = -self.embedding_u(negative_words) # [batch_size, num_neg, emb_size]\n",
    "        \n",
    "        positive_score = target_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, 1, emb_size] @ [batch_size, emb_size, 1] = [batch_size, 1, 1] = [batch_size, 1]\n",
    "        \n",
    "        negative_score = torch.sum(neg_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2), 1).view(neg_embeds.size(0), -1) # BxK -> Bx1\n",
    "        #[batch_size, k, emb_size] @ [batch_size, emb_size, 1] = [batch_size, k, 1] = [batch_size, k] ==sum==> [batch_size, 1]\n",
    "        \n",
    "        # This is what had been changed from the normal one.\n",
    "        loss = self.logsigmoid(positive_score) + self.logsigmoid(negative_score)\n",
    "        \n",
    "        return -torch.mean(loss)\n",
    "    \n",
    "    def prediction(self, inputs):\n",
    "        embeds = self.embedding_v(inputs)\n",
    "        \n",
    "        return embeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize parameter\n",
    "batch_size     = 10 # mini-batch size\n",
    "embedding_size = 100 #so we can later plot\n",
    "model          = SkipgramNegSampling(voc_size, embedding_size)\n",
    "num_neg        = 10 # num of negative sampling\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 | cost: 9.755800 | time: 0m 0s\n",
      "Epoch: 200 | cost: 16.796337 | time: 0m 0s\n",
      "Epoch: 300 | cost: 14.862808 | time: 0m 0s\n",
      "Epoch: 400 | cost: 24.771328 | time: 0m 0s\n",
      "Epoch: 500 | cost: 16.948902 | time: 0m 0s\n",
      "Total time use in negative sampling 4 miniute(s) 3 second\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_train_time = time.time()\n",
    "\n",
    "# Training\n",
    "num_epochs = 500\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    input_batch, target_batch = random_batch(batch_size, corpus, 2)\n",
    "\n",
    "    # Neat trick to avoid nd.array object (This is bad practice!)\n",
    "    input_batch = list(input_batch)\n",
    "\n",
    "    # Padding since we do not cut the sentence so, It will not be in the same shape sometimes.\n",
    "    lenght_batch0 = len(input_batch[0])\n",
    "    lenght_batch1 = len(input_batch[1])\n",
    "    pad_num = np.abs(lenght_batch0 - lenght_batch1)\n",
    "\n",
    "    # pad the zero dimension\n",
    "    if lenght_batch0 < lenght_batch1:\n",
    "        input_batch[0].extend(list(np.full((pad_num, ), 0))) # Padding with zero\n",
    "    # pad the first dimension\n",
    "    elif lenght_batch0 > lenght_batch1:\n",
    "        input_batch[1].extend(list(np.full((pad_num, ), 0)))\n",
    "\n",
    "    \n",
    "    #input_batch: [batch_size, 1]\n",
    "    input_batch = torch.LongTensor(input_batch)\n",
    "    \n",
    "    #target_batch: [batch_size, 1]\n",
    "    target_batch = torch.LongTensor(target_batch)\n",
    "    \n",
    "    #negs_batch:   [batch_size, num_neg]\n",
    "    negs_batch = negative_sampling(target_batch, unigram_table, num_neg)\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "        \n",
    "    loss = model(input_batch, target_batch, negs_batch)\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        print(f\"Epoch: {epoch + 1} | cost: {loss:.6f} | time: {epoch_mins}m {epoch_secs}s\")\n",
    "\n",
    "end_train_time = time.time()\n",
    "neg_train_time_mins, neg_train_time_secs = epoch_time(start_train_time, end_train_time)\n",
    "print(f'Total time use in negative sampling {neg_train_time_mins} miniute(s) {neg_train_time_secs} second')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/Neg_Skipgrams_500.pth'\n",
    "torch.save(model.state_dict(), path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[': capital-common-countries', 'Athens Greece Baghdad Iraq', 'Athens Greece Bangkok Thailand']\n"
     ]
    }
   ],
   "source": [
    "# Load the data\n",
    "def read_data(path):\n",
    "    file = open(path, 'r') # Dataset from amamda\n",
    "    contents = file.read()\n",
    "    contents = contents.split('\\n') # Seperate chunk of text into substring\n",
    "    file.close()\n",
    "    return contents\n",
    "\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/questions-words.txt'\n",
    "text = read_data(path)\n",
    "print(text[0:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, ': capital-common-countries'),\n",
       " (507, ': capital-world'),\n",
       " (5032, ': currency'),\n",
       " (5899, ': city-in-state'),\n",
       " (8367, ': family'),\n",
       " (8874, ': gram1-adjective-to-adverb'),\n",
       " (9867, ': gram2-opposite'),\n",
       " (10680, ': gram3-comparative'),\n",
       " (12013, ': gram4-superlative'),\n",
       " (13136, ': gram5-present-participle'),\n",
       " (14193, ': gram6-nationality-adjective'),\n",
       " (15793, ': gram7-past-tense'),\n",
       " (17354, ': gram8-plural'),\n",
       " (18687, ': gram9-plural-verbs')]"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the seperator name and index\n",
    "seperator = [(idx, sent) for idx, sent in enumerate(text) if sent[0] == ':']\n",
    "seperator "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acceptable unacceptable aware unaware\n",
      "woman women snake snakes\n"
     ]
    }
   ],
   "source": [
    "# Let's use opposite and plural\n",
    "opposite = text[9868:10680]\n",
    "plural = text[17355:18687]\n",
    "\n",
    "# Concatenate\n",
    "test_text = opposite + plural\n",
    "\n",
    "# Checking\n",
    "print(test_text[0])\n",
    "print(test_text[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['acceptable', 'unacceptable', 'aware', 'unaware'], ['acceptable', 'unacceptable', 'certain', 'uncertain'], ['acceptable', 'unacceptable', 'clear', 'unclear'], ['acceptable', 'unacceptable', 'comfortable', 'uncomfortable'], ['acceptable', 'unacceptable', 'competitive', 'uncompetitive']]\n"
     ]
    }
   ],
   "source": [
    "test_opposite = [sent.split(\" \") for sent in opposite]\n",
    "test_plural = [sent.split(\" \") for sent in plural]\n",
    "test_corpus = [sent.split(\" \") for sent in test_text]\n",
    "print(test_corpus[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['melon', 'pears', 'elephants', 'cat', 'dogs']"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Flatten and get Unique words\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "test_vocab = list(set(flatten(test_corpus)))\n",
    "test_vocab[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "62\n",
      "car\n"
     ]
    }
   ],
   "source": [
    "# Word2index and Index2word for test set\n",
    "# Word2index and Index2word\n",
    "\n",
    "# assign id to those vocabs\n",
    "test_word2index = dict()\n",
    "test_word2index.update({\"<UNK>\":  0})\n",
    "for idx, v in enumerate(test_vocab):\n",
    "        test_word2index.update({v:  idx + 1})\n",
    "\n",
    "#add <UNK>, which is a very normal token exists in the world\n",
    "test_vocab.append('<UNK>') #chaky, can it be ##UNK, or UNKKKKKK, or anything\n",
    "\n",
    "# Testing\n",
    "print(test_word2index['car'])\n",
    "\n",
    "# index2word\n",
    "test_index2word = {v:k for k, v in test_word2index.items()}\n",
    "\n",
    "print(test_index2word[test_word2index['car']])\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'''  \n",
    "batch_size     = 10 # mini-batch size  \n",
    "embedding_size = 100 #so we can later plot  \n",
    "model          = GloVe(voc_size, embedding_size)  \n",
    "model          = Skipgram(voc_size, embedding_size)  \n",
    "model          = Cbow(voc_size, embedding_size)  \n",
    "model          = SkipgramNegSampling(voc_size, embedding_size)  \n",
    "num_neg        = 10 # num of negative sampling  \n",
    "  \n",
    "criterion = nn.CrossEntropyLoss()  \n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)  \n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to get embedding\n",
    "def get_embed(word, current_model):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except :\n",
    "        index = word2index['<UNK>'] #unknown\n",
    "    word = torch.LongTensor([index])\n",
    "    \n",
    "    embed =  (current_model.embedding_v(word)+current_model.embedding_u(word))/2\n",
    "    return np.array(embed[0].detach().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will put it in a loop soon!\n",
    "models_weight_list = ['Glove_500', 'Glove_1000', 'Cbow_500', 'Skipgrams_500', 'Neg_Skipgrams_500']\n",
    "model_list = [GloVe(voc_size, embedding_size), \n",
    "              GloVe(voc_size, embedding_size), \n",
    "              Cbow(voc_size, embedding_size), \n",
    "              Skipgram(voc_size, embedding_size), \n",
    "              SkipgramNegSampling(voc_size, embedding_size)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.31000853e-01,  5.93215227e-02,  6.83412910e-01,  7.73887038e-01,\n",
       "        7.61745512e-01, -1.96667090e-01, -6.44544661e-01, -7.25385070e-01,\n",
       "       -1.08609486e+00,  3.33659559e-01, -1.16339207e-01, -3.97296727e-01,\n",
       "        8.78399163e-02, -5.61352253e-01, -1.10439849e+00,  1.13723803e+00,\n",
       "       -3.92948955e-01,  8.52313399e-01,  1.18349522e-01,  5.32606244e-02,\n",
       "        1.05900615e-01, -9.84223962e-01,  3.51580799e-01,  8.27544630e-01,\n",
       "       -8.03473473e-01, -2.28011340e-01, -4.28224951e-02, -9.74792480e-01,\n",
       "       -2.17144549e-01,  5.62470794e-01,  6.65428877e-01,  7.59494007e-01,\n",
       "        3.60655874e-01,  1.39651954e-01,  4.55759913e-01, -3.43697608e-01,\n",
       "        6.35424137e-01,  6.83626294e-01, -9.65903521e-01, -3.18383455e-01,\n",
       "        4.19828475e-01, -4.20420974e-01, -1.35199845e+00,  1.29635715e+00,\n",
       "       -1.89303786e-01, -1.25500464e+00, -1.54869437e-01, -1.63623929e-01,\n",
       "       -2.98520237e-01, -1.66727638e+00, -8.15379620e-01, -1.83288842e-01,\n",
       "       -1.97563738e-01, -2.65820146e-01, -2.85834461e-01,  1.86055586e-01,\n",
       "        1.23784624e-01,  6.21519983e-01,  6.43439367e-02,  1.52537853e-01,\n",
       "       -6.77276433e-01,  1.35898590e-03,  5.97366929e-01, -2.33517408e+00,\n",
       "        8.67358804e-01, -7.58132100e-01, -5.00101149e-01,  1.25842631e+00,\n",
       "       -3.54874015e-01,  6.49252385e-02, -1.09245682e+00, -6.67635441e-01,\n",
       "        1.84379101e-01, -1.03851628e+00, -4.10769045e-01,  5.81666082e-03,\n",
       "        4.68698293e-01,  6.18872464e-01, -9.14894998e-01, -6.01576328e-01,\n",
       "        6.68547153e-02,  1.20744729e+00, -3.85032892e-01, -9.56796527e-01,\n",
       "        6.52815163e-01,  7.62771487e-01,  9.91259441e-02,  1.13045371e+00,\n",
       "       -3.33464861e-01,  1.36139286e+00, -1.56553641e-01, -1.02501905e+00,\n",
       "       -9.03348327e-02,  1.40614200e+00, -1.26394719e-01,  1.81912184e-02,\n",
       "        2.87367016e-01, -3.14804345e-01,  3.50907803e-01, -7.44261146e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test embeded\n",
    "testing_word = 'Queen'\n",
    "current_model = model_list[0]\n",
    "\n",
    "weight_path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/Glove_500.pth'\n",
    "\n",
    "current_weight = models_weight_list[0]\n",
    "current_model.load_state_dict(torch.load(weight_path))\n",
    "current_model.eval()\n",
    "\n",
    "test_embed = get_embed(testing_word, current_model)\n",
    "test_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#numpy version\n",
    "from numpy import dot\n",
    "from numpy.linalg import norm\n",
    "\n",
    "def cos_sim(a, b):\n",
    "    cos_sim = dot(a, b)/(norm(a)*norm(b))\n",
    "    return cos_sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_analogy(a,b,c,vocabs=vocab):\n",
    "    emb_a, emb_b, emb_c = get_embed(a, current_model), get_embed(b, current_model), get_embed(c, current_model)\n",
    "    vector = emb_b - emb_a + emb_c\n",
    "    # vector_norm = (vector ** 2).sum() ** (1 / 2)\n",
    "    # vector = vector / vector_norm\n",
    "    # print(vector.shape)\n",
    "    similarity = -1 \n",
    "    \n",
    "    for vocab in vocabs:\n",
    "        if vocab not in [a,b,c]: #ignore input words itself\n",
    "            current_sim = cos_sim(vector,get_embed(vocab, current_model))\n",
    "            if current_sim > similarity:\n",
    "                similarity = current_sim #update better one\n",
    "                d = (vocab, similarity)\n",
    "    return d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('signals', 0.34757987)"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing find_analogy functions\n",
    "find_analogy('man', 'woman', 'adult')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'signals'"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_analogy('man', 'woman', 'adult')[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will put it in a loop soon!\n",
    "models_weight_list = ['Glove_500', 'Glove_1000', 'Cbow_500', 'Skipgrams_500', 'Neg_Skipgrams_500']\n",
    "models_name = ['Glove', 'Glove', 'Cbow', 'Skipgrams', 'Neg_Skipgrams']\n",
    "voc_size = len(vocab)\n",
    "embedding_size = 100\n",
    "model_list = [GloVe(voc_size, embedding_size), \n",
    "              GloVe(voc_size, embedding_size), \n",
    "              Cbow(voc_size, embedding_size), \n",
    "              Skipgram(voc_size, embedding_size), \n",
    "              SkipgramNegSampling(voc_size, embedding_size)]\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "test_list = [test_opposite, test_plural]\n",
    "test_list_name = ['test_opposite', 'test_plural']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_accruacy(y, yhat):\n",
    "    if y == yhat:\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "def test_accruacy_batch(data, current_model):\n",
    "    counter = 0\n",
    "    for sent in data:\n",
    "        label = sent[-1]\n",
    "        a, b, c = sent[:-1]\n",
    "        yhat = find_analogy(a, b, c)[0] # It's return in tuple form, so we need to slice to get word\n",
    "        if check_accruacy(label, yhat) == True:\n",
    "            counter = counter + 1\n",
    "    \n",
    "    return counter\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current model = Glove\n",
      "Current weight = Glove_500\n",
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n",
      "Current model = Glove\n",
      "Current weight = Glove_1000\n",
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n",
      "Current model = Cbow\n",
      "Current weight = Cbow_500\n",
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n",
      "Current model = Skipgrams\n",
      "Current weight = Skipgrams_500\n",
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n",
      "Current model = Neg_Skipgrams\n",
      "Current weight = Neg_Skipgrams_500\n",
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "# Have the model to predict the last word.\n",
    "# Actually, It is better to random the label but mine model is very weak. so I will just predict last word\n",
    "\n",
    "#models_weight_list = ['Glove_500', 'Glove_1000', 'Cbow_500', 'Skipgrams_500', 'Neg_Skipgrams_500']\n",
    "\n",
    "main_results = list()\n",
    "main_accruacy = list()\n",
    "main_results_name = list()\n",
    "results = list()\n",
    "accruacy = list()\n",
    "results_name = list()\n",
    "\n",
    "for models_idx in range(len(models_weight_list)):\n",
    "    weight_path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/' + models_weight_list[models_idx] + '.pth'\n",
    "    current_model = model_list[models_idx]\n",
    "    current_model.load_state_dict(torch.load(weight_path))\n",
    "    current_model.eval()\n",
    "    print(f'Current model = {models_name[models_idx]}')\n",
    "    print(f'Current weight = {models_weight_list[models_idx]}')\n",
    "    \n",
    "    for idx, current_test in enumerate(test_list):\n",
    "        sample_list = random.choices(current_test, k=100)\n",
    "        print(f'Current_test = {test_list_name[idx]}')\n",
    "        accruacy = test_accruacy_batch(sample_list, current_model)\n",
    "        print(accruacy)\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim.test.utils import datapath\n",
    "from gensim.models import KeyedVectors\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "\n",
    "#you have to put this file in some python/gensim directory; just run it and it will inform where to put....\n",
    "# Download from https://github.com/stanfordnlp/GloVe\n",
    "glove_file = datapath('/root/projects/NLP/Assignment/19_Jan_Glove/glove.6B.100d.txt')\n",
    "model = KeyedVectors.load_word2vec_format(glove_file, binary=False, no_header=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current_test = test_opposite\n",
      "0\n",
      "Current_test = test_plural\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "#result = model.most_similar(positive=['woman', 'king'], negative=['man'])\n",
    "\n",
    "for idx, current_test in enumerate(test_list):\n",
    "    sample_list = random.choices(current_test, k=100)\n",
    "    print(f'Current_test = {test_list_name[idx]}')\n",
    "\n",
    "    counter = 0\n",
    "    for sent in current_test:\n",
    "        label = sent[-1]\n",
    "        a, b, c = sent[:-1]\n",
    "        yhat = model.most_similar(positive=[a, b], negative=[c])\n",
    "        yhat = yhat[0][0]\n",
    "\n",
    "        if check_accruacy(label, yhat) == True:\n",
    "            counter = counter + 1\n",
    "\n",
    "    print(accruacy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['woman', 'women', 'snake', 'snakes'], 'men', 'snakes')"
      ]
     },
     "execution_count": 261,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the results\n",
    "sent, yhat, label"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tiger</td>\n",
       "      <td>cat</td>\n",
       "      <td>7.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tiger</td>\n",
       "      <td>tiger</td>\n",
       "      <td>10.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>plane</td>\n",
       "      <td>car</td>\n",
       "      <td>5.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>train</td>\n",
       "      <td>car</td>\n",
       "      <td>6.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>television</td>\n",
       "      <td>radio</td>\n",
       "      <td>6.77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0      1      2\n",
       "0       tiger    cat   7.35\n",
       "1       tiger  tiger  10.00\n",
       "2       plane    car   5.77\n",
       "3       train    car   6.31\n",
       "4  television  radio   6.77"
      ]
     },
     "execution_count": 278,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import datasets\n",
    "import pandas as pd\n",
    "path = '/root/projects/NLP/Assignment/19_Jan_Glove/wordsim353_sim_rel/wordsim_similarity_goldstandard.txt'\n",
    "df = pd.read_table(path, header=None)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tiger</td>\n",
       "      <td>cat</td>\n",
       "      <td>7.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tiger</td>\n",
       "      <td>tiger</td>\n",
       "      <td>10.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>plane</td>\n",
       "      <td>car</td>\n",
       "      <td>5.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>train</td>\n",
       "      <td>car</td>\n",
       "      <td>6.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>television</td>\n",
       "      <td>radio</td>\n",
       "      <td>6.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>media</td>\n",
       "      <td>radio</td>\n",
       "      <td>7.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>bread</td>\n",
       "      <td>butter</td>\n",
       "      <td>6.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>cucumber</td>\n",
       "      <td>potato</td>\n",
       "      <td>5.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>doctor</td>\n",
       "      <td>nurse</td>\n",
       "      <td>7.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>professor</td>\n",
       "      <td>doctor</td>\n",
       "      <td>6.62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0       1      2\n",
       "0       tiger     cat   7.35\n",
       "1       tiger   tiger  10.00\n",
       "2       plane     car   5.77\n",
       "3       train     car   6.31\n",
       "4  television   radio   6.77\n",
       "5       media   radio   7.42\n",
       "6       bread  butter   6.19\n",
       "7    cucumber  potato   5.92\n",
       "8      doctor   nurse   7.00\n",
       "9   professor  doctor   6.62"
      ]
     },
     "execution_count": 281,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "synthetic_test_set = df.iloc[:10]\n",
    "synthetic_test_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_x1 = synthetic_test_set[0]\n",
    "input_x2 = synthetic_test_set[1]\n",
    "label = synthetic_test_set[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to get embedding\n",
    "def get_embed(word, current_model):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except :\n",
    "        index = word2index['<UNK>'] #unknown\n",
    "    word = torch.LongTensor([index])\n",
    "    \n",
    "    embed =  (current_model.embedding_v(word)+current_model.embedding_u(word))/2\n",
    "    return np.array(embed[0].detach().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will put it in a loop soon!\n",
    "models_weight_list = ['Glove_500', 'Glove_1000', 'Cbow_500', 'Skipgrams_500', 'Neg_Skipgrams_500']\n",
    "models_name = ['Glove', 'Glove', 'Cbow', 'Skipgrams', 'Neg_Skipgrams']\n",
    "voc_size = len(vocab)\n",
    "embedding_size = 100\n",
    "model_list = [GloVe(voc_size, embedding_size), \n",
    "              GloVe(voc_size, embedding_size), \n",
    "              Cbow(voc_size, embedding_size), \n",
    "              Skipgram(voc_size, embedding_size), \n",
    "              SkipgramNegSampling(voc_size, embedding_size)]\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "test_list = [test_opposite, test_plural]\n",
    "test_list_name = ['test_opposite', 'test_plural']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.034287428742874285"
      ]
     },
     "execution_count": 291,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test\n",
    "from scipy import stats\n",
    "current_model = model_list[0]\n",
    "current_model.eval()\n",
    "\n",
    "cat = get_embed('cat', current_model)\n",
    "dog = get_embed('dog', current_model)\n",
    "\n",
    "stats.spearmanr(cat, dog)[0]\n",
    "\n",
    "#res = stats.spearmanr([1, 2, 3, 4, 5], [5, 6, 7, 8, 7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current model = Glove\n",
      "Current weight = Glove_500\n",
      "Current model = Glove\n",
      "Current weight = Glove_1000\n",
      "Current model = Cbow\n",
      "Current weight = Cbow_500\n",
      "Current model = Skipgrams\n",
      "Current weight = Skipgrams_500\n",
      "Current model = Neg_Skipgrams\n",
      "Current weight = Neg_Skipgrams_500\n"
     ]
    }
   ],
   "source": [
    "# Have the model to find word similarity.\n",
    "#models_weight_list = ['Glove_500', 'Glove_1000', 'Cbow_500', 'Skipgrams_500', 'Neg_Skipgrams_500']\n",
    "\n",
    "main_results = list()\n",
    "main_results_name = list()\n",
    "results = list()\n",
    "results_name = list()\n",
    "\n",
    "for models_idx in range(len(models_weight_list)):\n",
    "    weight_path = '/root/projects/NLP/Assignment/19_Jan_Glove/models/' + models_weight_list[models_idx] + '.pth'\n",
    "    current_model = model_list[models_idx]\n",
    "    current_model.load_state_dict(torch.load(weight_path))\n",
    "    current_model.eval()\n",
    "    print(f'Current model = {models_name[models_idx]}')\n",
    "    print(f'Current weight = {models_weight_list[models_idx]}')\n",
    "    results = list()\n",
    "    for idx in range(10): # We test with only fix 10 samples\n",
    "        emb_x1 = get_embed(input_x1[idx], current_model)\n",
    "        emb_x2 = get_embed(input_x2[idx], current_model)\n",
    "        yhat = stats.spearmanr(emb_x1, emb_x2)\n",
    "        yhat = yhat[0]\n",
    "        #print(yhat)\n",
    "        results.append(yhat)\n",
    "\n",
    "    main_results.append(results)\n",
    "    main_results_name.append(models_name[models_idx])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 316,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: Glove MSE: 25.57 Mean: 6.93\n",
      "Model: Glove MSE: 23.51 Mean: 6.93\n",
      "Model: Cbow MSE: 29.40 Mean: 6.93\n",
      "Model: Skipgrams MSE: 23.95 Mean: 6.93\n",
      "Model: Neg_Skipgrams MSE: 25.50 Mean: 6.93\n"
     ]
    }
   ],
   "source": [
    "# Find mean sqare error\n",
    "from sklearn.metrics import mean_squared_error as mse\n",
    "\n",
    "# Convert to be on the same scale\n",
    "for models_idx in range(len(models_weight_list)):\n",
    "    yhat = [i * 10 for i in main_results[models_idx]]\n",
    "    y = label\n",
    "    loss = mse(y, yhat)\n",
    "    meany = np.mean(y)\n",
    "    print(f'Model: {models_name[models_idx]} MSE: {loss:0.2f} Mean: {meany:0.2f}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test with gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 317,
   "metadata": {},
   "outputs": [],
   "source": [
    "#you have to put this file in some python/gensim directory; just run it and it will inform where to put....\n",
    "# Download from https://github.com/stanfordnlp/GloVe\n",
    "glove_file = datapath('/root/projects/NLP/Assignment/19_Jan_Glove/glove.6B.100d.txt')\n",
    "model = KeyedVectors.load_word2vec_format(glove_file, binary=False, no_header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get yhat from gensim\n",
    "yhat = list()\n",
    "for i in range(10):\n",
    "    emb_x1 = model[input_x1[i]]\n",
    "    emb_x2 = model[input_x2[i]]\n",
    "    result = stats.spearmanr(emb_x1, emb_x2)\n",
    "    result = result[0]\n",
    "    yhat.append(result)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: Glove with gensim MSE: 1.43 Mean: 6.93\n"
     ]
    }
   ],
   "source": [
    "# Find yhat mse\n",
    "yhat = [i * 10 for i in yhat]\n",
    "gensim_mse = mse(label, yhat)\n",
    "meany = np.mean(label)\n",
    "print(f'Model: Glove with gensim MSE: {gensim_mse:0.2f} Mean: {meany:0.2f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 325,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model[input_x1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}