2. Sentiment Analysis using Doc2Vec, N-Gram & Phrase Modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis \n",
    "\n",
    "# Introduction\n",
    "Analyze & classify sentiment of text data, articles into positive or negative\n",
    "\n",
    "# Objective\n",
    "Sentiment analysis notebooks dives in very depth of various concepts, methods related to text analysis and understand the meaning of it semantically and/or syntactly. They are classified in the following five based notebooks based on different methods & tools used to analyze & classify text.\n",
    "\n",
    "1. Sentiment Analysis with Text Blob, Word Cloud, Count Vectorizer, N-Gram\n",
    "2. Sentiment Analysis using Doc2Vec, N-Gram & Phrase Modelling\n",
    "3. Sentiment Analysis with Chi2 Square & PCA Dimension Reduction\n",
    "4. Sentiment Analysis with Keras & Tensorflow\n",
    "5. Sentiment Analysis with Keras & Tensorflow using Doc2Vec, Pretrained GloVe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Due\n",
    "## 2. Sentiment Analysis with Doc2Vec, N-Gram & Phrase Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from multiprocessing import Process\n",
    "\n",
    "# # Multiprocessing to spawn processes using an API similar to threading module\n",
    "#     proc = Process(target=model.run_pipeline, args=())\n",
    "\n",
    "#     proc.start()\n",
    "#     proc.join()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic import\n",
    "\n",
    "import re\n",
    "import pandas as pd  \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "tqdm.pandas(desc=\"progress-bar\")\n",
    "\n",
    "from gensim.models import Doc2Vec\n",
    "from gensim.models.doc2vec import LabeledSentence\n",
    "from gensim.models.phrases import Phrases, Phraser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1600000 entries, 0 to 1599999\n",
      "Data columns (total 3 columns):\n",
      "sentiment        1600000 non-null int64\n",
      "text             1600000 non-null object\n",
      "pre_clean_len    1600000 non-null int64\n",
      "dtypes: int64(2), object(1)\n",
      "memory usage: 48.8+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "      <th>pre_clean_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>awww that bummer you shoulda got david carr of...</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>is upset that he can not update his facebook b...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>dived many times for the ball managed to save ...</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>no it not behaving at all mad why am here beca...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                               text  pre_clean_len\n",
       "0          0  awww that bummer you shoulda got david carr of...            115\n",
       "1          0  is upset that he can not update his facebook b...            111\n",
       "2          0  dived many times for the ball managed to save ...             89\n",
       "3          0     my whole body feels itchy and like its on fire             47\n",
       "4          0  no it not behaving at all mad why am here beca...            111"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read TF dataframe\n",
    "\n",
    "df = pd.read_hdf('./data/redstone.hdf')\n",
    "df.info()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1600000 entries, 0 to 1599999\n",
      "Data columns (total 3 columns):\n",
      "sentiment        1600000 non-null int64\n",
      "text             1600000 non-null object\n",
      "pre_clean_len    1600000 non-null int64\n",
      "dtypes: int64(2), object(1)\n",
      "memory usage: 36.6+ MB\n"
     ]
    }
   ],
   "source": [
    "# Santitizing dataframe\n",
    "\n",
    "df.dropna(inplace=True)\n",
    "df.reset_index(drop=True,inplace=True)\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn import utils\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "train = df.text\n",
    "label = df.sentiment\n",
    "SEED = 21\n",
    "\n",
    "# Splitting data into train, test & validation sets\n",
    "x_train, x_val_test, y_train, y_val_test = train_test_split(train, label, test_size=.02, random_state=SEED)\n",
    "\n",
    "x_val, x_test, y_val, y_test = train_test_split(x_val_test, y_val_test, test_size=.5, random_state=SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2vec consists of two techniques – CBOW(Continuous bag of words) and Skip-gram model. Both of these techniques learn weights which act as word vector representations. \n",
    "With a corpus, CBOW model predicts the current word from a window of surrounding context words, while Skip-gram model predicts surrounding context words given the current word.\n",
    "\n",
    "eg. \"I love dogs\". \n",
    "CBOW model tries to predict the word \"love\" when given \"I\", \"dogs\" as inputs.\n",
    "Skip-gram model tries to predict \"I\", \"dogs\" when given the word \"love\" as input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](images/word2vec.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word vectors are actually the weights of the trained models, not the predicted results. After extracting the weights, such a vector comes to represent in some abstract way the ‘meaning’ of a word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2vec uses the same logic as word2vec, but applies it on the document level. According to Mikolov et al. (2014), \"every paragraph is mapped to a unique vector, represented by a column in matrix D and every word is also mapped to a unique vector, represented by a column in matrix W. \n",
    "\n",
    "The paragraph vector and word vectors are averaged or concatenated to predict the next word in a context. \n",
    "\n",
    "The paragraph token can be thought of as another word. It acts as a memory that remembers what is missing from the current context – or the topic of the paragraph.\n",
    "\n",
    "https://cs.stanford.edu/~quocle/paragraph_vector.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](images/doc2vec.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labelling tweets using genism library for unsupervised learning\n",
    "\n",
    "def label_tweets_unigram(tweets, label):\n",
    "    result = []\n",
    "    prefix = label\n",
    "    \n",
    "    # Split tweets & attach label with index\n",
    "    for index, tweet in zip(tweets.index, tweets):\n",
    "        result.append(LabeledSentence(tweet.split(), [prefix + '_%s' % index]))\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All cores of CPU\n",
    "\n",
    "cores = multiprocessing.cpu_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training Doc2Vec, I have used the whole data set. The rationale behind is that the doc2vec training is completely unsupervised and hence there is no need to hold out any (unlabelled) data.\n",
    "\n",
    "\"An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation\"\n",
    "https://arxiv.org/pdf/1607.05368.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:9: DeprecationWarning: Call to deprecated `LabeledSentence` (Class will be removed in 4.0.0, use TaggedDocument instead).\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1600000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vec_train = label_tweets_unigram(df.text , 'all')\n",
    "len(word_vec_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[LabeledSentence(words=['awww', 'that', 'bummer', 'you', 'shoulda', 'got', 'david', 'carr', 'of', 'third', 'day', 'to', 'do', 'it'], tags=['all_0']),\n",
       " LabeledSentence(words=['is', 'upset', 'that', 'he', 'can', 'not', 'update', 'his', 'facebook', 'by', 'texting', 'it', 'and', 'might', 'cry', 'as', 'result', 'school', 'today', 'also', 'blah'], tags=['all_1']),\n",
       " LabeledSentence(words=['dived', 'many', 'times', 'for', 'the', 'ball', 'managed', 'to', 'save', 'the', 'rest', 'go', 'out', 'of', 'bounds'], tags=['all_2']),\n",
       " LabeledSentence(words=['my', 'whole', 'body', 'feels', 'itchy', 'and', 'like', 'its', 'on', 'fire'], tags=['all_3']),\n",
       " LabeledSentence(words=['no', 'it', 'not', 'behaving', 'at', 'all', 'mad', 'why', 'am', 'here', 'because', 'can', 'not', 'see', 'you', 'all', 'over', 'there'], tags=['all_4']),\n",
       " LabeledSentence(words=['not', 'the', 'whole', 'crew'], tags=['all_5']),\n",
       " LabeledSentence(words=['need', 'hug'], tags=['all_6']),\n",
       " LabeledSentence(words=['hey', 'long', 'time', 'no', 'see', 'yes', 'rains', 'bit', 'only', 'bit', 'lol', 'fine', 'thanks', 'how', 'you'], tags=['all_7']),\n",
       " LabeledSentence(words=['nope', 'they', 'did', 'not', 'have', 'it'], tags=['all_8']),\n",
       " LabeledSentence(words=['que', 'me', 'muera'], tags=['all_9'])]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vec_train[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DBOW (Document Bag Of Words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW: This is the Doc2Vec model analogous to Skip-gram model in Word2Vec. The paragraph vectors are obtained by training a neural network on the task of predicting a probability distribution of words in a paragraph given a randomly-sampled word from the paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3914399.94it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Bag Of Words parameters & building word vocabulary\n",
    "\n",
    "dbow_ug_model = Doc2Vec(dm=0, vector_size=100, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dbow_ug_model.build_vocab([w_v for w_v in tqdm(word_vec_train)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One caveat of the way this algorithm is that, since the learning rate decrease over the course of iterating over the data, labels which are only seen in a single LabeledSentence during training will only be trained with a fixed learning rate. This frequently produces less than optimal results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below iteration implement explicit multiple-pass, alpha-reduction approach with added shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3751607.30it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4146295.09it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4147855.79it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4028767.00it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4132487.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4271302.23it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4237114.75it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4199004.26it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4193069.67it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4221891.42it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4336461.33it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4241836.43it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4211315.08it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4131311.82it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4146807.50it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4284621.14it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4154714.76it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4194112.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4012946.41it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4179347.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4249042.29it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4242429.05it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4237288.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4215868.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4164284.71it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4196145.06it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4146666.58it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4145022.28it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4193690.67it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4117696.82it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27min 19s, sys: 1min 52s, total: 29min 11s\n",
      "Wall time: 18min 59s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dbow_ug_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train)]), total_examples=len(word_vec_train), epochs=1)\n",
    "    dbow_ug_model.alpha -= 0.002\n",
    "    dbow_ug_model.min_alpha = dbow_ug_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets using above dbow_ug_model (Document Bag Of Words Unigram Model)\n",
    "\n",
    "def vectorize(model, corpus, size):\n",
    "    # Numpy zeros initialization\n",
    "    vectors = np.zeros((len(corpus), size))\n",
    "    \n",
    "    for idx, count in zip(corpus.index, range(len(corpus.index))):\n",
    "        prefix = 'all_' + str(idx)\n",
    "        vectors[count] = model.docvecs[prefix]\n",
    "\n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dbow = vectorize(dbow_ug_model, x_train, 100)\n",
    "val_vecs_dbow = vectorize(dbow_ug_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.10094115e-01, -4.17756796e-01, -2.21768126e-01,\n",
       "         1.46800011e-01, -1.88945279e-01,  7.03278463e-03,\n",
       "         5.17965779e-02,  1.32102147e-01, -3.78717393e-01,\n",
       "         1.81373477e-01,  2.59546846e-01,  3.60601097e-01,\n",
       "        -3.41447711e-01, -2.85278633e-02,  4.47911054e-01,\n",
       "         6.52484521e-02,  3.95397097e-02,  2.68053282e-02,\n",
       "        -1.71497181e-01,  3.02577972e-01,  7.94886146e-03,\n",
       "         1.98269516e-01, -2.79568553e-01,  4.29900661e-02,\n",
       "         4.18800950e-01, -3.09199154e-01, -1.86751425e-01,\n",
       "         2.34446064e-01,  4.91926447e-02,  7.46935308e-02,\n",
       "        -9.96179208e-02, -1.91968068e-01,  1.58625469e-01,\n",
       "         1.78015947e-01,  2.22942159e-01,  3.40586193e-02,\n",
       "         2.42498498e-02, -6.34907261e-02,  3.00550491e-01,\n",
       "        -1.70043394e-01, -3.26719761e-01,  1.29000649e-01,\n",
       "        -3.77572387e-01, -5.38290203e-01, -1.70218259e-01,\n",
       "         6.27116263e-02,  5.01648366e-01, -3.81974764e-02,\n",
       "        -7.97180384e-02,  6.05147064e-01, -2.96413392e-01,\n",
       "        -1.19330622e-01, -5.55795915e-02,  1.83108747e-01,\n",
       "         4.82813150e-01,  1.61903352e-02,  4.10973979e-03,\n",
       "         5.41451395e-01,  1.23052403e-01, -2.62035608e-01,\n",
       "        -2.53300071e-02,  1.80141523e-01,  1.41121864e-01,\n",
       "        -7.30332211e-02, -2.39548966e-01,  5.13505578e-01,\n",
       "        -1.77203845e-02, -4.34440672e-02,  5.68535626e-01,\n",
       "         1.25307113e-01, -1.90070823e-01, -2.59700626e-01,\n",
       "        -1.14151768e-01, -9.09467995e-01,  1.00169815e-02,\n",
       "         2.49821190e-02, -3.28205585e-01, -5.66473484e-01,\n",
       "        -5.06949663e-01,  5.26988268e-01, -4.65795040e-01,\n",
       "        -2.88841218e-01,  4.04271752e-01,  8.83002803e-02,\n",
       "        -2.54730910e-01,  6.43194001e-03,  2.35326484e-01,\n",
       "         6.48466572e-02,  2.63503969e-01, -2.27892607e-01,\n",
       "         1.37172744e-01,  1.13815762e-01,  1.08010262e-01,\n",
       "         5.20638287e-01,  6.10709414e-02,  4.80677605e-01,\n",
       "         3.33484143e-01, -1.12993039e-01,  1.79996625e-01,\n",
       "         6.83277249e-01],\n",
       "       [ 8.84441361e-02,  4.11435887e-02, -1.11366130e-01,\n",
       "         5.08974530e-02, -1.48737550e-01,  2.40089461e-01,\n",
       "        -2.64955848e-01, -5.66833496e-01, -3.00765157e-01,\n",
       "        -3.00845474e-01,  6.23528183e-01, -9.36045051e-02,\n",
       "         4.11023289e-01,  1.52642861e-01,  3.12509626e-01,\n",
       "         2.21295774e-01, -1.01006620e-01,  1.38815641e-01,\n",
       "        -1.04288943e-01,  1.30315349e-01, -3.38175803e-01,\n",
       "         6.37656525e-02,  3.90026383e-02,  4.47897434e-01,\n",
       "        -5.11996187e-02,  3.44206989e-01,  7.19040409e-02,\n",
       "        -2.44712114e-01, -8.37034583e-02, -1.84803128e-01,\n",
       "        -2.77552485e-01, -3.58681053e-01, -5.06151915e-01,\n",
       "        -7.56675676e-02, -8.37269247e-01,  4.44677740e-01,\n",
       "        -3.49808395e-01, -4.96638864e-01, -1.19319167e-02,\n",
       "        -1.22846968e-01,  2.61212826e-01,  7.35739470e-02,\n",
       "         2.39500329e-01,  1.46515921e-01, -5.26423156e-01,\n",
       "         1.29159167e-01,  4.79315035e-03,  9.63594168e-02,\n",
       "         3.53069425e-01,  2.80649643e-02, -4.53251600e-01,\n",
       "        -5.93383014e-01,  4.79928017e-01, -1.26470581e-01,\n",
       "        -3.42575222e-01, -3.53402585e-01,  8.84644091e-02,\n",
       "         4.14988697e-01,  3.01941365e-01, -8.10429752e-02,\n",
       "         3.43828917e-01, -4.44355488e-01,  2.36462742e-01,\n",
       "         1.41367733e-01, -5.43038726e-01,  1.85565650e-01,\n",
       "         1.21213518e-01, -3.00686568e-01,  1.97351590e-01,\n",
       "         2.01499805e-01, -5.28704941e-01,  3.39770198e-01,\n",
       "         6.13549259e-03, -2.09608600e-01,  1.69464484e-01,\n",
       "         1.18535966e-01, -5.92877194e-02,  1.09546460e-01,\n",
       "        -3.84333253e-01,  2.63615072e-01, -4.14252505e-02,\n",
       "         2.96881258e-01, -1.44748494e-01, -5.58803499e-01,\n",
       "        -1.61264807e-01,  8.93458799e-02,  3.70948762e-01,\n",
       "         1.96136519e-01,  3.79283488e-01,  2.61858582e-01,\n",
       "         5.88393033e-01, -2.36751102e-02, -3.20480406e-01,\n",
       "        -4.62804109e-01, -2.69191056e-01,  6.96425200e-01,\n",
       "         1.98636100e-01,  4.86532748e-01, -1.96349949e-01,\n",
       "         5.51578820e-01],\n",
       "       [-1.02327041e-01, -3.93479764e-02, -7.27334201e-01,\n",
       "        -4.92561519e-01,  6.71333373e-02,  4.13284987e-01,\n",
       "         5.43401837e-02, -3.32429171e-01, -6.81836605e-01,\n",
       "         5.67559637e-02,  3.70914191e-01,  1.25868499e-01,\n",
       "         1.99072123e-01, -4.82168347e-01,  2.67142773e-01,\n",
       "        -2.33173847e-01,  2.34666005e-01, -3.39154154e-01,\n",
       "         2.44797245e-01, -1.66671425e-01, -4.87275362e-01,\n",
       "         1.10960998e-01, -1.55045629e-01, -2.84287125e-01,\n",
       "         1.87542275e-01, -2.58376360e-01,  2.31540546e-01,\n",
       "         1.85770750e-01,  2.60301143e-01, -2.03124925e-01,\n",
       "        -1.79130659e-01,  2.56970108e-01,  1.74443454e-01,\n",
       "        -1.38792619e-01,  2.43793011e-01,  8.31232592e-02,\n",
       "        -2.38514364e-01,  5.67257293e-02,  2.09852289e-02,\n",
       "        -4.00598049e-01,  3.38931739e-01,  9.91020501e-02,\n",
       "        -9.79926586e-02,  4.70937878e-01,  5.08350492e-01,\n",
       "         1.75825767e-02,  6.46413490e-02,  7.58232251e-02,\n",
       "        -1.69745937e-01,  1.67807057e-01, -1.31411195e-01,\n",
       "         3.96385342e-01,  9.28597450e-02,  2.56106347e-01,\n",
       "         1.74624438e-03, -3.92606348e-01,  1.51331857e-01,\n",
       "         3.60913396e-01, -2.13284671e-01, -1.70499295e-01,\n",
       "         4.06764656e-01, -6.96415678e-02,  9.47116241e-02,\n",
       "         4.68663454e-01, -5.49035426e-03,  9.64720622e-02,\n",
       "         4.54174578e-01,  1.90135747e-01,  3.75371128e-01,\n",
       "        -1.75625488e-01,  3.03923637e-01, -1.35957673e-01,\n",
       "         2.58198529e-01, -3.31448987e-02, -1.12155773e-01,\n",
       "         1.64930403e-01, -1.26702428e-01,  1.05327051e-02,\n",
       "        -2.45981127e-01, -4.69943658e-02, -2.23650321e-01,\n",
       "         1.44154444e-01,  8.54572803e-02,  1.86440259e-01,\n",
       "        -4.28749442e-01,  3.47786099e-01,  1.07635953e-01,\n",
       "         1.80436194e-01,  2.96966314e-01, -4.23196740e-02,\n",
       "         3.38526666e-01, -2.53891140e-01, -4.62584764e-01,\n",
       "         6.39801472e-02,  3.45353901e-01,  3.39401692e-01,\n",
       "         2.78757751e-01,  2.27436557e-01, -4.41081896e-02,\n",
       "        -1.86499536e-01],\n",
       "       [-5.63367665e-01, -1.46633789e-01, -3.20594758e-01,\n",
       "         2.60801435e-01,  9.37472582e-02,  2.98281014e-01,\n",
       "        -2.43321732e-01, -9.29332245e-03,  7.32020959e-02,\n",
       "         2.86278099e-01,  3.75789911e-01,  2.20113829e-01,\n",
       "        -2.19745815e-01,  9.99465883e-01, -1.18741386e-01,\n",
       "         2.92565346e-01, -5.45422435e-01,  2.11306244e-01,\n",
       "        -4.06271189e-01,  2.16979131e-01, -9.72692668e-02,\n",
       "        -7.38006011e-02,  7.29974061e-02, -5.19469976e-01,\n",
       "         3.49924862e-01, -2.74043739e-01, -4.95324463e-01,\n",
       "         4.47708547e-01,  9.48359430e-01, -7.05682874e-01,\n",
       "        -2.35626251e-01, -8.72347772e-01,  6.52465820e-01,\n",
       "        -3.96173477e-01, -1.64434016e-01, -3.46602380e-01,\n",
       "         2.20198214e-01,  6.46657348e-02, -1.26297519e-01,\n",
       "         3.53272915e-01,  8.62014711e-01,  3.08198869e-01,\n",
       "        -9.16133821e-02,  5.58581412e-01,  5.16546428e-01,\n",
       "        -5.04934229e-02,  1.47994131e-01, -2.00276837e-01,\n",
       "         3.03827584e-01,  2.42428184e-01,  5.06355278e-02,\n",
       "         3.85737628e-01,  4.01436836e-01,  6.78319573e-01,\n",
       "        -1.69873267e-01, -1.28748091e-02,  3.74780715e-01,\n",
       "         3.88501614e-01, -1.70537695e-01,  2.53024884e-02,\n",
       "         9.69808698e-02, -9.17617157e-02, -2.42931947e-01,\n",
       "        -4.47244160e-02, -1.32641783e-02,  7.08809674e-01,\n",
       "        -8.09010625e-01, -4.60060626e-01,  3.79894257e-01,\n",
       "         5.03319383e-01, -2.91510195e-01, -7.59214342e-01,\n",
       "        -8.02150294e-02, -1.19868827e+00, -4.12412405e-01,\n",
       "        -2.86249429e-01,  1.76874995e-01,  6.23543799e-01,\n",
       "        -2.74090379e-01,  1.68595448e-01, -7.71430433e-01,\n",
       "        -3.59061301e-01,  2.62708813e-01, -6.39202297e-01,\n",
       "        -2.57332951e-01, -1.34272456e-01,  2.02632487e-01,\n",
       "         3.92084420e-01, -7.61527270e-02, -9.62313414e-02,\n",
       "         3.02955627e-01,  3.24070305e-01,  2.95654267e-01,\n",
       "        -6.39404416e-01, -3.04449260e-01,  3.91652435e-01,\n",
       "         4.37230647e-01, -6.36042515e-03, -2.78351247e-01,\n",
       "         4.14625049e-01],\n",
       "       [-2.64360577e-01,  1.27900064e-01,  1.11659832e-01,\n",
       "         7.71547332e-02, -1.35072484e-03,  3.52291882e-01,\n",
       "        -1.24969542e-01,  3.22654545e-02, -1.67878672e-01,\n",
       "        -5.35088442e-02,  3.50510448e-01,  1.70340925e-01,\n",
       "         6.21611960e-02, -2.57849783e-01,  4.84607190e-01,\n",
       "        -2.60414541e-01, -1.18220262e-01, -8.56004581e-02,\n",
       "        -2.41053671e-01,  9.36191902e-02, -3.90203357e-01,\n",
       "        -3.16637531e-02, -1.54839545e-01, -5.83241224e-01,\n",
       "        -3.21791321e-02, -3.67777906e-02, -3.11594278e-01,\n",
       "        -2.68467635e-01,  1.54466450e-01, -5.71529031e-01,\n",
       "        -3.08863401e-01, -1.51412889e-01, -1.06394969e-01,\n",
       "        -1.25925168e-01, -1.57674432e-01,  3.00642848e-01,\n",
       "        -1.79815263e-01, -2.78736919e-01, -1.19733848e-01,\n",
       "        -1.49741679e-01,  4.10457760e-01, -1.03242129e-01,\n",
       "         2.75393650e-02, -2.14503873e-02,  2.41194278e-01,\n",
       "         1.36674225e-01,  3.46428245e-01, -3.50339830e-01,\n",
       "         2.10104883e-01,  4.33485895e-01,  1.87540099e-01,\n",
       "         5.58115877e-02, -4.32749726e-02, -5.05273491e-02,\n",
       "         3.55621874e-02, -2.16397598e-01, -2.17676312e-01,\n",
       "         1.33927196e-01,  1.09083399e-01,  1.41434267e-01,\n",
       "         3.44695836e-01, -2.63102740e-01,  2.60687202e-01,\n",
       "         2.26568297e-01, -2.40449548e-01,  3.60589027e-01,\n",
       "        -1.56369865e-01, -1.37637764e-01,  1.52576819e-01,\n",
       "         3.36324573e-01, -1.29503846e-01,  1.89983174e-02,\n",
       "         5.06050825e-01, -9.91564244e-02,  2.41562352e-02,\n",
       "        -5.98045327e-02, -1.26304671e-01, -1.00093789e-01,\n",
       "         6.63390756e-02, -3.96524340e-01, -2.87839621e-01,\n",
       "         3.42268735e-01,  1.12519108e-01,  8.58683437e-02,\n",
       "        -3.70178074e-02,  1.44627377e-01, -8.98892879e-02,\n",
       "        -2.02238306e-01,  3.29748094e-01, -9.17518511e-03,\n",
       "         2.33277857e-01, -1.98065892e-01, -8.21265951e-02,\n",
       "        -8.83227438e-02, -2.84212172e-01,  4.44864035e-01,\n",
       "         2.81023890e-01,  1.97892249e-01, -8.40243399e-02,\n",
       "         8.66452008e-02],\n",
       "       [-5.96973598e-01, -5.44405030e-03, -8.79134461e-02,\n",
       "        -6.72522262e-02, -1.01441526e+00, -6.15900829e-02,\n",
       "         3.54869664e-01, -1.33345187e-01, -4.55320060e-01,\n",
       "         4.63548660e-01, -3.75558853e-01,  3.99548113e-01,\n",
       "         5.09714067e-01,  2.59241760e-01,  2.54485190e-01,\n",
       "        -5.27688384e-01,  1.26977161e-01, -7.73901120e-02,\n",
       "        -1.53371841e-01,  1.64425984e-01, -3.45212281e-01,\n",
       "         1.45758346e-01, -6.65903091e-01,  4.04840082e-01,\n",
       "        -2.39156693e-01, -5.30147016e-01,  2.31897786e-01,\n",
       "        -1.78594649e-01,  9.12474573e-01, -2.14467019e-01,\n",
       "         1.66024610e-01, -3.70420724e-01, -2.96221711e-02,\n",
       "         4.08067256e-01,  4.79924768e-01,  1.40950829e-02,\n",
       "         8.57512057e-02, -4.56310868e-01,  5.99239051e-01,\n",
       "         1.21226937e-01,  3.73089820e-01,  5.42435408e-01,\n",
       "         3.64453852e-01,  3.19480985e-01,  8.11221749e-02,\n",
       "        -4.65863377e-01, -1.05906978e-01,  4.25655693e-01,\n",
       "         2.94210374e-01, -9.15771872e-02, -7.88316250e-01,\n",
       "        -3.15158218e-02, -1.46883801e-01,  4.19487298e-01,\n",
       "         6.16964102e-02,  2.49877438e-01, -3.67568523e-01,\n",
       "         7.09889412e-01, -2.94525206e-01, -8.11107606e-02,\n",
       "        -1.38915675e-02, -1.98622607e-02,  4.45029259e-01,\n",
       "        -4.29390222e-01,  5.39591789e-01,  2.05973703e-02,\n",
       "         4.51992825e-02, -1.34097293e-01,  4.48300779e-01,\n",
       "         2.12517649e-01, -4.24339660e-02, -1.59888402e-01,\n",
       "        -4.05889571e-01,  7.26722538e-01,  1.69057027e-01,\n",
       "         8.16973746e-02, -3.17439027e-02,  1.07470818e-01,\n",
       "        -1.84153736e-01,  3.19478542e-01, -2.17993185e-01,\n",
       "         5.07203758e-01,  3.13237220e-01, -1.44606724e-01,\n",
       "        -1.80541232e-01,  7.78009474e-01,  3.54217023e-01,\n",
       "        -3.16450983e-01, -2.72067487e-01, -1.10232331e-01,\n",
       "        -2.97011614e-01, -3.41193438e-01, -2.16097385e-01,\n",
       "        -2.44827256e-01,  4.90321696e-01,  6.06388509e-01,\n",
       "         1.52668446e-01,  3.80783617e-01, -2.91599631e-01,\n",
       "         3.56070399e-01],\n",
       "       [-1.87385917e-01,  8.42796266e-02, -1.88072130e-01,\n",
       "         1.53432544e-02,  1.58027753e-01,  2.46003687e-01,\n",
       "         2.70984858e-01, -7.79396892e-02, -3.67368385e-02,\n",
       "         5.58864586e-02,  1.11829266e-01, -8.63479897e-02,\n",
       "         7.28401765e-02, -1.33907184e-01,  2.94963747e-01,\n",
       "        -9.40888226e-02,  5.71754724e-02, -7.92883113e-02,\n",
       "         8.03018138e-02,  8.63481387e-02, -1.03659391e-01,\n",
       "         8.26761425e-02,  6.56743273e-02, -6.14229701e-02,\n",
       "         2.69094855e-01, -1.43953592e-01, -7.57078826e-02,\n",
       "        -1.88593522e-01,  1.47772118e-01, -3.96398574e-01,\n",
       "        -2.52381146e-01, -1.81270123e-01,  7.47133419e-02,\n",
       "        -1.13758825e-01,  1.25368997e-01,  1.50067970e-01,\n",
       "        -1.57714099e-01, -1.17680565e-01, -3.06144685e-01,\n",
       "        -2.53107190e-01,  1.61256105e-01, -1.43110007e-01,\n",
       "        -5.18815368e-02, -6.07840940e-02,  2.17110640e-03,\n",
       "         1.30324885e-01, -1.19182561e-03, -1.18248686e-01,\n",
       "        -1.53615817e-01,  1.64417431e-01,  9.61183012e-02,\n",
       "         7.73887783e-02,  1.71721056e-01, -1.66365318e-02,\n",
       "        -2.53780317e-02,  2.29781508e-01, -5.98947704e-02,\n",
       "         3.83411795e-01, -1.85590953e-01, -1.80930167e-01,\n",
       "         1.94460265e-02, -2.56834090e-01, -5.53037710e-02,\n",
       "        -3.18591505e-01,  3.15088741e-02,  2.91655809e-01,\n",
       "         2.49851383e-02, -2.97853857e-01,  1.88889638e-01,\n",
       "         9.78308395e-02, -9.71948951e-02,  7.86878690e-02,\n",
       "         2.85374131e-02, -1.44875601e-01,  1.79883718e-01,\n",
       "         1.88836098e-01, -9.26145688e-02, -5.14575839e-03,\n",
       "        -2.96757489e-01,  2.34797776e-01, -9.60598364e-02,\n",
       "         4.37983684e-02,  1.53431132e-01,  1.07347465e-03,\n",
       "        -2.00016797e-02, -1.23241708e-01,  2.04938576e-01,\n",
       "        -3.08653079e-02, -2.82113582e-01, -2.48862088e-01,\n",
       "        -8.82004127e-02,  2.32803472e-03,  1.97919115e-01,\n",
       "        -3.72272372e-01, -1.03223979e-01,  1.90873817e-01,\n",
       "         2.58683413e-01,  2.40155652e-01,  7.39399791e-02,\n",
       "         3.09639484e-01],\n",
       "       [ 3.31694260e-02, -1.75390571e-01, -2.72250891e-01,\n",
       "        -1.02387555e-01,  1.25883240e-02, -5.60566485e-02,\n",
       "         3.81453708e-02,  1.83852389e-01,  5.54934442e-02,\n",
       "         1.53385043e-01,  3.37316781e-01,  3.39803427e-01,\n",
       "         1.58046827e-01, -4.96315807e-02,  1.98175669e-01,\n",
       "        -1.10868342e-01,  8.12067837e-02,  5.09006483e-03,\n",
       "        -1.19307965e-01,  3.78682837e-02, -7.27584958e-02,\n",
       "         2.93748882e-02, -1.76439360e-01,  1.63779169e-01,\n",
       "         3.12768430e-01, -1.46838367e-01,  1.07811227e-01,\n",
       "        -2.33156100e-01,  1.27745330e-01, -6.80858493e-02,\n",
       "         1.24208583e-02, -3.31618965e-01,  4.20101508e-02,\n",
       "        -6.52680919e-02,  3.14511955e-01,  2.00848117e-01,\n",
       "        -5.62939607e-02, -3.00984174e-01,  3.54579508e-01,\n",
       "         1.04500420e-01,  2.37430539e-02, -2.47211039e-01,\n",
       "         2.95941252e-02, -1.00000994e-02,  1.04649417e-01,\n",
       "        -1.82535246e-01,  3.35448459e-02, -2.43382201e-01,\n",
       "         1.64677411e-01,  3.51093322e-01,  4.53917116e-01,\n",
       "        -4.59628552e-02, -2.93978214e-01,  1.63379535e-01,\n",
       "         1.25534832e-01, -5.61293781e-01, -2.78131038e-01,\n",
       "         3.33201230e-01,  7.18626939e-03,  3.04237008e-01,\n",
       "        -4.71340939e-02, -2.86791235e-01,  1.59060106e-01,\n",
       "         6.44922033e-02, -1.18715212e-01,  2.14011386e-01,\n",
       "        -1.64907902e-01, -1.13233574e-01,  3.41512024e-01,\n",
       "         7.34175369e-02,  7.98583403e-02, -1.33529827e-01,\n",
       "        -9.35154632e-02, -2.06918582e-01, -4.77682054e-02,\n",
       "         1.42589778e-01,  1.27463579e-01, -1.02351449e-01,\n",
       "        -1.71830088e-01,  9.14016888e-02, -9.75441039e-02,\n",
       "        -8.04022476e-02,  1.85627621e-02, -3.47194709e-02,\n",
       "        -4.43904437e-02,  2.06116050e-01,  1.90191418e-02,\n",
       "        -7.78002059e-03, -1.55750349e-01, -2.39028722e-01,\n",
       "         2.84676582e-01,  1.93123013e-01, -1.27098829e-01,\n",
       "        -3.80513012e-01,  7.78305605e-02,  3.19951206e-01,\n",
       "         3.74598205e-01,  4.97872457e-02,  1.18179945e-03,\n",
       "         2.04207644e-01],\n",
       "       [-1.16934530e-01,  1.67681009e-01,  1.12737797e-01,\n",
       "         1.77290067e-01,  5.21935463e-01,  3.91364872e-01,\n",
       "        -1.78262070e-01, -8.26305151e-02,  9.29501560e-03,\n",
       "        -2.65159339e-01,  1.66139588e-01, -1.08647324e-01,\n",
       "        -2.48107135e-01, -1.08539118e-02,  6.25049055e-01,\n",
       "        -4.16909844e-01,  3.88346091e-02, -3.73603016e-01,\n",
       "        -1.54741094e-01,  2.72209704e-01, -4.42236334e-01,\n",
       "        -5.26430309e-02, -2.78292708e-02, -7.66619593e-02,\n",
       "         6.70318305e-02,  1.58709154e-01, -2.70342559e-01,\n",
       "         1.82378843e-01,  3.90402079e-02, -1.87220648e-01,\n",
       "        -1.16504185e-01, -2.57819206e-01,  2.52311915e-01,\n",
       "         3.50276858e-01, -1.23371795e-01, -3.41013223e-01,\n",
       "        -3.82775128e-01, -5.55718720e-01, -2.27358669e-01,\n",
       "        -2.77379692e-01,  1.47331059e-01, -3.18370253e-01,\n",
       "        -9.11528990e-02,  2.73339868e-01,  2.37457216e-01,\n",
       "        -4.06899840e-01, -3.04028783e-02, -4.57288846e-02,\n",
       "         5.26371777e-01,  6.09299541e-01, -5.45895807e-02,\n",
       "        -1.09053895e-01,  2.36018002e-02,  6.34168535e-02,\n",
       "         3.59631717e-01,  1.31118476e-01, -3.99210811e-01,\n",
       "         3.13411653e-01, -3.84079069e-01,  1.25815332e-01,\n",
       "         1.08532846e-01,  3.56266379e-01,  5.48676014e-01,\n",
       "         7.77381705e-03, -2.02552453e-02,  2.34980062e-01,\n",
       "         1.22667894e-01, -6.86060265e-02,  4.79582846e-01,\n",
       "         4.77470607e-01,  2.81972349e-01, -5.34707963e-01,\n",
       "         1.42762378e-01, -2.97736377e-01, -1.21604477e-03,\n",
       "         3.14174265e-01, -8.60688090e-02, -1.90656319e-01,\n",
       "        -1.79357737e-01,  2.03524068e-01, -4.20602947e-01,\n",
       "        -1.57053471e-01, -1.30981475e-01,  9.16363746e-02,\n",
       "         1.83049485e-01, -3.57766360e-01, -1.12050928e-01,\n",
       "         3.09794992e-01,  1.73522457e-01, -2.90985167e-01,\n",
       "         1.56253040e-01,  4.45658892e-01, -1.23126477e-01,\n",
       "        -3.63117784e-01,  2.66368985e-01,  7.19166517e-01,\n",
       "        -8.92350897e-02, -1.68934256e-01,  2.58152127e-01,\n",
       "         2.65577048e-01],\n",
       "       [ 3.12904827e-02,  8.98391753e-02, -1.07615486e-01,\n",
       "        -3.61804962e-01,  4.17472899e-01,  6.12905733e-02,\n",
       "         2.54992634e-01,  2.21909061e-02, -2.65263647e-01,\n",
       "        -3.81327093e-01,  3.14309634e-02,  3.49262625e-01,\n",
       "        -2.18481362e-01, -2.18662351e-01,  5.31520665e-01,\n",
       "        -2.94444878e-02,  1.05179496e-01,  5.77324890e-02,\n",
       "        -1.47410994e-02, -4.03953552e-01, -2.96891123e-01,\n",
       "        -4.58907150e-02, -6.52828515e-01,  2.29620278e-01,\n",
       "        -6.27888963e-02,  4.19883698e-01, -7.21382678e-01,\n",
       "        -1.37269869e-01,  3.23234111e-01, -4.65559363e-01,\n",
       "         1.18118368e-01, -3.86831552e-01,  4.04760927e-01,\n",
       "         2.29537748e-02,  4.35432605e-02,  2.56982949e-02,\n",
       "        -2.33474106e-01, -2.83340141e-02, -4.10213917e-02,\n",
       "        -4.09175068e-01,  4.54602212e-01, -4.98300120e-02,\n",
       "        -1.32414773e-01, -1.55155137e-01,  4.05266553e-01,\n",
       "        -2.42777422e-01, -3.70170414e-01,  2.48308368e-02,\n",
       "         3.27006459e-01,  1.44624457e-01, -9.55731869e-02,\n",
       "        -1.86886638e-01,  4.63845069e-03,  3.94135773e-01,\n",
       "         3.33817564e-02,  1.06255680e-01,  5.51248305e-02,\n",
       "         3.72095108e-01, -6.02943480e-01,  2.66624480e-01,\n",
       "        -1.20373756e-01, -1.83095291e-01,  7.64347315e-01,\n",
       "        -4.02562469e-01,  5.05241454e-01,  4.12913978e-01,\n",
       "        -3.43310982e-01,  1.94781704e-03, -8.04615319e-02,\n",
       "         1.69444919e-01,  6.33815706e-01,  3.56587738e-01,\n",
       "        -1.39464915e-01,  8.22409540e-02, -1.35598570e-01,\n",
       "         3.09107244e-01, -9.02692750e-02, -8.20718408e-02,\n",
       "        -2.27763966e-01,  1.38404965e-01, -1.84738547e-01,\n",
       "         1.49863064e-01,  2.73751825e-01, -8.88544694e-02,\n",
       "        -7.69460201e-02,  3.58092219e-01,  2.18243062e-01,\n",
       "         3.32689762e-01,  1.58326983e-01,  5.23973465e-01,\n",
       "         2.68130243e-01, -1.37599587e-01,  1.39859328e-02,\n",
       "        -6.16541982e-01, -3.49405408e-01,  5.36158621e-01,\n",
       "         4.18236494e-01,  2.06434265e-01,  9.76303145e-02,\n",
       "        -2.31414974e-01]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_vecs_dbow[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7320625"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW model doesn't learn the semantic understanding of words but it's features obtained from it does a decent job with a simple Logistic Regression classifier.\n",
    "The result doesn't seem to excel count vectorizer or Tfidf vectorizer. It might even not be a direct comparison as count vectorizer or tfidf vectorizer uses a large number of features to represent a tweet rather than using 200 dimensions as in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dbow_ug_model\n",
    "\n",
    "dbow_ug_model.save('./data/dbow_ug_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dbow_ug_model and delete temporary training data\n",
    "\n",
    "dbow_ug_model = Doc2Vec.load('./data/dbow_ug_model.doc2vec')\n",
    "dbow_ug_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed Memory "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Concatenated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4041921.05it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory parameters & building word vocabulary\n",
    "\n",
    "dm_ug_model = Doc2Vec(dm=1, dm_concat=1, vector_size=100, window=2, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dm_ug_model.build_vocab([w_v for w_v in tqdm(word_vec_train)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3830051.72it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3978165.31it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3984594.78it/s]\n",
      "100%|██████████| 1600000/1600000 [00:01<00:00, 1241856.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3972896.94it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3864424.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4043486.99it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4053387.97it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4112617.14it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4072470.94it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4052362.41it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4083731.86it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4178236.07it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4120113.63it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4106891.38it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4122611.78it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4024799.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3932004.87it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4119094.48it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4113711.25it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3951438.70it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4113890.30it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4067134.86it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3897927.52it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4033062.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3991802.36it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3815839.50it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4138613.78it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4052325.70it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4087659.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37min 59s, sys: 2min 20s, total: 40min 20s\n",
      "Wall time: 21min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dm_ug_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train)]), total_examples=len(word_vec_train), epochs=1)\n",
    "    dm_ug_model.alpha -= 0.002\n",
    "    dm_ug_model.min_alpha = dm_ug_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_ug_model\n",
    "\n",
    "dm_ug_model.save('./data/dm_ug_model.doc2vec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Doc2Vec, one can also retrieve individual word vectors alongwith document vectors. \n",
    "However, a Doc2Vec DBOW model doesn't learn the semantic meaning of the words. Hence the word vectors retrieved from pure DBOW model will be the automatic randomly-initialized vectors with no meaning. \n",
    "But in DM model, the word vectors has the semantic understanding about words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('gd', 0.795044481754303),\n",
       " ('goood', 0.743361234664917),\n",
       " ('great', 0.7367996573448181),\n",
       " ('gud', 0.7260905504226685),\n",
       " ('goodly', 0.682033360004425),\n",
       " ('gooooooood', 0.6817256212234497),\n",
       " ('gooood', 0.6782906651496887),\n",
       " ('goooood', 0.6629056930541992),\n",
       " ('gooooooooooooood', 0.6597786545753479),\n",
       " ('gooooood', 0.6576763391494751)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'good'\n",
    "\n",
    "dm_ug_model.most_similar('good')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('hapi', 0.7175722718238831),\n",
       " ('hapy', 0.7089468836784363),\n",
       " ('happyy', 0.6959027647972107),\n",
       " ('happppy', 0.6892277002334595),\n",
       " ('pleased', 0.6851349472999573),\n",
       " ('ebar', 0.6754604578018188),\n",
       " ('happpy', 0.672626256942749),\n",
       " ('happpppy', 0.6442862749099731),\n",
       " ('delighted', 0.6398066878318787),\n",
       " ('happpyyyyy', 0.6274683475494385)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'happy'\n",
    "\n",
    "dm_ug_model.most_similar('happy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('bing', 0.6888588070869446),\n",
       " ('yahoo', 0.6703177094459534),\n",
       " ('gmail', 0.6586295962333679),\n",
       " ('linkedin', 0.6292126178741455),\n",
       " ('stocktwits', 0.6011006832122803),\n",
       " ('mixero', 0.5956629514694214),\n",
       " ('wikipedia', 0.5900527238845825),\n",
       " ('facebook', 0.5883870720863342),\n",
       " ('blogspot', 0.5877808928489685),\n",
       " ('myfoodtrip', 0.5857887864112854)]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'google'\n",
    "\n",
    "dm_ug_model.most_similar('google')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's interesting that the model has also learnt about corrupted form of the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('smaller', 0.6095108389854431),\n",
       " ('fewer', 0.5827668905258179),\n",
       " ('larger', 0.5684505701065063),\n",
       " ('tastier', 0.554661214351654),\n",
       " ('shorter', 0.5472672581672668),\n",
       " ('deadlier', 0.5471574068069458),\n",
       " ('tiny', 0.5415958762168884),\n",
       " ('awesumer', 0.5377195477485657),\n",
       " ('saner', 0.5373097658157349),\n",
       " ('deader', 0.5307070016860962)]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Words similar to the equation : Embed(bigger) + Embed(small) - Embed('big)\n",
    "\n",
    "dm_ug_model.most_similar(positive=['bigger', 'small'], negative=['big'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dm = vectorize(dm_ug_model, x_train, 100)\n",
    "val_vecs_dm = vectorize(dm_ug_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.668125"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dm, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_ug_model\n",
    "\n",
    "dm_ug_model.save('./data/dm_ug_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dm_ug_model and delete temporary training data\n",
    "\n",
    "dm_ug_model = Doc2Vec.load('./data/dm_ug_model.doc2vec')\n",
    "dm_ug_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed Memory "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4000905.24it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory Mean parameters & building word vocabulary\n",
    "\n",
    "dmm_ug_model = Doc2Vec(dm=1, dm_mean=1, vector_size=100, window=4, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dmm_ug_model.build_vocab([w_v for w_v in tqdm(word_vec_train)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3817282.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4133260.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4141032.21it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4269630.96it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4155182.95it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4129372.20it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4222353.62it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4197985.11it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4180500.51it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4130457.45it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3914726.47it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4134643.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4104068.36it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4164393.25it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4107939.70it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4121277.54it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4211143.31it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4077920.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4150313.27it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4306772.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4211455.15it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4198263.49it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4245571.93it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4121811.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4233760.00it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4200271.01it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4124045.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4178610.71it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4124035.59it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4025837.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 49min 54s, sys: 9min 32s, total: 59min 26s\n",
      "Wall time: 35min 49s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dmm_ug_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train)]), total_examples=len(word_vec_train), epochs=1)\n",
    "    dmm_ug_model.alpha -= 0.002\n",
    "    dmm_ug_model.min_alpha = dmm_ug_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_ug_model\n",
    "\n",
    "dmm_ug_model.save('./data/dmm_ug_model.doc2vec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Doc2Vec, one can also retrieve individual word vectors alongwith document vectors. \n",
    "However, a Doc2Vec DBOW model doesn't learn the semantic meaning of the words. Hence the word vectors retrieved from pure DBOW model will be the automatic randomly-initialized vectors with no meaning. \n",
    "But in DM model, the word vectors has the semantic understanding about words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('great', 0.9271145462989807),\n",
       " ('bad', 0.8902649283409119),\n",
       " ('nice', 0.8806440830230713),\n",
       " ('sad', 0.8632817268371582),\n",
       " ('wonderful', 0.8587142825126648),\n",
       " ('alone', 0.856594979763031),\n",
       " ('busy', 0.8555803894996643),\n",
       " ('weird', 0.8548664450645447),\n",
       " ('that', 0.8542935848236084),\n",
       " ('but', 0.8539254665374756)]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'good'\n",
    "\n",
    "dmm_ug_model.most_similar('good')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('sad', 0.8552143573760986),\n",
       " ('bummed', 0.8075793385505676),\n",
       " ('upset', 0.8072896599769592),\n",
       " ('excited', 0.8055294752120972),\n",
       " ('busy', 0.7970961928367615),\n",
       " ('lame', 0.7911461591720581),\n",
       " ('good', 0.7825235724449158),\n",
       " ('nervous', 0.7758963704109192),\n",
       " ('jealous', 0.7745265960693359),\n",
       " ('sick', 0.7709179520606995)]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'happy'\n",
    "\n",
    "dmm_ug_model.most_similar('happy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('smaller', 0.7307447195053101),\n",
       " ('nicer', 0.6246125102043152),\n",
       " ('shorter', 0.6226528286933899),\n",
       " ('cheaper', 0.6045967936515808),\n",
       " ('larger', 0.59166419506073),\n",
       " ('useable', 0.58817058801651),\n",
       " ('clearer', 0.5850169658660889),\n",
       " ('higher', 0.5828668475151062),\n",
       " ('tiny', 0.5705291628837585),\n",
       " ('different', 0.5560811758041382)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Words similar to the equation : Embed(bigger) + Embed(small) - Embed('big)\n",
    "\n",
    "dmm_ug_model.most_similar(positive=['bigger', 'small'], negative=['big'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dmm = vectorize(dmm_ug_model, x_train, 100)\n",
    "val_vecs_dmm = vectorize(dmm_ug_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dmm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7258125"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dmm, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dmm_ug_model\n",
    "\n",
    "dmm_ug_model.save('./data/dmm_ug_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dmm_ug_model and delete temporary training data\n",
    "\n",
    "dmm_ug_model = Doc2Vec.load('./data/dmm_ug_model.doc2vec')\n",
    "dmm_ug_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pretty Nice.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display('Pretty Nice.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training on concatanated document vectors obtained from Distributed Bag Of Words & Distributed Memory Concatanated models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Updated vectorize_concate function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets using above dbow_ug_model (Document Bag Of Words Unigram Model)\n",
    "\n",
    "def vectorize_concate(model1, model2, corpus, size):\n",
    "    # Numpy zeros initialization\n",
    "    vectors = np.zeros((len(corpus), size))\n",
    "    \n",
    "    for idx, count in zip(corpus.index, range(len(corpus.index))):\n",
    "        prefix = 'all_' + str(idx)\n",
    "        # Appending document vectors\n",
    "        vectors[count] = np.append(model1.docvecs[prefix], model2.docvecs[prefix])\n",
    "\n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "train_vecs_dbow_dm = vectorize_concate(dbow_ug_model, dm_ug_model, x_train, 200)\n",
    "val_vecs_dbow_dm = vectorize_concate(dbow_ug_model, dm_ug_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.742375"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dm, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training on concatanated document vectors obtained from Distributed Bag Of Words & Distributed Memory Mean models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory Mean \n",
    "\n",
    "train_vecs_dbow_dmm = vectorize_concate(dbow_ug_model, dmm_ug_model, x_train, 200)\n",
    "val_vecs_dbow_dmm = vectorize_concate(dbow_ug_model, dmm_ug_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory Mean\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dmm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7504375"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dmm, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Populate table with Models & it's Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "mydata = [['Distributed Bag Of Words', 0.7320625], \n",
    "          ['Distributed Memory Concatanated', 0.668125],\n",
    "          ['Distributed Memory Mean', 0.7258125],\n",
    "          ['Distributed Memory BoW & Concatanated', 0.742375],\n",
    "          ['Distributed Memory BoW & Mean', 0.7504375]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>Model                                </th><th style=\"text-align: right;\">  Accuracy</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>Distributed Bag Of Words             </td><td style=\"text-align: right;\">    0.7321</td></tr>\n",
       "<tr><td>Distributed Memory Concatanated      </td><td style=\"text-align: right;\">    0.6681</td></tr>\n",
       "<tr><td>Distributed Memory Mean              </td><td style=\"text-align: right;\">    0.7258</td></tr>\n",
       "<tr><td>Distributed Memory BoW & Concatanated</td><td style=\"text-align: right;\">    0.7424</td></tr>\n",
       "<tr><td>Distributed Memory BoW & Mean        </td><td style=\"text-align: right;\">    0.7504</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tabulate import tabulate\n",
    "from IPython.display import HTML\n",
    "\n",
    "display(HTML(tabulate(mydata, headers= ['Model', 'Accuracy'], floatfmt='.4f', tablefmt='html')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phrase Modeling¶"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenizing train set\n",
    "\n",
    "tokenized_train = [tweet.split() for tweet in x_train]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenized tweets corpus will be fed to genism library Phrase functions to get the frequently used phrases and connect them together with underbar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 4s, sys: 222 ms, total: 1min 4s\n",
      "Wall time: 1min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Getting Phrases from the tokens & thereafter bigram token from the phrases\n",
    "\n",
    "phrases = Phrases(tokenized_train)\n",
    "bigram = Phraser(phrases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'mayor', 'of', 'new_york', 'was', 'there']"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example\n",
    "\n",
    "check = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']\n",
    "display(bigram[check])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see that the bigram can find out the most frequently used phrase in the example \"new_york\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labelling tweets using genism phrase library for unsupervised learning\n",
    "\n",
    "def label_tweets_bigram(tweets, label):\n",
    "    result = []\n",
    "    prefix = label\n",
    "    \n",
    "    # Split tweets & attach label with index\n",
    "    for index, tweet in zip(tweets.index, tweets):\n",
    "        result.append(LabeledSentence(bigram[tweet.split()], [prefix + '_%s' % index]))\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:9: DeprecationWarning: Call to deprecated `LabeledSentence` (Class will be removed in 4.0.0, use TaggedDocument instead).\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1600000"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vec_train_bg = label_tweets_bigram(df.text , 'all')\n",
    "len(word_vec_train_bg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All cores of CPU\n",
    "\n",
    "cores = multiprocessing.cpu_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DBOW (Document Bag Of Words) Bigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW Bigram: This is the Doc2Vec model analogous to Skip-gram model in Word2Vec. The paragraph vectors are obtained by training a neural network on the task of predicting a probability distribution of words in a paragraph given a randomly-sampled word from the paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4073397.91it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Bag Of Words Bigram parameters & building word vocabulary\n",
    "\n",
    "dbow_bg_model = Doc2Vec(dm=0, vector_size=100, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dbow_bg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_bg)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One caveat of the way this algorithm is that, since the learning rate decrease over the course of iterating over the data, labels which are only seen in a single LabeledSentence during training will only be trained with a fixed learning rate. This frequently produces less than optimal results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below iteration implement explicit multiple-pass, alpha-reduction approach with added shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3964611.74it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3915788.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4228147.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4056659.02it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4037393.24it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4105476.88it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3941255.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4115886.08it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4247767.46it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4174262.32it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3970551.04it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4251526.89it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3975575.30it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3935482.10it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4351857.56it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3976069.95it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3870703.51it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4033723.97it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3876169.79it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3883336.15it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4014115.38it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4042902.36it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3776355.82it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3943175.48it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4095205.15it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4095025.23it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4183038.55it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4218409.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3534430.28it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4263511.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 29min 6s, sys: 2min 9s, total: 31min 16s\n",
      "Wall time: 20min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dbow_bg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_bg)]), total_examples=len(word_vec_train_bg), epochs=1)\n",
    "    dbow_bg_model.alpha -= 0.002\n",
    "    dbow_bg_model.min_alpha = dbow_bg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets using above dbow_ug_model (Document Bag Of Words Unigram Model)\n",
    "\n",
    "def vectorize(model, corpus, size):\n",
    "    # Numpy zeros initialization\n",
    "    vectors = np.zeros((len(corpus), size))\n",
    "    \n",
    "    for idx, count in zip(corpus.index, range(len(corpus.index))):\n",
    "        prefix = 'all_' + str(idx)\n",
    "        vectors[count] = model.docvecs[prefix]\n",
    "\n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dbow_bg = vectorize(dbow_bg_model, x_train, 100)\n",
    "val_vecs_dbow_bg = vectorize(dbow_bg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_bg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.742875"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_bg, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW model doesn't learn the semantic understanding of words but it's features obtained from it does a decent job with a simple Logistic Regression classifier.\n",
    "The result doesn't seem to excel count vectorizer or Tfidf vectorizer. It might even not be a direct comparison as count vectorizer or tfidf vectorizer uses a large number of features to represent a tweet rather than using 200 dimensions as in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dbow_bg_model\n",
    "\n",
    "dbow_bg_model.save('./data/dbow_bg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dbow_ug_model and delete temporary training data\n",
    "\n",
    "dbow_bg_model = Doc2Vec.load('./data/dbow_bg_model.doc2vec')\n",
    "dbow_bg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed Memory Bigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Concatenated Bigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4037378.66it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory Bigram parameters & building word vocabulary\n",
    "\n",
    "dm_bg_model = Doc2Vec(dm=1, dm_concat=1, vector_size=100, window=2, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dm_bg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_bg)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3899667.09it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4049567.40it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3943645.87it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3983012.65it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4088690.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3595165.22it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4086942.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3979476.91it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3929414.78it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3976567.07it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4034230.77it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3730645.40it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4059670.12it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3739878.39it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4002217.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4064220.99it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4045927.21it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3812871.49it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4020796.58it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3982934.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4070847.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4048573.08it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4132759.46it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3873849.49it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3970208.09it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3968271.27it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4118384.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4056180.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4050080.63it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3999322.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 38min 37s, sys: 2min 15s, total: 40min 52s\n",
      "Wall time: 22min 14s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dm_bg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_bg)]), total_examples=len(word_vec_train_bg), epochs=1)\n",
    "    dm_bg_model.alpha -= 0.002\n",
    "    dm_bg_model.min_alpha = dm_bg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_bg_model\n",
    "\n",
    "dm_bg_model.save('./data/dm_bg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Doc2Vec, one can also retrieve individual word vectors alongwith document vectors. \n",
    "However, a Doc2Vec DBOW model doesn't learn the semantic meaning of the words. Hence the word vectors retrieved from pure DBOW model will be the automatic randomly-initialized vectors with no meaning. \n",
    "But in DM model, the word vectors has the semantic understanding about words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('great', 0.7463171482086182),\n",
       " ('gooood', 0.7296287417411804),\n",
       " ('gd', 0.7225816249847412),\n",
       " ('goood', 0.7052001357078552),\n",
       " ('gud', 0.6986550092697144),\n",
       " ('haveno', 0.6722873449325562),\n",
       " ('horrrrible', 0.6549490690231323),\n",
       " ('nice', 0.6540406346321106),\n",
       " ('goooood', 0.6462099552154541),\n",
       " ('lebay', 0.6451210379600525)]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'good'\n",
    "\n",
    "dm_bg_model.most_similar('good')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('pleased', 0.6988117694854736),\n",
       " ('greatful', 0.6522352695465088),\n",
       " ('hapy', 0.6349790692329407),\n",
       " ('delighted', 0.6305642127990723),\n",
       " ('thankful', 0.6269996166229248),\n",
       " ('proud', 0.622420608997345),\n",
       " ('happpy', 0.6171808838844299),\n",
       " ('thrilled', 0.6133846044540405),\n",
       " ('happpppy', 0.609881579875946),\n",
       " ('upset', 0.6092841625213623)]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'happy'\n",
    "\n",
    "dm_bg_model.most_similar('happy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('smaller', 0.5979503989219666),\n",
       " ('considerably_less', 0.5903685092926025),\n",
       " ('poorer', 0.5861400961875916),\n",
       " ('larger', 0.5799989104270935),\n",
       " ('reclined', 0.5764734148979187),\n",
       " ('different', 0.5636833310127258),\n",
       " ('healthier', 0.5633150339126587),\n",
       " ('stuffier', 0.5453404188156128),\n",
       " ('venomous', 0.5452708005905151),\n",
       " ('shorter', 0.5448837876319885)]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Words similar to the equation : Embed(bigger) + Embed(small) - Embed('big)\n",
    "\n",
    "dm_bg_model.most_similar(positive=['bigger', 'small'], negative=['big'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dm_bg = vectorize(dm_bg_model, x_train, 100)\n",
    "val_vecs_dm_bg = vectorize(dm_bg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dm_bg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6571875"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dm_bg, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_bg_model\n",
    "\n",
    "dm_bg_model.save('./data/dm_bg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dm_bg_model and delete temporary training data\n",
    "\n",
    "dm_bg_model = Doc2Vec.load('./data/dm_bg_model.doc2vec')\n",
    "dm_bg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Bigram Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4160714.04it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory Bigram Mean parameters & building word vocabulary\n",
    "\n",
    "dmm_bg_model = Doc2Vec(dm=1, dm_mean=1, vector_size=100, window=4, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dmm_bg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_bg)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3632233.96it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4175659.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4068483.60it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4160414.82it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4041168.95it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4119794.93it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4088239.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4134373.67it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3936137.65it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4103855.04it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4157388.98it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4185503.98it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4066237.84it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4090781.66it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4051797.23it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4106305.87it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4092790.02it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4034839.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4199277.52it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4220324.94it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3935069.03it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4106047.08it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4109685.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3857491.92it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4120672.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3983873.32it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4140023.12it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4168858.33it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4185180.31it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3917610.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 50min 59s, sys: 9min 45s, total: 1h 45s\n",
      "Wall time: 36min 41s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dmm_bg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_bg)]), total_examples=len(word_vec_train_bg), epochs=1)\n",
    "    dmm_bg_model.alpha -= 0.002\n",
    "    dmm_bg_model.min_alpha = dmm_bg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dmm_bg_model\n",
    "\n",
    "dmm_bg_model.save('./data/dmm_bg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Doc2Vec, one can also retrieve individual word vectors alongwith document vectors. \n",
    "However, a Doc2Vec DBOW model doesn't learn the semantic meaning of the words. Hence the word vectors retrieved from pure DBOW model will be the automatic randomly-initialized vectors with no meaning. \n",
    "But in DM model, the word vectors has the semantic understanding about words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('great', 0.9449338316917419),\n",
       " ('bad', 0.9191789627075195),\n",
       " ('nice', 0.9140607714653015),\n",
       " ('sad', 0.8977138996124268),\n",
       " ('cool', 0.8960317373275757),\n",
       " ('ok', 0.8918680548667908),\n",
       " ('that', 0.8916789293289185),\n",
       " ('you', 0.8908518552780151),\n",
       " ('awesome', 0.8881815075874329),\n",
       " ('crazy', 0.8871078491210938)]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'good'\n",
    "\n",
    "dmm_bg_model.most_similar('good')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('sad', 0.9064098000526428),\n",
       " ('excited', 0.8870008587837219),\n",
       " ('upset', 0.8546335101127625),\n",
       " ('lucky', 0.8500240445137024),\n",
       " ('cool', 0.8468711376190186),\n",
       " ('jealous', 0.8455446362495422),\n",
       " ('depressed', 0.8413048982620239),\n",
       " ('good', 0.8396432995796204),\n",
       " ('bummed', 0.837196946144104),\n",
       " ('nice', 0.8340071439743042)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similar words to 'happy'\n",
    "\n",
    "dmm_bg_model.most_similar('happy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('smaller', 0.7743483185768127),\n",
       " ('shorter', 0.6514254808425903),\n",
       " ('tiny', 0.6247305870056152),\n",
       " ('different', 0.6246160268783569),\n",
       " ('their_own', 0.6177682876586914),\n",
       " ('messy', 0.6109225153923035),\n",
       " ('pricey', 0.6010245680809021),\n",
       " ('cheaper', 0.5943189859390259),\n",
       " ('empty', 0.592384397983551),\n",
       " ('diff', 0.5899870991706848)]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Words similar to the equation : Embed(bigger) + Embed(small) - Embed('big)\n",
    "\n",
    "dmm_bg_model.most_similar(positive=['bigger', 'small'], negative=['big'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize Bigram train, validation sets\n",
    "\n",
    "train_vecs_dmm_bi = vectorize(dmm_bg_model, x_train, 100)\n",
    "val_vecs_dmm_bi = vectorize(dmm_bg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Memory Bigram\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dmm_bi, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7385"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dmm_bi, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dmm_bg_model\n",
    "\n",
    "dmm_bg_model.save('./data/dmm_bg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dmm_bg_model and delete temporary training data\n",
    "\n",
    "dmm_bg_model = Doc2Vec.load('./data/dmm_bg_model.doc2vec')\n",
    "dmm_bg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pretty Nice.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display('Pretty Nice.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on concatanated document vectors obtained from Bigram Distributed Bag Of Words & Distributed Memory Concatanated models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Updated vectorize_concate function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets using Bigram Model\n",
    "\n",
    "def vectorize_concate(model1, model2, corpus, size):\n",
    "    # Numpy zeros initialization\n",
    "    vectors = np.zeros((len(corpus), size))\n",
    "    \n",
    "    for idx, count in zip(corpus.index, range(len(corpus.index))):\n",
    "        prefix = 'all_' + str(idx)\n",
    "        # Appending document vectors\n",
    "        vectors[count] = np.append(model1.docvecs[prefix], model2.docvecs[prefix])\n",
    "\n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "train_vecs_dbow_dm_bi = vectorize_concate(dbow_bg_model, dm_bg_model, x_train, 200)\n",
    "val_vecs_dbow_dm_bi = vectorize_concate(dbow_bg_model, dm_bg_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dm_bi, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7499375"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dm_bi, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on concatanated document vectors obtained from Bigram Distributed Bag Of Words & Distributed Memory Mean models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Bigram Distributed Bag Of Words & Distributed Memory Mean\n",
    "\n",
    "train_vecs_dbow_dmm_bi = vectorize_concate(dbow_bg_model, dmm_bg_model, x_train, 200)\n",
    "val_vecs_dbow_dmm_bi = vectorize_concate(dbow_bg_model, dmm_bg_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Bigram Distributed Bag Of Words & Distributed Memory Mean\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dmm_bi, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7584375"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dmm_bi, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will run the same phrase detection on trigram phrases (generated from the corpus) to generate trigram phrases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 3s, sys: 234 ms, total: 2min 4s\n",
      "Wall time: 2min 4s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Getting Bigram Phrases from the tokens & thereafter triagram phrases from the Bigram phrases\n",
    "\n",
    "tg_phrases = Phrases(bigram[tokenized_train])\n",
    "trigram = Phraser(tg_phrases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['last',\n",
       " 'cream',\n",
       " 'time',\n",
       " 'ice',\n",
       " 'with',\n",
       " 'nutella',\n",
       " 'and',\n",
       " 'vanilla',\n",
       " 'sadface']"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "check = [u'last', u'cream', u'time', u'ice', u'with', u'nutella', u'and', u'vanilla', u'sadface']\n",
    "display(bigram[check])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['last',\n",
       " 'cream',\n",
       " 'time',\n",
       " 'ice',\n",
       " 'with',\n",
       " 'nutella',\n",
       " 'and',\n",
       " 'vanilla',\n",
       " 'sadface']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example\n",
    "\n",
    "check = [u'last', u'cream', u'time', u'ice', u'with', u'nutella', u'and', u'vanilla', u'sadface']\n",
    "trigram[bigram[check]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see that the trigram can find out the most frequently used phrase in the example \"vanilla_ice_cream\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labelling tweets using genism phrase library for unsupervised learning\n",
    "\n",
    "def label_tweets_trigram(tweets, label):\n",
    "    result = []\n",
    "    prefix = label\n",
    "    \n",
    "    # Split tweets & attach label with index\n",
    "    for index, tweet in zip(tweets.index, tweets):\n",
    "        result.append(LabeledSentence(trigram[tweet.split()], [prefix + '_%s' % index]))\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/ipykernel_launcher.py:9: DeprecationWarning: Call to deprecated `LabeledSentence` (Class will be removed in 4.0.0, use TaggedDocument instead).\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1600000"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vec_train_tg = label_tweets_trigram(df.text , 'all')\n",
    "len(word_vec_train_tg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All cores of CPU\n",
    "\n",
    "cores = multiprocessing.cpu_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DBOW (Document Bag Of Words) Trigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW Trigram: This is the Doc2Vec model analogous to Skip-gram model in Word2Vec. The paragraph vectors are obtained by training a neural network on the task of predicting a probability distribution of words in a paragraph given a randomly-sampled word from the paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4302349.44it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Bag Of Words Trigram parameters & building word vocabulary\n",
    "\n",
    "dbow_tg_model = Doc2Vec(dm=0, vector_size=100, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dbow_tg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_tg)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One caveat of the way this algorithm is that, since the learning rate decrease over the course of iterating over the data, labels which are only seen in a single LabeledSentence during training will only be trained with a fixed learning rate. This frequently produces less than optimal results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below iteration implement explicit multiple-pass, alpha-reduction approach with added shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3955698.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3838712.10it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4079791.89it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4083977.89it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4118222.41it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3981169.60it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4061537.42it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4284777.07it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4226352.96it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4121130.75it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4235857.76it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4305728.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4096767.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4182775.23it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4168601.96it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4191974.83it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4202959.47it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4219664.18it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4163948.81it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4140291.31it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3972767.58it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3890743.34it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4129069.86it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4185172.48it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4122186.35it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4154272.39it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4268175.45it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4176306.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4223706.28it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4112594.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27min 25s, sys: 2min 3s, total: 29min 28s\n",
      "Wall time: 18min 34s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dbow_tg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_tg)]), total_examples=len(word_vec_train_tg), epochs=1)\n",
    "    dbow_tg_model.alpha -= 0.002\n",
    "    dbow_tg_model.min_alpha = dbow_bg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets using above dbow_tg_model (Document Bag Of Words Trigram Model)\n",
    "\n",
    "def vectorize(model, corpus, size):\n",
    "    # Numpy zeros initialization\n",
    "    vectors = np.zeros((len(corpus), size))\n",
    "    \n",
    "    for idx, count in zip(corpus.index, range(len(corpus.index))):\n",
    "        prefix = 'all_' + str(idx)\n",
    "        vectors[count] = model.docvecs[prefix]\n",
    "\n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dbow_tg = vectorize(dbow_tg_model, x_train, 100)\n",
    "val_vecs_dbow_tg = vectorize(dbow_tg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7403125"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_tg, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBOW model doesn't learn the semantic understanding of words but it's features obtained from it does a decent job with a simple Logistic Regression classifier.\n",
    "The result doesn't seem to excel count vectorizer or Tfidf vectorizer. It might even not be a direct comparison as count vectorizer or tfidf vectorizer uses a large number of features to represent a tweet rather than using 200 dimensions as in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dbow_tg_model\n",
    "\n",
    "dbow_tg_model.save('./data/dbow_tg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dbow_tg_model and delete temporary training data\n",
    "\n",
    "dbow_tg_model = Doc2Vec.load('./data/dbow_tg_model.doc2vec')\n",
    "dbow_tg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed Memory Trigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Concatenated Trigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3968097.63it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory Trigram parameters & building word vocabulary\n",
    "\n",
    "dm_tg_model = Doc2Vec(dm=1, dm_concat=1, vector_size=100, window=2, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dm_tg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_tg)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3755285.33it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3890833.57it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3938480.03it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3939654.58it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3831453.39it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3861249.51it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3899821.19it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3610144.30it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3982093.27it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4006465.86it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3968857.98it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3796121.68it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3859770.46it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3947928.58it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3787305.91it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3852633.22it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4232876.08it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3960421.32it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3844035.49it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3871147.84it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4051635.78it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3776755.37it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4063731.24it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3959365.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3951715.59it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4174888.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3999510.35it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3999715.35it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3725064.60it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3816525.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 39min 32s, sys: 2min 41s, total: 42min 13s\n",
      "Wall time: 22min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dm_tg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_tg)]), total_examples=len(word_vec_train_tg), epochs=1)\n",
    "    dm_tg_model.alpha -= 0.002\n",
    "    dm_tg_model.min_alpha = dm_tg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_tg_model\n",
    "\n",
    "dm_tg_model.save('./data/dm_tg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dm_tg = vectorize(dm_tg_model, x_train, 100)\n",
    "val_vecs_dm_tg = vectorize(dm_tg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.7 s, sys: 2.9 s, total: 13.6 s\n",
      "Wall time: 2min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Train a Logistic Regression model on Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dm_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6563125"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dm_tg, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dm_tg_model\n",
    "\n",
    "dm_tg_model.save('./data/dm_tg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dm_tg_model and delete temporary training data\n",
    "\n",
    "dm_tg_model = Doc2Vec.load('./data/dm_tg_model.doc2vec')\n",
    "dm_tg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Distributed Memory Trigram Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DM is the Doc2Vec model analogous to CBOW model in Word2vec. The paragraph vectors are obtained by training a neural network on the task of inferring a centre word based on context words and a context paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3917843.80it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initializing Distributed Memory Trigram Mean parameters & building word vocabulary\n",
    "\n",
    "dmm_tg_model = Doc2Vec(dm=1, dm_mean=1, vector_size=100, window=4, negative=5, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "dmm_tg_model.build_vocab([w_v for w_v in tqdm(word_vec_train_tg)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4065400.32it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4177864.10it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4213694.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4218218.66it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4231530.89it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4198137.42it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4102530.40it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4194372.16it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4165403.91it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4220107.32it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4027964.18it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4071309.73it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4190985.26it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4198444.72it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4301635.17it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4152280.33it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3983249.06it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4114485.55it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4096927.71it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3981623.11it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4116661.20it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4162388.88it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4151484.04it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4164057.33it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 3945625.99it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4153431.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4171511.90it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4217436.64it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4229786.62it/s]\n",
      "100%|██████████| 1600000/1600000 [00:00<00:00, 4015061.61it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 48min 34s, sys: 9min 50s, total: 58min 25s\n",
      "Wall time: 34min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Multiple epochs iterating over labels more than once with decreasing learning rate\n",
    "\n",
    "for epoch in range(30):\n",
    "    \n",
    "    # Shuffling word_vec_train & reducing aplha over multiple passes\n",
    "    dmm_tg_model.train(utils.shuffle([w_v for w_v in tqdm(word_vec_train_tg)]), total_examples=len(word_vec_train_tg), epochs=1)\n",
    "    dmm_tg_model.alpha -= 0.002\n",
    "    dmm_tg_model.min_alpha = dmm_tg_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dmm_tg_model\n",
    "\n",
    "dmm_tg_model.save('./data/dmm_tg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize train, validation sets\n",
    "\n",
    "train_vecs_dmm_tg = vectorize(dmm_tg_model, x_train, 100)\n",
    "val_vecs_dmm_tg = vectorize(dmm_tg_model, x_val, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dmm_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73325"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dmm_tg, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the dmm_tg_model\n",
    "\n",
    "dmm_tg_model.save('./data/dmm_tg_model.doc2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dmm_ug_model and delete temporary training data\n",
    "\n",
    "dmm_tg_model = Doc2Vec.load('./data/dmm_tg_model.doc2vec')\n",
    "dmm_tg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pretty Nice.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display('Pretty Nice.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on concatanated document vectors obtained from Distributed Bag Of Words & Distributed Memory Concatanated models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "train_vecs_dbow_dm_tg = vectorize_concate(dbow_tg_model, dm_tg_model, x_train, 200)\n",
    "val_vecs_dbow_dm_tg = vectorize_concate(dbow_tg_model, dm_tg_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dm_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7496875"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dm_tg, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on concatanated document vectors obtained from Distributed Bag Of Words & Distributed Memory Mean models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "train_vecs_dbow_dmm_tg = vectorize_concate(dbow_tg_model, dmm_tg_model, x_train, 200)\n",
    "val_vecs_dbow_dmm_tg = vectorize_concate(dbow_tg_model, dmm_tg_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_dmm_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75875"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_dmm_tg, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Populate table with Models & it's Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "mydata = [['Distributed Bag Of Words', 0.7320625, 0.668125, 0.7258125],\n",
    "          ['Distributed Memory Concatanated', 0.742875, 0.6571875, 0.7385],\n",
    "          ['Distributed Memory Mean', 0.7403125, 0.6563125, 0.73325],\n",
    "          ['Distributed Memory BoW & Concatanated', 0.742375, 0.7499375, 0.7496875],\n",
    "          ['Distributed Memory BoW & Mean', 0.7504375, 0.7584375, 0.75875]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>                                     </th><th style=\"text-align: right;\">  Unigram</th><th style=\"text-align: right;\">  Bigram</th><th style=\"text-align: right;\">  Trigram</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>Distributed Bag Of Words             </td><td style=\"text-align: right;\">   0.7321</td><td style=\"text-align: right;\">  0.6681</td><td style=\"text-align: right;\">   0.7258</td></tr>\n",
       "<tr><td>Distributed Memory Concatanated      </td><td style=\"text-align: right;\">   0.7429</td><td style=\"text-align: right;\">  0.6572</td><td style=\"text-align: right;\">   0.7385</td></tr>\n",
       "<tr><td>Distributed Memory Mean              </td><td style=\"text-align: right;\">   0.7403</td><td style=\"text-align: right;\">  0.6563</td><td style=\"text-align: right;\">   0.7332</td></tr>\n",
       "<tr><td>Distributed Memory BoW & Concatanated</td><td style=\"text-align: right;\">   0.7424</td><td style=\"text-align: right;\">  0.7499</td><td style=\"text-align: right;\">   0.7497</td></tr>\n",
       "<tr><td>Distributed Memory BoW & Mean        </td><td style=\"text-align: right;\">   0.7504</td><td style=\"text-align: right;\">  0.7584</td><td style=\"text-align: right;\">   0.7588</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tabulate import tabulate\n",
    "from IPython.display import HTML\n",
    "\n",
    "display(HTML(tabulate(mydata, headers= [ '', 'Unigram', 'Bigram', 'Trigram'], floatfmt='.4f', tablefmt='html')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Joint Operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using best of above n-gram models, we will train on joint vectors (Distributed Bag Of Words Unigram + Distributed Memory BoW & Mean\tTriagram) having best performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dbow_ug_model and delete temporary training data\n",
    "\n",
    "dbow_ug_model = Doc2Vec.load('./data/dbow_ug_model.doc2vec')\n",
    "dbow_ug_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dmm_tg_model and delete temporary training data\n",
    "\n",
    "dmm_tg_model = Doc2Vec.load('./data/dmm_tg_model.doc2vec')\n",
    "dmm_tg_model.delete_temporary_training_data(keep_doctags_vectors=True, keep_inference=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vectorize & concate document vectors of train, validation sets obtained from Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "train_vecs_dbow_ug_dmm_tg = vectorize_concate(dbow_ug_model, dmm_tg_model, x_train, 200)\n",
    "val_vecs_dbow_ug_dmm_tg = vectorize_concate(dbow_ug_model, dmm_tg_model, x_val, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train a Logistic Regression model on Distributed Bag Of Words & Distributed Memory \n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(train_vecs_dbow_ug_dmm_tg, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.754"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(val_vecs_dbow_ug_dmm_tg, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithms Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic imports\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB, BernoulliNB\n",
    "from sklearn.linear_model import RidgeClassifier\n",
    "from sklearn.linear_model import PassiveAggressiveClassifier\n",
    "from sklearn.linear_model import Perceptron\n",
    "from sklearn.neighbors import NearestCentroid\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "from datetime import datetime\n",
    "from collections import Counter\n",
    "\n",
    "# Classifiers names list\n",
    "names = [\"Logistic Regression\", \"Multinomial NB\", \"Bernoulli NB\", \"Ridge Classifier\", \n",
    "         \"AdaBoost\", \"Perceptron\", \"Passive-Aggresive\", \"Nearest Centroid\"]\n",
    "\n",
    "classifiers = [\n",
    "    LogisticRegression(),\n",
    "    MultinomialNB(),\n",
    "    BernoulliNB(),\n",
    "    RidgeClassifier(),\n",
    "    AdaBoostClassifier(),\n",
    "    Perceptron(),\n",
    "    PassiveAggressiveClassifier(),\n",
    "    NearestCentroid()\n",
    "    ]\n",
    "\n",
    "# Zipping all of them together\n",
    "zipped_clf = zip(names,classifiers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scaling inputs\n",
    "\n",
    "mmscaler = MinMaxScaler()\n",
    "\n",
    "train_vecs_dbow_ug_dmm_tg_scaled = mmscaler.fit_transform(train_vecs_dbow_ug_dmm_tg)\n",
    "val_vecs_dbow_ug_dmm_tg_scaled = mmscaler.fit_transform(val_vecs_dbow_ug_dmm_tg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate accuracy & summmary of different set of features\n",
    "\n",
    "def accuracy_features(pipeline, x_train, y_train, x_test, y_test):\n",
    "    \n",
    "    counter = Counter(y_test)\n",
    "\n",
    "    if (counter[0] / (len(y_test)*1.)) > 0.5:\n",
    "        baseline_accuracy = counter[0] / (len(y_test)*1.)\n",
    "    else:\n",
    "        baseline_accuracy = 1. - (counter[0] / (len(y_test)*1.))\n",
    "   \n",
    "    # Timer starts\n",
    "    timer = datetime.now()\n",
    "    \n",
    "    model = pipeline.fit(x_train, y_train)\n",
    "    y_pred = model.predict(x_test)\n",
    "    \n",
    "    elapsed_time = datetime.now() - timer\n",
    "    # Timer stops\n",
    "\n",
    "    accuracy = accuracy_score(y_test, y_pred)\n",
    "    \n",
    "    \n",
    "    print('Baseline accuracy: {:.2f}%'.format(baseline_accuracy*100))\n",
    "    print('Accuracy score: {:.2f}%'.format(accuracy*100))\n",
    "    \n",
    "    if(accuracy > baseline_accuracy):\n",
    "        print('\\nModel accuracy:{:.2f}% - Baseline accuracy:{:.2f}%: Increase of {:.2f}%'.format(accuracy*100, baseline_accuracy*100, (accuracy-baseline_accuracy)*100))\n",
    "    else:\n",
    "        print('Model accuracy:{:.2f}% - Baseline accuracy:{:.2f}%: Decrease of {:.2f}%'.format(accuracy*100, baseline_accuracy*100, (accuracy-baseline_accuracy)*100))\n",
    "    \n",
    "    print('Overall Train and Prediction time: {:.2f}s'.format(elapsed_time.total_seconds()))\n",
    "    print('-'*89)\n",
    "          \n",
    "    return accuracy, elapsed_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Comparing different classifiers using pipeline\n",
    "\n",
    "def classifier_comparator(train, val, classifier=zipped_clf):\n",
    "    result = []\n",
    "    \n",
    "    for clf_name, clf in zipped_clf:\n",
    "        pipeline = Pipeline([\n",
    "            ('classifier', clf)\n",
    "        ])\n",
    "        \n",
    "        print(\"\\nValidation result for {} classifier\".format(clf_name))\n",
    "        print(clf)\n",
    "        \n",
    "        # Calculate accuracy & summmary\n",
    "        clf_accuracy, clf_time = accuracy_features(pipeline, train, y_train, val, y_val)\n",
    "        result.append((clf_name, clf_accuracy, clf_time))\n",
    "        \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for joint vectors operation \n",
      "(Distributed Bag Of Words Unigram + Distributed Memory BoW & Mean Triagram)\n",
      "Running Different Classifiers now .......................\n",
      "\n",
      "\n",
      "Validation result for Logistic Regression classifier\n",
      "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 75.43%\n",
      "\n",
      "Model accuracy:75.43% - Baseline accuracy:50.01%: Increase of 25.42%\n",
      "Overall Train and Prediction time: 70.29s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Multinomial NB classifier\n",
      "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 73.88%\n",
      "\n",
      "Model accuracy:73.88% - Baseline accuracy:50.01%: Increase of 23.87%\n",
      "Overall Train and Prediction time: 1.70s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Bernoulli NB classifier\n",
      "BernoulliNB(alpha=1.0, binarize=0.0, class_prior=None, fit_prior=True)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 49.98%\n",
      "Model accuracy:49.98% - Baseline accuracy:50.01%: Decrease of -0.03%\n",
      "Overall Train and Prediction time: 3.90s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Ridge Classifier classifier\n",
      "RidgeClassifier(alpha=1.0, class_weight=None, copy_X=True, fit_intercept=True,\n",
      "        max_iter=None, normalize=False, random_state=None, solver='auto',\n",
      "        tol=0.001)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 75.26%\n",
      "\n",
      "Model accuracy:75.26% - Baseline accuracy:50.01%: Increase of 25.25%\n",
      "Overall Train and Prediction time: 9.90s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for AdaBoost classifier\n",
      "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
      "          learning_rate=1.0, n_estimators=50, random_state=None)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 70.29%\n",
      "\n",
      "Model accuracy:70.29% - Baseline accuracy:50.01%: Increase of 20.27%\n",
      "Overall Train and Prediction time: 3208.33s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Perceptron classifier\n",
      "Perceptron(alpha=0.0001, class_weight=None, eta0=1.0, fit_intercept=True,\n",
      "      max_iter=None, n_iter=None, n_jobs=1, penalty=None, random_state=0,\n",
      "      shuffle=True, tol=None, verbose=0, warm_start=False)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/sklearn/linear_model/stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been added in <class 'sklearn.linear_model.perceptron.Perceptron'> in 0.19. If both are left unset, they default to max_iter=5 and tol=None. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter will be 1000, and default tol will be 1e-3.\n",
      "  \"and default tol will be 1e-3.\" % type(self), FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 71.85%\n",
      "\n",
      "Model accuracy:71.85% - Baseline accuracy:50.01%: Increase of 21.84%\n",
      "Overall Train and Prediction time: 5.02s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Passive-Aggresive classifier\n",
      "PassiveAggressiveClassifier(C=1.0, average=False, class_weight=None,\n",
      "              fit_intercept=True, loss='hinge', max_iter=None, n_iter=None,\n",
      "              n_jobs=1, random_state=None, shuffle=True, tol=None,\n",
      "              verbose=0, warm_start=False)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/sklearn/linear_model/stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been added in <class 'sklearn.linear_model.passive_aggressive.PassiveAggressiveClassifier'> in 0.19. If both are left unset, they default to max_iter=5 and tol=None. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter will be 1000, and default tol will be 1e-3.\n",
      "  \"and default tol will be 1e-3.\" % type(self), FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 67.92%\n",
      "\n",
      "Model accuracy:67.92% - Baseline accuracy:50.01%: Increase of 17.91%\n",
      "Overall Train and Prediction time: 6.60s\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Validation result for Nearest Centroid classifier\n",
      "NearestCentroid(metric='euclidean', shrink_threshold=None)\n",
      "Baseline accuracy: 50.01%\n",
      "Accuracy score: 73.87%\n",
      "\n",
      "Model accuracy:73.87% - Baseline accuracy:50.01%: Increase of 23.86%\n",
      "Overall Train and Prediction time: 2.36s\n",
      "-----------------------------------------------------------------------------------------\n",
      "CPU times: user 55min 5s, sys: 10.7 s, total: 55min 16s\n",
      "Wall time: 55min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "print('Result for joint vectors operation \\n(Distributed Bag Of Words Unigram + Distributed Memory BoW & Mean Triagram)\\nRunning Different Classifiers now .......................\\n')\n",
    "classifier_comparator(train_vecs_dbow_ug_dmm_tg_scaled, val_vecs_dbow_ug_dmm_tg_scaled, zipped_clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}