ner_BiLSTM_CRF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bqbS2gLMhNog"
   },
   "source": [
    "# Named Entity Recognition BiLSTM-CRF\n",
    "\n",
    "Named Enity Recognition is one of the most common NLP problems. The goal is classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. What can you use it for? Here are a few ideas - social media, chatbot, customer support tickets, survey responses, and data mining!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MObExv0FhXmd"
   },
   "source": [
    "Let's start by importing some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oNUyGjRQCZ3o"
   },
   "outputs": [],
   "source": [
    "files_path = \"./\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tG83Y0AhC05p"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "import tensorflow as tf\n",
    "\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DB-FhT-hEefx"
   },
   "source": [
    "# Training Parameters\n",
    "We'll set the hyperparameters for training our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uZlKveosDFtS"
   },
   "outputs": [],
   "source": [
    "# Hyperparams if GPU is available\n",
    "if len(tf.config.list_physical_devices('GPU')) > 0:\n",
    "    BATCH_SIZE = 512  # Number of examples used in each iteration\n",
    "    EPOCHS = 5  # Number of passes through entire dataset\n",
    "    MAX_LEN = 75  # Max length of review (in words)\n",
    "    EMBEDDING = 40  # Dimension of word embedding vector\n",
    "\n",
    "    \n",
    "# Hyperparams for CPU training\n",
    "else:\n",
    "    BATCH_SIZE = 32\n",
    "    EPOCHS = 5\n",
    "    MAX_LEN = 75\n",
    "    EMBEDDING = 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8d7tHjXbE6G4"
   },
   "source": [
    "# Data\n",
    "\n",
    "Let's take a look at data. [Annotated Corpus for Named Entity Recognition](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus?select=ner_dataset.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 427
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5542,
     "status": "ok",
     "timestamp": 1595428692287,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "1S3bXmbXEm5g",
    "outputId": "6926ea83-d218-4ce0-86df-da23c66444cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences:  47959\n",
      "Number of words in the dataset:  35178\n",
      "Tags: ['I-org', 'B-tim', 'I-art', 'I-per', 'B-org', 'B-geo', 'B-art', 'B-nat', 'I-geo', 'B-eve', 'I-tim', 'I-gpe', 'B-per', 'I-eve', 'I-nat', 'B-gpe', 'O']\n",
      "Number of Labels:  17\n",
      "What the dataset looks like:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentence #</th>\n",
       "      <th>Word</th>\n",
       "      <th>POS</th>\n",
       "      <th>Tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>Thousands</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>demonstrators</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>have</td>\n",
       "      <td>VBP</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>marched</td>\n",
       "      <td>VBN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>through</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>London</td>\n",
       "      <td>NNP</td>\n",
       "      <td>B-geo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>to</td>\n",
       "      <td>TO</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>protest</td>\n",
       "      <td>VB</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Sentence #           Word  POS    Tag\n",
       "0  Sentence: 1      Thousands  NNS      O\n",
       "1  Sentence: 1             of   IN      O\n",
       "2  Sentence: 1  demonstrators  NNS      O\n",
       "3  Sentence: 1           have  VBP      O\n",
       "4  Sentence: 1        marched  VBN      O\n",
       "5  Sentence: 1        through   IN      O\n",
       "6  Sentence: 1         London  NNP  B-geo\n",
       "7  Sentence: 1             to   TO      O\n",
       "8  Sentence: 1        protest   VB      O\n",
       "9  Sentence: 1            the   DT      O"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(f\"{files_path}ner_dataset.csv\", encoding=\"latin1\")\n",
    "data = data.fillna(method=\"ffill\")\n",
    "\n",
    "print(\"Number of sentences: \", len(data.groupby(['Sentence #'])))\n",
    "\n",
    "words = list(set(data[\"Word\"].values))\n",
    "n_words = len(words)\n",
    "print(\"Number of words in the dataset: \", n_words)\n",
    "\n",
    "tags = list(set(data[\"Tag\"].values))\n",
    "print(\"Tags:\", tags)\n",
    "n_tags = len(tags)\n",
    "print(\"Number of Labels: \", n_tags)\n",
    "\n",
    "print(\"What the dataset looks like:\")\n",
    "# Show the first 10 rows\n",
    "data.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12880,
     "status": "ok",
     "timestamp": 1595428699840,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "HrIVsOi2Eu7o",
    "outputId": "b90f44b4-dbd2-48f7-855b-3f1af666173c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is what a sentence looks like:\n",
      "[('Thousands', 'NNS', 'O'), ('of', 'IN', 'O'), ('demonstrators', 'NNS', 'O'), ('have', 'VBP', 'O'), ('marched', 'VBN', 'O'), ('through', 'IN', 'O'), ('London', 'NNP', 'B-geo'), ('to', 'TO', 'O'), ('protest', 'VB', 'O'), ('the', 'DT', 'O'), ('war', 'NN', 'O'), ('in', 'IN', 'O'), ('Iraq', 'NNP', 'B-geo'), ('and', 'CC', 'O'), ('demand', 'VB', 'O'), ('the', 'DT', 'O'), ('withdrawal', 'NN', 'O'), ('of', 'IN', 'O'), ('British', 'JJ', 'B-gpe'), ('troops', 'NNS', 'O'), ('from', 'IN', 'O'), ('that', 'DT', 'O'), ('country', 'NN', 'O'), ('.', '.', 'O')]\n"
     ]
    }
   ],
   "source": [
    "class SentenceGetter(object):\n",
    "    \"\"\"Class to Get the sentence in this format:\n",
    "    [(Token_1, Part_of_Speech_1, Tag_1), ..., (Token_n, Part_of_Speech_1, Tag_1)]\"\"\"\n",
    "    def __init__(self, data):\n",
    "        \"\"\"Args:\n",
    "            data is the pandas.DataFrame which contains the above dataset\"\"\"\n",
    "        self.n_sent = 1\n",
    "        self.data = data\n",
    "        self.empty = False\n",
    "        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s[\"Word\"].values.tolist(),\n",
    "                                                           s[\"POS\"].values.tolist(),\n",
    "                                                           s[\"Tag\"].values.tolist())]\n",
    "        self.grouped = self.data.groupby(\"Sentence #\").apply(agg_func)\n",
    "        self.sentences = [s for s in self.grouped]\n",
    "    \n",
    "    def get_next(self):\n",
    "        \"\"\"Return one sentence\"\"\"\n",
    "        try:\n",
    "            s = self.grouped[\"Sentence: {}\".format(self.n_sent)]\n",
    "            self.n_sent += 1\n",
    "            return s\n",
    "        except:\n",
    "            return None\n",
    "        \n",
    "getter = SentenceGetter(data)\n",
    "sent = getter.get_next()\n",
    "print('This is what a sentence looks like:')\n",
    "print(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12644,
     "status": "ok",
     "timestamp": 1595428699842,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "k4VggbtJEvLJ",
    "outputId": "7eac7b3d-b977-4b11-f9a6-339cb978e312"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcy0lEQVR4nO3debQmVXnv8e/PBhlEaYaWkIbYIEQv3EQgHcTIjQS4jCrGOOA12hK8naxlHJMomgGjcoNDRIhTUFAkiiIaaYeABMFoiECDiAyy7EATmiA0MojiBDz3j9pH3z726Xobz3vG72etd52qXbuqnjrV/T5n167alapCkqQNecR0ByBJmvlMFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplstCsleSAJGumOw5pPjBZaEZI8v2Bz0NJfjgw/8Lpjm+2SfLGJP803XFo7thkugOQAKpqq7HpJKuBl1bVv05fRJMvSYBU1UPTHYu0sWxZaEZLslmSdyX57/Z5V5LNJqj7iiTXJdmprfeOJP+V5PYk70+yRat3QJI1Sf4syR1JbktyzAZiuDjJ3yW5LMn3kpybZNuB5fsluSTJPUm+keSAceuekOTfgfuBXdez/dcluTXJfUluSHJQK39EkuOS/GeS7yY5e2y/SZYkqSTL2jHemeQv27LDgDcAz28ts2+08q2TnNaO99Ykb0myoC17SZKvtt/Z3UluSnL4QIzbJvlQOwd3J/nMwLKnJ7mqHf8lSX5ziFOrWcZkoZnuL4H9gL2AJwH7An81vlKSvwFeAjytqtYAJwK/3tbbDVgM/M3AKr8CbN3KjwXek2SbDcTxYuCPgB2BB4BT2n4XA58H3gJsC/w58KkkiwbWfRGwHHg0cPO4uJ8A/Cnw21X1aOBQYHVb/HLgWcDTgF8F7gbeMy6u/YEnAAcBf5Pkf1TVecD/Az5RVVtV1ZNa3Q+32HcD9gYOAV46sK0nAzcA2wNvA05rrSGAM4EtgT2BxwIntfj3Bk4H/hjYDvhHYMVECV2zWFX58TOjPnRflge36f8EjhhYdiiwuk0fANwKvBP4KrB1Kw/wA+DxA+s9BbhpYL0fApsMLL8D2G+CeC4GThyY3wP4CbAAeB1w5rj65wPLBtZ90waOdbe274OBTcctux44aGB+R+CndJePlwAF7DSw/DLg6Db9RuCfBpbtAPwY2GKg7AXARW36JcCqgWVbtu3/StvvQ8A264n/fcCbx5XdQJe0p/3fkp/J+9hnoZnuV1n3r/GbW9mYhXR/tT+/qu5tZYvovuyu+PkfxoTuy33Md6vqgYH5+4GtmNgt42LYlO4v8McBz03yjIHlmwIXTbDuOqpqVZJX0X2575nkfOA1VfXfbdv/nGSwj+NBui/+Md8Z8hge1+K6beB38ohxsf1sW1V1f6u3FV2L6a6qunuC7S5L8vKBskey7jnSHOBlKM10Y1+aY36tlY25G3g68KEkT21ld9K1HPasqoXts3UNdKI/DDuPi+GnbT+30LUsFg58HlVVJw7U3+DQzlX1saran+44C3hrW3QLcPi4bW9eVbcOEe/4fd5C17LYfmBbj6mqPYfY1i3AtkkWTrDshHExbllVZw2xXc0iJgvNdGcBf5VkUZLt6fod1rkltKouBl4IfDrJvtXdbfQB4KQkj4WubyHJob9EHH+YZI8kWwJvAs6pqgdbLM9IcmiSBUk2bx3oOw2z0SRPSHJgu8b/I7okN9aSeD9wQpLHtbqLkhw1ZLy3A0uSPAKgqm4Dvgj8fZLHtM7zxyd5Wt+G2rr/Arw3yTZJNk3yu23xB4A/SfLkdB6V5Mgkjx4yTs0SJgvNdG8BVgJXA98Ermxl66iqC+g6oD+bZB+6voRVwNeSfA/4V7qO4IfrTLoO4u8AmwOvaPu9BTiK7u6jtXR/af8Fw//f2oyuM/7Otu3HAq9vy04GVgBfTHIf8DW6TuhhfLL9/G6SK9v0i+kuEV1H1yI7h64/YhgvomtNfYuuj+VVAFW1Evi/wLvbNlfR9X9ojkmVLz+SNiTJxXSdxR+c7lik6WLLQpLUy2QhSerlZShJUi9bFpKkXnPyobztt9++lixZMt1hSNKscsUVV9xZVYvWt2ykySLd6KH30T11+kBVLW0DoX2CbriC1cDzquruNgbNycARdE+ivqSqrmzbWcbPxwN6S1WdsaH9LlmyhJUrV07+AUnSHJbk5omWTcVlqN+rqr2qammbPw64sKp2By5s8wCHA7u3z3K6MWdoyeV4uvvL9wWO7xnwTZI0yaajz+IoYKxlcAbdqJpj5R+pzteAhUl2pBs47oKqGhub5gLgsKkOWpLms1Eni6J7+vSKJMtb2Q5t+ADonlgdGxRtMesOaramlU1Uvo4ky5OsTLJy7dq1k3kMkjTvjbqDe/+qurWNz3NBkm8NLqyqSjIp9+5W1anAqQBLly71fmBJmkQjbVmMjY5ZVXcA/0zX53B7u7xE+3lHq34r647suVMrm6hckjRFRpYs2uiTjx6bpnsr1zV0A6Mta9WWAee26RXAi9vIlfsB97bLVecDh7TRLrdp2zl/VHFLkn7RKC9D7UD34pax/Xysqs5LcjlwdpJj6V4i87xW/wt0t82uort19hiAqroryZuBy1u9N1XVXSOMW5I0zpwc7mPp0qXlcxaStHGSXDHwmMM6HO5DktRrTg73ofVbctzn11u++sQjpzgSSbONLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m4oTXiXFHinlKSOLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m6oOWhDdzdJ0sNhy0KS1MtkIUnqZbKQJPUyWUiSetnBrQ3yhUmSwJaFJGkIJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqNPFkkWZDk60k+1+Z3SXJpklVJPpHkka18sza/qi1fMrCN17fyG5IcOuqYJUnrmoqWxSuB6wfm3wqcVFW7AXcDx7byY4G7W/lJrR5J9gCOBvYEDgPem2TBFMQtSWpGmiyS7AQcCXywzQc4EDinVTkDeFabPqrN05Yf1OofBXy8qn5cVTcBq4B9Rxm3JGldo25ZvAt4LfBQm98OuKeqHmjza4DFbXoxcAtAW35vq/+z8vWs8zNJlidZmWTl2rVrJ/s4JGleG9k7uJM8Hbijqq5IcsCo9jOmqk4FTgVYunRpjXp/M8FE78eWpMk2smQBPBV4ZpIjgM2BxwAnAwuTbNJaDzsBt7b6twI7A2uSbAJsDXx3oHzM4DqSpCkwsstQVfX6qtqpqpbQdVB/qapeCFwEPKdVWwac26ZXtHna8i9VVbXyo9vdUrsAuwOXjSpuSdIvGmXLYiKvAz6e5C3A14HTWvlpwJlJVgF30SUYquraJGcD1wEPAC+rqgenPmxJmr+mJFlU1cXAxW36RtZzN1NV/Qh47gTrnwCcMLoIJUkb4hPckqReJgtJUq/p6LPQHDDRbburTzxyiiORNBVsWUiSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSLwcSnAV817ak6WbLQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVKv3mSR5PFJNmvTByR5RZKFow9NkjRTDPPyo08BS5PsBpwKnAt8DDhilIFpdproRU2rTzxyiiORNJmGuQz1UFU9APw+8A9V9RfAjqMNS5I0kwyTLH6a5AXAMuBzrWzTvpWSbJ7ksiTfSHJtkr9t5bskuTTJqiSfSPLIVr5Zm1/Vli8Z2NbrW/kNSQ7d2IOUJP1yhkkWxwBPAU6oqpuS7AKcOcR6PwYOrKonAXsBhyXZD3grcFJV7QbcDRzb6h8L3N3KT2r1SLIHcDSwJ3AY8N4kC4Y9QEnSL683WVTVdcDrgCvb/E1V9dYh1quq+n6b3bR9CjgQOKeVnwE8q00f1eZpyw9Kklb+8ar6cVXdBKwC9h3i2CRJk2SYu6GeAVwFnNfm90qyYpiNJ1mQ5CrgDuAC4D+Be1ofCMAaYHGbXgzcAtCW3wtsN1i+nnUG97U8ycokK9euXTtMeJKkIQ1zGeqNdH/J3wNQVVcBuw6z8ap6sKr2AnZq23jiwwtzqH2dWlVLq2rpokWLRrUbSZqXhurgrqp7x5U9tDE7qap7gIvo+j4WJhm7ZXcn4NY2fSuwM0BbvjXw3cHy9awjSZoCwySLa5P8H2BBkt2T/ANwSd9KSRaNPbyXZAvgfwPX0yWN57Rqy+ie2wBY0eZpy79UVdXKj253S+0C7A5cNtTRSZImxTDJ4uV0dyL9GDgL+B7wqiHW2xG4KMnVwOXABVX1ObrO8tckWUXXJ3Faq38asF0rfw1wHEBVXQucDVxH12/ysqp6cLjDkyRNht4nuKvqfuAv22doVXU1sPd6ym9kPXczVdWPgOdOsK0TgBM2Zv+SpMkzYbJI8lm6W13Xq6qeOZKIJEkzzoZaFu+YsigkSTPahMmiqr48Nt2G5HgiXUvjhqr6yRTENu9MNAifJE233j6LJEcC76d7oC7ALkn+uKr+ZdTBSZJmhmGGKP974PeqahV077cAPg+YLCRpnhjm1tn7xhJFcyNw34jikSTNQMO0LFYm+QLdsw5Fd3vr5UmeDVBVnx5hfJKkGWCYZLE5cDvwtDa/FtgCeAZd8jBZSNIcN8xDecdMRSCSpJlrmLuhdqEb8mPJYH0fypOk+WOYy1CfoRu36bNs5GizkqS5YZhk8aOqOmXkkUiSZqxhksXJSY4Hvkg38iwAVXXlyKKSJM0owySL3wBeRPfu7LHLUGPv0pYkzQPDJIvnArs6HpQkzV/DPMF9DbBw1IFIkmauYVoWC4FvJbmcdfssvHVWkuaJYZLF8SOPQpI0ow3zBPeX++pIfSZ6V8fqE4+c4kgkPRy9fRZJ9ktyeZLvJ/lJkgeTfG8qgpMkzQzDdHC/G3gB8G26AQRfCrxnlEFJkmaWYZIF7X0WC6rqwar6EHDYaMOSJM0kw3Rw39/ewX1VkrcBtzFkkpEkzQ3DfOm/qNX7U+AHwM7AH4wyKEnSzDLM3VA3t8kfJTkF2Hnca1YlSXPcMHdDXZzkMUm2Ba4EPpDknaMPTZI0UwxzGWrrqvoe8GzgI1X1ZODg0YYlSZpJhkkWmyTZEXge8LkRxyNJmoGGSRZvAs4HVlXV5Ul2pXvmQpI0TwzTwf1J4JMD8zfi3VCSNK8M85yFJtlE4yRJ0kzlw3WSpF4mC0lSr2Ges/irgenNht1wkp2TXJTkuiTXJnllK982yQVJvt1+btPKk+SUJKuSXJ1kn4FtLWv1v51k2cYdoiTplzVhskjyuiRPAZ4zUPwfG7HtB4A/q6o9gP2AlyXZAzgOuLCqdgcubPMAhwO7t89y4H0tjm3pXsD0ZGBf4PixBCNJmhoball8C3gusGuSryT5ALBdkicMs+Gquq2qrmzT9wHXA4uBo4AzWrUzgGe16aPoHvqrqvoasLA933EocEFV3VVVdwMX4Ki3kjSlNpQs7gHeAKwCDgBObuXHJblkY3aSZAmwN3ApsENV3dYWfQfYoU0vBm4ZWG1NK5uofPw+lidZmWTl2rVrNyY8SVKPDSWLQ4HPA48H3kl3GegHVXVMVf3OsDtIshXwKeBVbdiQn6mqAmqjo16Pqjq1qpZW1dJFixZNxiYlSc2EyaKq3lBVBwGrgTOBBcCiJF9N8tlhNp5kU7pE8dGq+nQrvr1dXqL9vKOV30o3/PmYnVrZROWSpCkyzK2z51fVyqo6FVhTVfsDx/StlCTAacD1VTU4Su0KYOyOpmXAuQPlL253Re0H3NsuV50PHJJkm9axfUgrkyRNkWGG+3jtwOxLWtmdQ2z7qXQvTvpmkqta2RuAE4GzkxwL3Ew3QCHAF4Aj6PpI7qclpKq6K8mbgctbvTdV1V1D7F+SNEk2ariPqvrGRtT9KpAJFh+0nvoFvGyCbZ0OnD7sviVJk8snuCVJvUwWkqReJgtJUi+ThSSpl++z0Iw00Ts/Vp945BRHIglsWUiShmCykCT18jKUppWvmJVmB1sWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1cojyEXL4bUlzhS0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUa2TJIsnpSe5Ics1A2bZJLkjy7fZzm1aeJKckWZXk6iT7DKyzrNX/dpJlo4pXkjSxUbYsPgwcNq7sOODCqtoduLDNAxwO7N4+y4H3QZdcgOOBJwP7AsePJRhJ0tQZWbKoqn8D7hpXfBRwRps+A3jWQPlHqvM1YGGSHYFDgQuq6q6quhu4gF9MQJKkEZvqPosdquq2Nv0dYIc2vRi4ZaDemlY2UfkvSLI8ycokK9euXTu5UUvSPDdtHdxVVUBN4vZOraqlVbV00aJFk7VZSRJTnyxub5eXaD/vaOW3AjsP1NuplU1ULkmaQlOdLFYAY3c0LQPOHSh/cbsraj/g3na56nzgkCTbtI7tQ1qZJGkKjey1qknOAg4Atk+yhu6uphOBs5McC9wMPK9V/wJwBLAKuB84BqCq7kryZuDyVu9NVTW+01ySNGIjSxZV9YIJFh20nroFvGyC7ZwOnD6JoUmSNpJPcEuSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqNbLnLOaTJcd9frpDkKSRsmUhSeply0KzykStuNUnHjnFkUjziy0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb18zkJzgs9fSKNly0KS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSL5+z0Jzm8xfS5LBlIUnqZbKQJPUyWUiSetlnsREmuv6t2ce+DGnj2LKQJPUyWUiSenkZShrg5Slp/WxZSJJ6zZqWRZLDgJOBBcAHq+rEaQ5J88jDubnB1ojmklnRskiyAHgPcDiwB/CCJHtMb1SSNH/MlpbFvsCqqroRIMnHgaOA60axM2+R1WSYrH9HE7VQ7F/RVJotyWIxcMvA/BrgyYMVkiwHlrfZ7ye5YSP3sT1w58OOcHbxWGeRvHXoqtsDd25E/dls1p/XjTCVx/q4iRbMlmTRq6pOBU59uOsnWVlVSycxpBnLY52bPNa5aaYc66zoswBuBXYemN+plUmSpsBsSRaXA7sn2SXJI4GjgRXTHJMkzRuz4jJUVT2Q5E+B8+lunT29qq6d5N087EtYs5DHOjd5rHPTjDjWVNV0xyBJmuFmy2UoSdI0MllIknrN+2SR5LAkNyRZleS46Y5nMiXZOclFSa5Lcm2SV7bybZNckOTb7ec20x3rZEmyIMnXk3yuze+S5NJ2fj/RbpCY9ZIsTHJOkm8luT7JU+bqeU3y6vbv95okZyXZfC6d1ySnJ7kjyTUDZes9l+mc0o776iT7TFWc8zpZzINhRB4A/qyq9gD2A17Wju844MKq2h24sM3PFa8Erh+YfytwUlXtBtwNHDstUU2+k4HzquqJwJPojnnOndcki4FXAEur6n/S3eByNHPrvH4YOGxc2UTn8nBg9/ZZDrxvimKc38mCgWFEquonwNgwInNCVd1WVVe26fvovlAW0x3jGa3aGcCzpifCyZVkJ+BI4INtPsCBwDmtypw41iRbA78LnAZQVT+pqnuYo+eV7q7NLZJsAmwJ3MYcOq9V9W/AXeOKJzqXRwEfqc7XgIVJdpyKOOd7sljfMCKLpymWkUqyBNgbuBTYoapua4u+A+wwTWFNtncBrwUeavPbAfdU1QNtfq6c312AtcCH2iW3DyZ5FHPwvFbVrcA7gP+iSxL3AlcwN8/roInO5bR9Z833ZDEvJNkK+BTwqqr63uCy6u6dnvX3Tyd5OnBHVV0x3bFMgU2AfYD3VdXewA8Yd8lpDp3Xbej+mt4F+FXgUfziJZs5baacy/meLOb8MCJJNqVLFB+tqk+34tvHmq7t5x3TFd8keirwzCSr6S4nHkh3XX9hu3wBc+f8rgHWVNWlbf4cuuQxF8/rwcBNVbW2qn4KfJruXM/F8zpoonM5bd9Z8z1ZzOlhRNo1+9OA66vqnQOLVgDL2vQy4Nypjm2yVdXrq2qnqlpCdx6/VFUvBC4CntOqzZVj/Q5wS5IntKKD6Ibrn3Pnle7y035Jtmz/nseOdc6d13EmOpcrgBe3u6L2A+4duFw1UvP+Ce4kR9Bd6x4bRuSEaQ5p0iTZH/gK8E1+fh3/DXT9FmcDvwbcDDyvqsZ3sM1aSQ4A/ryqnp5kV7qWxrbA14E/rKofT2d8kyHJXnQd+Y8EbgSOofvjb86d1yR/Czyf7u6+rwMvpbtOPyfOa5KzgAPohiK/HTge+AzrOZctYb6b7lLc/cAxVbVySuKc78lCktRvvl+GkiQNwWQhSeplspAk9TJZSJJ6mSwkSb1MFpqRknx/BNvcIsmX2wCSI5NkdZLtR7mPtp+3t9FY3z6u/IAkvzPE+h9O8py+ekNs5x1JDvxlt6OZbVa8VlWaJH8EfLqqHpzuQCaSZJOBMY/6LAe2Xc/xHAB8H7hkMmPbgH8APgB8aYr2p2lgy0KzRpLHJzkvyRVJvpLkia38w22M/0uS3LiBv5ZfSHsStv31ffHAOyE+2h54WqdlkGRpkovb9BuTnNH2fXOSZyd5W5Jvtrg2HdjXa1v5ZUl2a+svSvKpJJe3z1MHtntmkn8Hzhx3zGktiGva9p7fylcAWwFXjJW18iXAnwCvTnJVkv+VZEmSL7X3H1yY5NfW87t9c/s9LkjyFy2+q9sDcbRtXJ/kA60188UkWwBU1c3Adkl+ZdhzqdnHZKHZ5FTg5VX1W8CfA+8dWLYjsD/wdODE8Su24Vx2rarVA8V7A6+ie5fJrnRjDvV5PN24U88E/gm4qKp+A/gh3fDoY+5t5e+mGyEAurGqTqqq3wb+gDaUerMHcHBVvWDc/p4N7EX3zoqDgbcn2bGqngn8sKr2qqpPjFVux/f+tp+9quordH/5n1FVvwl8FDhl3O/m7cAiuqfAD6J7V8K+bb+/leR3W9XdgfdU1Z7APe0YxlzJcL8/zVJehtKskG7k3N8BPtkaAACbDVT5TFU9BFyXZH1Dc29P9wU36LKqWtO2fxWwBPhqTyj/UlU/TfJNuiFizmvl32zrjzlr4OdJbfpgYI+B+B/TjgtgRVX9cD372x84q11quj3Jl4HfZuPGMHsKXdKBruXytoFlfw1cWlXLAZIcAhxCN4QGdK2X3enGaLqpqq5q5Vew7vHeQTcqrOYok4Vmi0fQvcNgrwmWD44LlPUs/yGw+QbWeZCf/394gJ+3ute7TlU9lOSn9fPxch5i3f9PtZ7pRwD7VdWPBjfYkscP1hPzVLicrvWwbRtHKsDfVdU/DlZql7fG/762GJjfnO53rDnKy1CaFdp7OG5K8lz42bX8J23E+ncDC5KM//Jfn9XAb7XpP9hAvQ15/sDP/2jTXwRePlahDQbY5yvA81tfwiK6N+Rd1rPOfcCjB+YvoRuJF7p+m68MLDuP7rLd55M8Gjgf+KOxFk+SxUkeO0Scvw5c01tLs5bJQjPVlknWDHxeQ/dFd2ySbwDXsvGvwP0i3WWdPn8LnJxkJd1f0A/HNkmupnsn+Ktb2SuApa3j+Dq6jug+/wxcDXyD7m6j17Yhyjfks8Dvj3Vw0yWoY1o8L2ox/UxVfZLubqYVdInkY8B/tEtt57Bu4vkFrWN/N2BKRj/V9HDUWc0bSfYBXl1VL5ruWOaSJL8P7FNVfz3dsWh0bFlo3qiqK4GLMuKH8uahTYC/n+4gNFq2LCRJvWxZSJJ6mSwkSb1MFpKkXiYLSVIvk4Ukqdf/B9dWWtejw9LFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get all the sentences\n",
    "sentences = getter.sentences\n",
    "\n",
    "# Plot sentence by lenght\n",
    "plt.hist([len(s) for s in sentences], bins=50)\n",
    "plt.title('Token per sentence')\n",
    "plt.xlabel('Len (number of token)')\n",
    "plt.ylabel('# samples')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PprAvnVuZTkE"
   },
   "source": [
    "# Data Preprocessing\n",
    "Before feeding the data into the model, we have to preprocess the text.\n",
    "\n",
    "* We will use the word2idx dictionary to convert each word to a corresponding integer ID and the tag2idx to do the same for the labels. Representing words as integers saves a lot of memory!\n",
    "* In order to feed the text into our Bi-LSTM-CRF, all texts should be the same length. We ensure this using the sequence.pad_sequences() method and MAX_LEN variable. All texts longer than MAX_LEN are truncated and shorter texts are padded to get them to the same length.\n",
    "The *Tokens per sentence* plot (see above) is useful for setting the MAX_LEN training hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 336
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14663,
     "status": "ok",
     "timestamp": 1595428702333,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "c_EVfMnwXpK4",
    "outputId": "a77a8ec9-d875-44f2-ed67-92fbc5b5baa0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The word Obama is identified by the index: 1322\n",
      "The labels B-geo(which defines Geopraphical Enitities) is identified by the index: 6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw Sample:  Thousands of demonstrators have marched through London to protest the war in Iraq and demand the withdrawal of British troops from that country .\n",
      "Raw Label:  O O O O O O B-geo O O O O O B-geo O O O O O B-gpe O O O O O\n",
      "After processing, sample: [26255 30548 27303 25603 13737 29075 18463 23259 32210  2268 23798 33487\n",
      " 30552 24236   357  2268 15849 30548 29112 31880 22976  7178  1805 27591\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0]\n",
      "After processing, labels: [[0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " ...\n",
      " [1. 0. 0. ... 0. 0. 0.]\n",
      " [1. 0. 0. ... 0. 0. 0.]\n",
      " [1. 0. 0. ... 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "# Vocabulary Key:word -> Value:token_index\n",
    "# The first 2 entries are reserved for PAD and UNK\n",
    "word2idx = {w: i + 2 for i, w in enumerate(words)}\n",
    "word2idx[\"UNK\"] = 1 # Unknown words\n",
    "word2idx[\"PAD\"] = 0 # Padding\n",
    "\n",
    "# Vocabulary Key:token_index -> Value:word\n",
    "idx2word = {i: w for w, i in word2idx.items()}\n",
    "\n",
    "# Vocabulary Key:Label/Tag -> Value:tag_index\n",
    "# The first entry is reserved for PAD\n",
    "tag2idx = {t: i+1 for i, t in enumerate(tags)}\n",
    "tag2idx[\"PAD\"] = 0\n",
    "\n",
    "# Vocabulary Key:tag_index -> Value:Label/Tag\n",
    "idx2tag = {i: w for w, i in tag2idx.items()}\n",
    "\n",
    "print(\"The word Obama is identified by the index: {}\".format(word2idx[\"Obama\"]))\n",
    "print(\"The labels B-geo(which defines Geopraphical Enitities) is identified by the index: {}\".format(tag2idx[\"B-geo\"]))\n",
    "\n",
    "\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "# Convert each sentence from list of Token to list of word_index\n",
    "X = [[word2idx[w[0]] for w in s] for s in sentences]\n",
    "# Padding each sentence to have the same lenght\n",
    "X = pad_sequences(maxlen=MAX_LEN, sequences=X, padding=\"post\", value=word2idx[\"PAD\"])\n",
    "\n",
    "# Convert Tag/Label to tag_index\n",
    "y = [[tag2idx[w[2]] for w in s] for s in sentences]\n",
    "# Padding each sentence to have the same lenght\n",
    "y = pad_sequences(maxlen=MAX_LEN, sequences=y, padding=\"post\", value=tag2idx[\"PAD\"])\n",
    "\n",
    "from keras.utils import to_categorical\n",
    "# One-Hot encode\n",
    "y = [to_categorical(i, num_classes=n_tags+1) for i in y]  # n_tags+1(PAD)\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.1)\n",
    "X_tr.shape, X_te.shape, np.array(y_tr).shape, np.array(y_te).shape\n",
    "\n",
    "print('Raw Sample: ', ' '.join([w[0] for w in sentences[0]]))\n",
    "print('Raw Label: ', ' '.join([w[2] for w in sentences[0]]))\n",
    "print('After processing, sample:', X[0])\n",
    "print('After processing, labels:', y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-ECaYYwbfDmd"
   },
   "source": [
    "# Model\n",
    "We will implement a model similar to Zhiheng Huang’s [Bidirectional LSTM-CRF Models for Sequence Tagging.](https://arxiv.org/pdf/1508.01991v1.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I2YTWFX4747R"
   },
   "outputs": [],
   "source": [
    "# Get CRF layer for Tensroflow 2.x\n",
    "# ! pip install tf2crf\n",
    "# ! pip install sklearn-crfsuite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qTPTG0DTclZk"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input,LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional\n",
    "import tensorflow_addons as tfa\n",
    "from tf2crf import CRF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 370
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 16014,
     "status": "ok",
     "timestamp": 1595428704631,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "dVbHtiW2gBBr",
    "outputId": "0a7b6446-dea9-4089-bff0-befc2e431713"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 75)]              0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 75, 40)            1407200   \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 75, 100)           36400     \n",
      "_________________________________________________________________\n",
      "time_distributed (TimeDistri (None, 75, 50)            5050      \n",
      "_________________________________________________________________\n",
      "crf (CRF)                    (None, 75, 50)            2500      \n",
      "=================================================================\n",
      "Total params: 1,451,150\n",
      "Trainable params: 1,451,150\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Model definition\n",
    "input = Input(shape=(MAX_LEN,))\n",
    "model = Embedding(input_dim=n_words+2, output_dim=EMBEDDING, # n_words + 2 (PAD & UNK)\n",
    "                  input_length=MAX_LEN, mask_zero=True)(input)  # default: 20-dim embedding\n",
    "model = Bidirectional(LSTM(units=50, return_sequences=True,\n",
    "                           recurrent_dropout=0.1))(model)  # variational biLSTM\n",
    "model = TimeDistributed(Dense(50, activation=\"relu\"))(model)  # a dense layer as suggested by neuralNer\n",
    "crf = CRF(n_tags+1)  # CRF layer, n_tags+1(PAD)\n",
    "out = crf(model)  # output\n",
    "\n",
    "model = Model(input, out)\n",
    "model.compile(optimizer=\"rmsprop\", loss=crf.loss, metrics=[crf.accuracy])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fe4_3W_Yggum"
   },
   "source": [
    "# Training & Evaluate\n",
    "*Note:* Accuracy isn't the best metric to choose for evaluating this type of task because most of the time it will correctly predict **'O'** or **'PAD'** without identifing the important Tags, which are the ones we are interested in. So after training for some epochs, we can monitor the **precision**, **recall** and **f1-score** for each of the Tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 185
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 423100,
     "status": "ok",
     "timestamp": 1595429112140,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "wbZ8YGQBgXFh",
    "outputId": "a9fd3022-2d94-4791-d999-d655908c3b86"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "76/76 - 79s - loss: 8.2463 - accuracy: 0.8030 - val_loss: 67.1546 - val_accuracy: 0.8494\n",
      "Epoch 2/5\n",
      "76/76 - 79s - loss: 4.3674 - accuracy: 0.8493 - val_loss: 64.7502 - val_accuracy: 0.8648\n",
      "Epoch 3/5\n",
      "76/76 - 79s - loss: 2.7135 - accuracy: 0.8859 - val_loss: 62.0125 - val_accuracy: 0.9051\n",
      "Epoch 4/5\n",
      "76/76 - 78s - loss: 2.1045 - accuracy: 0.9174 - val_loss: 59.6885 - val_accuracy: 0.9280\n",
      "Epoch 5/5\n",
      "76/76 - 79s - loss: 1.6352 - accuracy: 0.9356 - val_loss: 57.3798 - val_accuracy: 0.9467\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_tr, np.array(y_tr), batch_size=BATCH_SIZE, epochs=EPOCHS,\n",
    "                    validation_split=0.1, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZinaQ6P4g1WD"
   },
   "outputs": [],
   "source": [
    "hist = pd.DataFrame(history.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 700
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 422709,
     "status": "ok",
     "timestamp": 1595429112143,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "v2IbbscXg1xM",
    "outputId": "92534a03-97eb-4bb3-ed6b-257ecd006eef"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAAKrCAYAAAADCRC8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd4BU1f3//+e5W9nObF926UhHVFAERardaOzGqLHEjsYSo4m9Jx/z+eSrxpaoMSYo1thLEBAUZSlSliJFkF1ge+87c8/vj43+hKiwsMudmX09/tvskHnNm92ZF9dzzjXWWouIiIiIiHzL8TqAiIiIiEiwUUkWEREREdmFSrKIiIiIyC5UkkVEREREdqGSLCIiIiKyC5VkEREREZFdRHod4Pts377dk+dNS0ujvLzck+cORZpXx2heHaN5dYzm1TGaV8doXh2nmXWMV/PKycn5we/pSrKIiIiIyC5UkkVEREREdqGSLCIiIiKyC5VkEREREZFdqCSLiIiIiOxCJVlEREREZBcqySIiIiIiu1BJFhERERHZhUqyiIiIiMguVJJFRERERHahkiwiIiIisguVZBERERGRXagki4iIiIjsQiVZRERERGQXKskiIiIiIrtQSRYRERER2YVKsoiIiIjILlSSRURERER2oZIsIiIiIrILlWQRERERkV2oJIuIiIiI7EIlWURERERkFyrJIiIiIiK7UEkWEREREdmFSrKIiIiIyC5UkkVEREREdqGSLCIiIiKyC5VkEREREfFEYU0LD32yja+rGr2O8l8ivQ4gIiIiIt1LUW0Ls1ZVsGBLLTGRhqPLGxnV0+tUO1NJFhEREZH9YkddKy+uKmf+llqiHMNPh/n46VAf/XPTKC8v9zreTlSSRURERKRLFde1Mquggnmba4h0DD8Z4uOnw3ykxAZvFQ3eZCIiIiIS0krqW3m5oII5X9UQ4RhOGNyT04al0rNH8FfQ4E8oIiIiIiGlrKGNlwsq+OirasBw7AE9OW2Yj9S4KK+j7TGVZBERERHpFBWN7eX435tqAMv0ASmcPiKVtBAqx99QSRYRERGRfVLZ5OfV1RV8sKEa11qmDUjhjBGppMeHXjn+hkqyiIiIiOyV6iY/r66p4P0N1fhdy5T+yZw5IpXMhGivo+0zlWQRERER6ZCaZj+vrank3fVV+F3LpH7t5Tg7MfTL8TdUkkVERERkj9Q2+3l9bXs5bg1YJvZN4qwRaeQkhU85/oZKsoiIiIj8qLqWAG+sreStL6to8bsc2SeJs0amkpsc43W0LqOSLCIiIiLfq741wJvrKnlrXRWNbS4Teidy9qg0eodxOf6GSrKIiIiI7KShNcBbX1bx5tpKGtpcDs9L5OyRqfTtGet1tP1GJVlEREREAGhsC/D2l1W8sbaS+laXw3ITOHtkGv193accf0MlWURERKSba2pzeXd9Fa+vraSuJcDYXvGcPTKdgandrxx/QyVZREREpJtq8beX49fWVFLbEuCQnHjOHpnGAWk9vI7mOZVkERERkW6mxe/y/oZqXltTQXVzgNHZ8ZwzMo0h6SrH31BJFhEREekmWgMuH2yo5tU1lVQ1+RmVFcfNI9MYmhHndbSgo5IsIiIiEubaAi7/3lTDKwUVVDT5GZ7Rgxsn5DAiU+X4h6gki4iIiISptoDlo6+qebmggvJGP8PSe/Cr8dmMzIzDGON1vKCmkiwiIiISZvyuZc5XNbxcUE5pg5/BabHMGJfNgVkqx3tKJVlEREQkTARcy7zNNcwqqKCkvo1BqbFccWgWB2XHqxx3kEqyiIiISIgLuJb5W2qZVVDOjro2BvhiuHRSLofkqBzvLZVkERERkRAVcC2ffF3Li6sq2F7XSr+eMfx2Yi8OzU1QOd5HKskiIiIiIca1lk+/ruPFVeUU1bbSJyWGm4/sxWF5CTgqx51ij0ry8uXLefbZZ3Fdl6lTp3LKKafs9P2ysjIef/xxamtrSUhIYMaMGaSmpn77/cbGRq6//nrGjh3LxRdf3LmvQERERKSbcK3ls8I6Zq2s4OuaFvKSo7npiBwO752octzJdluSXdfl6aef5tZbbyU1NZVbbrmFMWPGkJub++1jnn/+eSZOnMikSZMoKChg5syZzJgx49vvz5o1i6FDh3bNKxAREREJc9ZaFhXV88LKcrZUt9ArKZobJuQwoXciEY7KcVdwdveAjRs3kpWVRWZmJpGRkYwfP57Fixfv9JiioiJGjBgBwPDhw1myZMm33/vqq6+oqanhwAMP7OToIiIiIuHNWkt+UR3Xv7eFB+ZvozXgct34bB45oR8T+yapIHeh3V5Jrqys3GnpRGpqKhs2bNjpMX369CE/P5/jjz+e/Px8mpqaqKurIz4+nr///e/MmDGDVatW/eBzzJ49m9mzZwPw4IMPkpaWtrevZ59ERkZ69tyhSPPqGM2rYzSvjtG8Okbz6hjNq+P2dWbWWj7bUsXTn29lXWk9Ocmx/G76II4ekkFkGBbjYPwZ65SNe+eddx7PPPMM8+bNY+jQofh8PhzH4cMPP+Sggw7aqWR/n2nTpjFt2rRvvy4vL++MWB2Wlpbm2XOHIs2rYzSvjtG8Okbz6hjNq2M0r47b25lZa/liRwMvrCxnfUUzGfFRzBiXxaR+yUQ6hurKii5I6z2vfsZycnJ+8Hu7Lck+n4+Kiv//L6SiogKfz/dfj7nxxhsBaG5uZtGiRcTHx7N+/XrWrl3Lhx9+SHNzM36/n9jYWM4999y9fS0iIiIiYcday8qSRmauKGddeRPpcZFcdVgWk/slExURfleOQ8FuS/KAAQPYsWMHpaWl+Hw+Fi5cyDXXXLPTY7451cJxHF5//XUmT54MsNPj5s2bx6ZNm1SQRURERL5jVUn7lePVpU2kxkVy+dhMpg1IJipit1vHpAvttiRHRERw0UUXcd999+G6LpMnTyYvL49Zs2YxYMAAxowZw5o1a5g5cybGGIYOHapj3kRERER2Y3VpIy+sLGdVSSM9e0Ry6ZhMpg9MJlrlOCgYa631OsSutm/f7snzas1Vx2heHaN5dYzm1TGaV8doXh2jeXXcj81sbVl7OV5R3EhKbASnD0/l6IEpxER233IckmuSRURERGTffVnexAsry/liRwPJMRFcdHAGxw7q3uU4mKkki4iIiHShDRXt5Xjp9gYSYyK4YHQ6xw/uSazKcVBTSRYRERHpAl9VNvPCqnLyi+pJiHY478B0jh+cQlxUhNfRZA+oJIuIiIh0oi1Vzfzx87XM31RBfLTDuaPSOHFIT5XjEKOSLCIiItIJvq5u4cVV5SzcWkd8dATnjGwvxwnRKsehSCVZREREZB8U1rSX40+/riM20uHMEalcOGEQrfXVXkeTfaCSLCIiIrIXttW2MmtVOfO31BITaThteConD/WRFBNBUmwk5fVeJ5R9oZIsIiIi0gE76trL8cdbaolyDD8d5uOUoT6SY1Wrwon+NkVERET2QHFdKy8VVDB3cw2RjuEnQ3z8dJiPFJXjsKS/VREREZEfUVrfxksF5cz5qgbHGE44oCenDk/F10M1Kpzpb1dERETke5Q1tPHK6gpmb6oGDMcOSuG04amkxkV5HU32A5VkERERke+oaGwvxx9urAEs0we0l+P0eJXjzmZLd2DfewX/uZdCZIzXcXaikiwiIiICVDb5eXV1BR9sqMa1lqkDkjljeBoZCSrHnc0Wb8O++zJ20TxwImg7dAIMPdjrWDtRSRYREZFurbrJz2trKnhvQzV+1zKlfzJnjkglMyHa62hhx+4oxL7zEjZ/AURGYqaciDnmVHoMPICG8nKv4+1EJVlERES6pZpmP6+vqeSd9VX4XcukfkmcOSKN7ESV485mt23FvjMLu+QTiIrGTD8Zc/QpmOSeXkf7QSrJIiIi0q3UtgT415oK3llfRWvAMrFPEmeOTKNXkspxZ7NFm3HfngVLF0JMD8yxp2Kmn4JJTPY62m6pJIuIiEi3UNcS4I21lbz1ZRUtfpcj+yRx1shUcpODa8NYOLBbN7WX4y8+h9gemOPPxEz/CSYhyetoe0wlWURERMJafWuAt9ZV8ua6KhrbXCb0TuTskWn0TlE57mx28wbcd2bBinzoEY856WzM1J9g4hO8jtZhKskiIiISlhrbAry1roo31lXS0OpyeF4CZ49Mo2/PWK+jhR27aV37leOCpRCXgDn5Z+2b8uJCrxx/QyVZREREwkpjW4B3vqziX2srqW91OSy3vRz396kcdza7cQ3uW7NgzReQkIj56XmYySdgesR5HW2fqSSLiIhIWGhqc3l3fRWvr62kriXA2F7xnD0ynYGpKsedza4vwH3rRVi3EhKTMaddgJl0HCY29MvxN1SSRUREJKS1+P9TjtdUUtMS4ODseM4ZlcYBaT28jhZWrLWwbmX7sor1BZCUgjnjIsxRx2Jiwu8fIirJIiIiEpJa/C4fbKzm1dUVVDcHGJ0Vxzmj0hmSrnLcmay1sHZ5+7KKjWsg2Yc56xLMxGMw0eG7+VElWUREREJKa8Dlw43VvLK6kqomP6My4/jNkWkMywif/9QfDKy1ULAM9+0X4asvoWca5meXYY6YjokK/zOlVZJFREQkJLQFXP69qYZXCiqoaPIzPKMHN0zIZmRmvNfRwoq1FlYubl9z/PVG8KVjzr0CM2EaJirK63j7jUqyiIiIBLW2gGXOVzW8VFBOeaOfoek9+NX4bEZmxmGM8Tpe2LCuCyvy268cb/0K0jIx51+NOXwyJrL7lONvqCSLiIhIUPK7lrn/KcelDX4Gp8Vy9bhsRmepHHcm67rwxWftG/KKtkB6FuYX12IOOwoT2X2rYvd95SIiIhKUAq5l3uYaXiqooLi+jUGpsVw+NouDc+JVjjuRdQPYpQuxb8+C7Vshsxfmouswh07ERER4Hc9zKskiIiISFAKuZf6WWmYVlLOjro3+PWO49ahcxvRSOe5MNhDALl6AfeclKC6C7DzMJTdgxh6BcVSOv6GSLCIiIp4KuJZPt9bx4qpyttW20q9nDL+d2ItDcxNUjjuRDQSwiz5uL8el26FXH8ylN2EOOVzl+HuoJIuIiIgnXGtZuLWOF1aWU1TbSp/kGG4+sheH5SXgqBx3Guv3Yz+fi333ZSgrhtx+OFfcDKPHYRzH63hBSyVZRERE9ivXWj4vrOPFlRV8XdNCXnI0vz4ih/G9E1WOO5H1t2EXzmkvxxWl0HsAzlW/hQMP0xX6PaCSLCIiIvuFtZZFRfW8uKqczVUt9EqK5oYJOUzonUiEo9LWWWxbG/bTf2PfewUqy6HvIJyfXQYjx6gcd4BKsoiIiHQpay1LtjXwwqoyNlW2kJ0YxXXjszmyT5LKcSeyba3Y+R9i338Vqiug/2Cc866C4QerHO8FlWQRERHpEtZalm1v4IVV5WyoaCYrIYprD8/mqL4qx53JtrRgF7yPff91qKmEgcNwLrwWhh6ocrwPVJJFRESkU1lrWV7cyMwVZayvaCYjPoqrD8ticv9kIlWOO41tacbOew/7wWtQVwODR+L88gY4YITKcSdQSRYREZFOYa1lZUkjL6wsZ21ZE2lxkVx5aBZT+icTFaHS1llscyN27nvYD1+H+loYeiDOiWdhDhjhdbSwopIsIiIi+6ygpJGZK8tYXdpEao9ILh+bybQByURF6IixzmKbGrFz3sb++w1oqIPhB+GceDZm4FCvo4UllWQRERHZa2tKG5m5spxVJY307BHJL8dkcPTAFKJVjjuNbazHfvQ2dvYb0NgAI8e0XznuP9jraGFNJVlEREQ6bF1ZEy+sLGN5cSMpsRFcfEgGxwxMISZS5biz2IY67Ow3sR+9BU2NMPqw9nLcZ6DX0boFlWQRERHZY+vLm3hhZTnLdjSQHBPBhQenc9ygnirHncjW1WJnv4Gd8zY0N8HBh+OccBamd3+vo3UrKskiIiKyWxsrmnlhZRlLtjeQGBPBBaPTOX5wT2JVjjuNra3Gfvgv7Lx3obUFc8gEzAlnYnL7eh2tW1JJFhERkR+0rbaVhz5bw4KvKkmIdvj5gWmcMLgncVERXkcLG7amCvvh69h570FbK2bskZjjz8T06u11tG5NJVlERET+S1vA5dXVlby8uoKYSIefjUrjpCEqx53JVldg338NO/8D8Psxhx2FOf4MTHau19EElWQRERHZxcriBh7PL2F7XStH9knk19OHYptqvY4VNmxlOfb9V7AL/g1uADNucns5zszxOpp8h0qyiIiIAFDT7OeZZaXM21xLVkIUd0zO5eCcBFLjoylv8jpd6LMVpdj3XsF+OhusxYyfijnudEx6ltfR5HuoJIuIiHRzrrV8tKmGv31RSrPf5YzhqZwxIlUnVnQSW1bcXo4XfgQYzIRpmONPx6RmeB1NfoRKsoiISDe2tbqFx/OLWVPWxLD0HlxxWBa9k2O8jhUWbOl27LsvYz+bC46DmXgM5tjTML50r6PJHlBJFhER6YZa/C4vFVTw+poK4qIcZozLYkr/ZBxjvI4W8mxxETX/fBx3/gcQEYmZfALmmFMxPVO9jiYdoJIsIiLSzSzbXs+Ti0sorm9jSv8kfnFQBsmxqgT7ym7fin3nZeziBTRHRWGmntRejpN7eh1N9oJ+I0RERLqJyiY/Ty8t4ZOv68hJjOaeqXmMyor3OlbIs0VbsO+8hF36KUTHYI4+hbSzL6LS73odTfaBSrKIiEiYc63lgw3VPL+8jJaA5ZxRaZw2zEdUhDbm7QtbuBn37VmwbCHE9Ghfbzz9FExiEk6KD8rLvY4o+0AlWUREJIxtrmrmsUXFrK9oZlRmHJcfmkWvpGivY4U0+/Um3LdfhOWLoEcc5sSzMNN+golP9DqadCKVZBERkTDU1Oby4qpy3lxXSWJ0BNeNz+aovkkYbczba3bz+vYrxysXQ1w85qRz2tcdxyd4HU26gEqyiIhImMkvquOpxSWUNfo5emAy54/OIDFGt5PeW3bTuvYrxwXLID4Rc8rP20+siNN67nCmkiwiIhImyhvb+MuSEj4vrKd3cjQPTO/NsIw4r2OFLLthDe5bL8DaFZCQhDn1Aszk4zCxmml3oJIsIiIS4gKu5Z31VfxzRTmutZw3Op2Th/iIitDSir1hv1yF+9aL8OUqSEzGnH4h5qhjMbE9vI4m+5FKsoiISAjbUNHE4/nFbKps4eDseC4bm0lWojbmdZS1FtatbF9WsX41JPfEnHkxZuKxmBjdgbA7UkkWEREJQY1tAf6xopz31leRHBPBr4/IYULvRG3M6yBrLaz+or0cb1oHKT7M2ZdijpyOiVY57s5UkkVEREKItZaFhXX8dUkpVU1+jjsghZ8fmE58tDbmdYS1FgqWti+r2LwefGmYcy/HTJiGidKVeFFJFhERCRkl9a08tbiEJdsb6Nczhlsm9uKANK2T7QhrLazIbz/K7euNkJqBOe9KzPipmMgor+NJEFFJFhERCXJ+1/Lm2kpeXFWOMXDRwRmcOLgnEY6WVuwp67qw/PP2cly4GdKzMBfMwIybjIlUHZL/pp8KERGRILaurInH8ov5urqFw3IT+OWYTNLjdcVzT1nXhWUL28vxtq8hIxtz4bWYQ49SOZYfpZ8OERGRIFTfEuDvy8v4cGM1vrhIbpnYi3F5uu3xnrJuALvkU+zbs2BHIWTlYi6+HjP2SEyE1m/L7qkki4iIBBFrLfO31PL0slLqWgKcNKQn54xKIy5KxW5P2EAAu3g+9p2XoHgb5PTGXPprzCHjMY5mKHtOJVlERCRI7Khr5Yn8YpYXNzLQF8sdk/MY4Iv1OlZIsH4/dtHH2HdfgtId0KsPzuW/gYMOxziO1/EkBKkki4iIeKwt4PL6mkpeKqgg0jFcOiaTYwelaGPeHrD+Nuxnc7HvvQJlxZDXD+eKW2D0YSrHsk9UkkVERDy0uqSRx/KLKaptZULvRC4+JIPUOG3M2x3b1oZd+FF7Oa4ohT4Dca6+FUaN1Q1VpFOoJIuIiHigttnP374o46OvasiIj+K2SbmM6ZXgdaygZ9tasZ/Mxr7/ClSWQ78DcM69AkYcrHIsnUolWUREZD+y1jJ3cy3PLCulsTXAqcN8nD0yjZhILQ34Mba1BbvgQ+z7r0J1JQwYgnP+DBg2WuVYuoRKsoiIyH5SVNPC44tLKChpZEhaD644NJO+PbUx78fYlhbsx+9hP3gNaqvhgOE4F10HQ0apHEuXUkkWERHpYq0Bl5cLKnhtTQUxkQ5XHprF9IHJOCp5P8g2N/2nHL8OdTUwZBTOpTdhBo/wOpp0EyrJIiIiXWj5jgaeWFzMjro2juqbxEUHZ5DSQx+/P8Q2N2Lnvov98F9QXwvDDsI58SzMoGFeR5NuZo9+S5cvX86zzz6L67pMnTqVU045Zafvl5WV8fjjj1NbW0tCQgIzZswgNTWVLVu28Je//IWmpiYcx+HUU09l/PjxXfJCREREgkl1k59nlpXy8ZZashOjuGtKHqOz472OFbRsYwN2ztvY2W9CQx2MOKS9HA8Y4nU06aZ2W5Jd1+Xpp5/m1ltvJTU1lVtuuYUxY8aQm5v77WOef/55Jk6cyKRJkygoKGDmzJnMmDGD6Ohorr76arKzs6msrOTmm2/mwAMPJD5ebxIiIhKeXGv598YanlteSovf5cwRqZwxIpXoCG3M+z62oR770ZvYj96CxgY48FCcE87C9BvkdTTp5nZbkjdu3EhWVhaZmZkAjB8/nsWLF+9UkouKijj//PMBGD58OP/zP/8DQE5OzreP8fl8JCcnU1tbq5IsIiJhaUtVM4/nl7CuvIkRGT244tAscpNjvI4VlGx9LXb2m9g5b0NTI4we137luM8Ar6OJAHtQkisrK0lNTf3269TUVDZs2LDTY/r06UN+fj7HH388+fn5NDU1UVdXR2Ji4reP2bhxI36//9uy/V2zZ89m9uzZADz44IOkpaXt9QvaF5GRkZ49dyjSvDpG8+oYzatjNK+O6ex5NbcFeHZRIS98sY2E6Ah+O30Qxw/NCJvTFzpzXm5NFQ1vvkjTu69imxuJOXwy8Wf8gqgwu3Ks38mOCcZ5dcrOgfPOO49nnnmGefPmMXToUHw+H853bgVZVVXFI488wlVXXbXT//6NadOmMW3atG+/Li8v74xYHZaWlubZc4cizatjNK+O0bw6RvPqmM6c15Jt9Ty5uITShjam9k/mFwelkxQbQUVFRaf8/weDzpiXra3Cfvgv7Lz3oLUFM+YInBPOxN+rDzUAYfbzq9/JjvFqXt9d9bCr3ZZkn8+30y96RUUFPp/vvx5z4403AtDc3MyiRYu+XVLR2NjIgw8+yDnnnMMBBxywVy9AREQk2FQ0tvHXpaUs3FpHblI0903rzYjMOK9jBR1bXYn94HXs/PegzY859EjMCWdisvO8jibyo3ZbkgcMGMCOHTsoLS3F5/OxcOFCrrnmmp0e882pFo7j8PrrrzN58mQA/H4/Dz30EBMnTmTcuHFd8wpERET2o4BreX9DNc8vL8PvWs4dlcZPh/mI0sa8ndiqCuwHr2HnfwABP+awSZjjz8Bk9fI6msge2W1JjoiI4KKLLuK+++7DdV0mT55MXl4es2bNYsCAAYwZM4Y1a9Ywc+ZMjDEMHTqUiy++GICFCxeydu1a6urqmDdvHgBXXXUVffv27crXJCIi0iW+qmzmsfxiNlQ0MzorjssPzSI7MdrrWEHFVpRh338V+8mHYC1m3GTM8adjMn74P2uLBCNjrbVeh9jV9u3bPXlerR/qGM2rYzSvjtG8Okbz6piOzqupzWXmyjLe/rKKxJgILjkkkyP7JIbNxrzd2ZN52fIS7HuvYD/9CAAzfgrmuNMx6Vn7I2LQ0e9kx4TkmmQREZHu7PPCOp5aUkJFo59jBqZw/uh0EmIivI4VNGxZMfbdl7GfzQFjMEdOxxx7GiY1w+toIvtEJVlEROR7lDW08dSSEvKL6umTEsNNR/RiSHoPr2MFDVuyvb0cfz4XnAjMUcdhjjkV4wuuY7xE9pZKsoiIyHcEXMvbX1Yxc2UZroULDkrnJ0N8RDrdY2nF7tgdRdh3X8Iumg+RkZgpJ7aX4xTf7v+wSAhRSRYREfmP9eVNPJZfzOaqFsbkxHPp2EwyE7QxD8Bu29pejhcvgKhozPSTMUefgknu6XU0kS6hkiwiIt1eQ2uAf6wo47311fTsEclvjszh8LzuszHvx9jqCqqf/RPuZ3MhOqb9qvH0kzFJKV5HE+lSKskiItJtWWv5dGsdf11aSnWTn+MH9+TnB6YRF6WNeQC2cDPuI/fQ0liPOe4MzLSfYBKTvI4lsl+oJIuISLdUUt/KE/klLNvRwABfDL87qheDUrUx7xt25WLcpx6CHnH4HniSmkQtq5DuRSVZRES6Fb9reX5xIc8s2opjDJccksHxB/QkQhvzvuV+9DZ21l8hrx/OjFuJ6jcIdOavdDMqySIi0m2sLW3ksfxitta0Mi4vgV+OySQtLsrrWEHDBgLYWX/Fzn0HDjwU55c3YmJivY4l4gmVZBERCXt1LQGe+6KUf2+qIT0ukt+fNJQhSUF3w1lP2ebG9uUVq5a0b8w7/RcYR2uzpftSSRYRkbBlreXjLbU8s7SUutYApwz1cfbINPKyU3XL4O+wlWW4j9wD27dizr0CZ9JxXkcS8ZxKsoiIhKVtta08sbiYlcWNHJAay12H5dGvp5YO7Mpu2YD76L3Q2oJzzR2Y4Qd5HUkkKKgki4hIWGkLuLy6ppJXCiqIijBcNjaTYwamaGPe97BffI7714cgMQXnursxvfp4HUkkaKgki4hI2FhZ3MATi0vYVtvKEX0SufiQTHw99FG3K2st9sN/YV/9G/QdhHP17zBJOuJN5Lv0ziEiIiGvptnPs8tKmbu5lsyEKO6YnMvBOQlexwpK1u/HvvAkdv4HmEMmYC76FSY6xutYIkFHJVlEREKWay0fbarhuS9KaWxzOX14KmeOSCUm0vE6WlCyjQ24T/4e1izHHHc65pSfYxzNSuT7qCSLiEhI2lrTwuOLillT1sSw9B5ccWgWvVN0RfSH2PIS3IfvhtLtmF9cgzNhmteRRIKaSrKIiISUFr/LSwUV/GttBbGRDlcflsXUAe54W6EAACAASURBVMk4RhvzfojdtA73z/dBwI/zq7swQ0Z5HUkk6Kkki4hIyPhiRwNP5BdTXN/GpH5JXHhwBimx+ij7Me7iT7DP/glSfDjX3I7JyvU6kkhI0DuLiIgEvaomP08vLWHB13XkJEZzz9Q8RmXFex0rqFlrse++jP3XP2DgUJwrf4dJTPI6lkjIUEkWEZGg5VrLBxuqeX55GS0Byzkj0zh1uI/oCG02+zHW34Z9/jHswo8whx6F+cUMTFS017FEQopKsoiIBKUtVc08ll/Ml+XNjMyM4/JDM8lN0sa83bENdbiPPQDrCzAnnY056RyM1muLdJhKsoiIBJVmv8sLK8t5c10lCdER/OrwbCb1S1LR2wO2dDvuw/dARQnm4utxxk3yOpJIyFJJFhGRoLG4qJ4nFxdT1uhn+oBkLjgog8SYCK9jhQS7fjXuY/eDAee6ezAHDPc6kkhIU0kWERHPlTe28dclJXxWWE9ecjT3T+/N8Iw4r2OFDPfzudjnHoHUTJxrbsNk5HgdSSTkqSSLiIhnAq7l3fVV/GNFOa61nHdgOicP9REVoaUVe8Jai33zBezbL8LgkThX3IyJT/Q6lkhYUEkWERFPbKxo35i3qbKZg7LjuWxsJtmJOoFhT9m2VuzfHsbmz8eMn4o570pMZJTXsUTChkqyiIjsV41tAf65opx311eRHBPBjRNyOKJPojbmdYCtq2lff7xxLean52GOO13zE+lkKskiIrJfWGv5vLCep5aUUNXk59hBKfx8dDoJ0dqY1xF2RxHuI3dDdSXOZTdhxhzhdSSRsKSSLCIiXa60vo2nlhSzeFsD/XrGcPPEXgxO6+F1rJBj167AfeJBiIjEueFezIAhXkcSCVsqySIi0mX8ruXNdZW8uLIcgAsPTuekwT4iHC0N6Cj309nY5/8MGTk4M27DpGd5HUkkrKkki4hIl/iyvInHFhWzpbqFsb0SuHRMJhkJ2ljWUdZ1sf96HvveqzBsdPsSi7gEr2OJhD2VZBER6VT1rQGeX17GBxuq8fWI5OaJvRiXm6CNZXvBtrbgPvN/sHQhZuIxmHMuw0Tqo1tkf9BvmoiIdAprLQu+ruPppSXUtgQ4cXBPfnZgGnFR2pi3N2xtFe6j98GWDZgzLsRMP0X/0BDZj1SSRURkn+2oa+WJxSUs39HAQF8st0/OY4Av1utYIctu+xr3kXugrgbnilswB43zOpJIt6OSLCIie60tYHl9bQUvF1QQYQy/HJPBcYN6amPePrAFy3Cf+gNEx+Lc9ACmz0CvI4l0SyrJIiKyV1aXNPJYfjFFta2M753IJYdkkBqnjXn7wp33HvaFJyGnd/sJFr50ryOJdFsqySIi0iG1LQGe+6KU2ZtqyIiP5LZJuYzppdMW9oV1A9iX/4ad/QaMHINz6Y2Y2DivY4l0ayrJIiKyR6y1zN1cy7PLSqlvDXDqMB9njUwjNtLxOlpIsy3NuH95CFbkY6aciDnzYkyENjuKeE0lWUREdquopoXHF5dQUNLI4LRYrjw0j749tTFvX9nqCtxH7oXCzZizL8WZeqLXkUTkP1SSRUTkB7UGXF5ZXcGrqyuJiTRccWgmRw9MwdFRZPvMbv2q/QSLpkacq3+HGTXW60gi8h0qySIi8r1WFDfwRH4x2+vamNg3iYsPziClhz42OoNduRj3qf+BuASc3zyIyevndSQR2YXe7UREZCfVzX6eXVrKvC21ZCVEcdeUPEZnx3sdK2y4H72FnfU05PXDmXErJiXV60gi8j1UkkVEBADXWmZvquG5L0pp9rucOSKV04enEqONeZ3CBgLYWX/Fzn0HRh+Gc8kNmBit6xYJVirJIiLC19UtPJ5fzNqyJoZn9OCKQ7PIS47xOlbYsE2N7csrCpZijj4Fc9oFGEcnWIgEM5VkEZFurMXvMmtVOf9aW0lcdAQzxmUxtX8yRhvzOo2tKMN95G7YUYj5+ZU4Rx3rdSQR2QMqySIi3dTSbfU8uaSEkvo2pvRP5sKD0kmK1cdCZ7JbNuA+ei+0tuBccwdm+EFeRxKRPaR3QxGRbqaisY2nl5by6dY6eiVFc++0PEZmamNeZ7PLPsN9+o+QmIJz3T2YXr29jiQiHaCSLCLSTQRcy/sbqvnHijLaApafjUrj1GE+oiK0Ma8zWWuxH76OffU56Duo/QzkpJ5exxKRDlJJFhHpBr6qbOax/GI2VDRzYFYcl4/NIicp2utYYcf6/diZT2AXfIg5ZALmol9horUBUiQUqSSLiISxpjaXF1aW8daXVSTGRHDd+GyO6pukjXldwDbW4z7xe1i7AnP8GZiTz8U4ukovEqpUkkVEwtSiwjqeWlJCeaOfYwamcP7odBJidOxYV7Blxe23mC7dgfnFtTgTpnodSUT2kUqyiEiYKWto4y9LSlhUVE+f5BhunJ7D0Iw4r2OFLbtpHe6f74NAAOe6uzCDR3odSUQ6gUqyiEiYCLiWt7+sYubKMlwL549O5+ShPiIdLa3oKu7iBdhn/gQ9U3GuuR2Tlet1JBHpJCrJIiJhYG1xHfd9uIXNVS0ckhPPZWMzyUzQxryuYq3Fvvsy9l//gIFDca78HSYxyetYItKJVJJFRELcG2sreXZZKSk9IrnpyBzG5yVqY14Xsv427N//jP1sDuawozAXXIOJivI6loh0MpVkEZEQ9tGmap5ZVsrEAalcfrCP+GhtzOtKtqEO97EHYH0B5qRzMCedrX+QiIQplWQRkRCVX1THo4uKOTArjruOHUxtdaXXkcKaLdmO+/DdUFmKufh6nHGTvI4kIl1IJVlEJAStLm3kfz7ZzgBfLDdP7EV0pM7j7Up2/Wrcx+4HA87192IGDfM6koh0MZVkEZEQs6WqmfvmFZEeH8Vtk3KJi9ISi67kfjYX+9wjkJ6JM+N2TEa215FEZD9QSRYRCSEl9a3cOaeQ2EiHOyfnkRyrt/GuYq3FvjkT+/YsGDwS54pbMPEJXscSkf1E764iIiGiusnP7R8V0uZaHpjem4wEnajQVWxbK/ZvD2Pz52MmTMX8/EpMpOYt0p2oJIuIhICG1gB3zi2kqsnP3VN70zslxutIYcvW1bSvP964FnPq+ZhjT9MJFiLdkEqyiEiQaw243D9/G1urW/jdUbkMSe/hdaSwZXcU4T5yN1RX4lx2E2bMEV5HEhGPqCSLiASxgGv546fbKShp5Lrx2RzSS2tiu4pduwL3iQchIhLnxvsw/Qd7HUlEPKSSLCISpKy1PJ5fzOeF9VxySAaT+iV7HSlsuZ/8G/uPxyAjB+ea2zFpmV5HEhGPqSSLiASpf6wo59+bajh9eConDfF5HScsWdfFvv489v1XYdhonMt+g4mL9zqWiAQBlWQRkSD05rpKXlldwdEDk/n5gWlexwlLtqUF95n/g2ULMROPxZxzKSZSH4si0k7vBiIiQWbe5hqeXlrK4XkJXD42SycrdIFAVQXuH38HWzZgzrgIM/1kzVlEdqKSLCISRJZuq+fhz3YwIjOO6yfkEOGouHU2u+1rKv98L9RUt98g5KBxXkcSkSCkkiwiEiTWlTXx4IJt9EmJ4XdH9SI6wvE6UtixBctwn/w9To94nJsewPQZ6HUkEQlSKskiIkFga3UL98wrJDUukjsm5xEXFeF1pLDjznsX+8JTkNMH3x3/RxX6R4iI/DCVZBERj5XWt3HnnEKiHMNdU/JI6aG35s5k3QD25b9hZ78BI8fgXHojEWkZUF7udTQRCWJ6JxYR8VBNs5875hTS7He5f3pvMhOivY4UVmxzE+5f/wgr8jFTT8KceRHG0VV6Edm9PSrJy5cv59lnn8V1XaZOncopp5yy0/fLysp4/PHHqa2tJSEhgRkzZpCamgrAvHnzeO211wA49dRTmTRpUue+AhGRENXYFuDuuUWUN7Zx55Q8+vaM9TpSWLFVFbiP3gOFWzDnXIoz5USvI4lICNltSXZdl6effppbb72V1NRUbrnlFsaMGUNubu63j3n++eeZOHEikyZNoqCggJkzZzJjxgzq6+t55ZVXePDBBwG4+eabGTNmDAkJuq2qiHRvbQGXB+Zv46uqZm6Z2IvhGXFeRwordutXuI/cA02NODNuxYwc43UkEQkxu921sHHjRrKyssjMzCQyMpLx48ezePHinR5TVFTEiBEjABg+fDhLliwB2q9Ajxo1ioSEBBISEhg1ahTLly/vgpchIhI6Aq7l/xbuYGVxIzPGZXNobqLXkcKKXbEY9w83gzE4v3lQBVlE9spuryRXVlZ+u3QCIDU1lQ0bNuz0mD59+pCfn8/xxx9Pfn4+TU1N1NXV/def9fl8VFZW/tdzzJ49m9mzZwPw4IMPkpbmzd2lIiMjPXvuUKR5dYzm1THhOi9rLf877ys+3VrHVUf05cxDcnf/h/ZAuM6rI6y1NL39EnV/e4TIfgeQ8ts/EOH7/ploXh2jeXWcZtYxwTivTtm4d9555/HMM88wb948hg4dis/nw3H2/GidadOmMW3atG+/Lvdox3FaWppnzx2KNK+O0bw6Jlzn9cLKMl5bVcFPh/o4uk9sp73GcJ3XnrKBAHbWX7Bz34XR43AvuZ4qlx88waK7z6ujNK+O08w6xqt55eTk/OD3dluSfT4fFRUV335dUVGBz+f7r8fceOONADQ3N7No0SLi4+Px+XysWbPm28dVVlYybNiwDr8AEZFw8M6XVby4qoKp/ZO54KB0r+OEDdvUiPvU/0DBUszRP8WcdgGmAxdqRES+z27fRQYMGMCOHTsoLS3F7/ezcOFCxozZeX1XbW0trusC8PrrrzN58mQARo8ezYoVK6ivr6e+vp4VK1YwevToLngZIiLBbcGWWv6ypIRDcxO46rAsjNHtpjuDrSjD/f1vYM0XmPOuxDnjQhVkEekUu72SHBERwUUXXcR9992H67pMnjyZvLw8Zs2axYABAxgzZgxr1qxh5syZGGMYOnQoF198MQAJCQmcdtpp3HLLLQCcfvrpOtlCRLqd5Tsa+NNn2xma3oMbJ+QQ4aggdwa7eUP7EW9trTjX3oEZdpDXkUQkjBhrrfU6xK62b9/uyfNq/VDHaF4do3l1TLjMa315E7d9tJWshGjum96bhOiuuZFFuMxrT9llC3Gf/l9ITMG55nZMTu8O/fnuNq99pXl1nGbWMSG5JllERPZOUU0Ld88rIjk2kjum5HVZQe5OrLXYD17Dvvoc9B+Mc9VvMUk9vY4lImFIJVlEpAuUN7Zxx5xCHAN3TcnD10Nvt/vK+v3YmU9gF3yIGXME5sJrMdExXscSkTCld20RkU5W2xLgjo8KaWh1uX96b7ITo72OFPJsYz3uE7+HtSswx5+JOfln2qAnIl1KJVlEpBM1+13unVdISX0bd0zJpb8v1utIIc+WFbffYrp0B+bCa3HGT/U6koh0AyrJIiKdpC1geXD+NjZUNHPTkb0YmRnvdaSQZzetw/3zfRAI4Fx3F2bwSK8jiUg3oZIsItIJXGt5+PMdfLGjgasOy+LwvESvI4U8d/EC7DN/gp6p7SdYZHXOLbxFRPaESrKIyD6y1vL00lLmb6nlvAPTOXpgiteRQpq1FvvOS9g3/gkDh+Fc+VtMYpLXsUSkm1FJFhHZRy+vruDtL6s4aUhPThvu8zpOSLNtbdjnH8V+NhczbhLm/BmYqCivY4lIN6SSLCKyDz7YUM0/V5QzqW8SFx2codtN7wNbX4v7+AOwfjXmJz/DnHiW5ikinlFJFhHZSwu31vLE4mIOyYlnxuHZOCp0e82WbMd9+G6oLMVccgPOYUd5HUlEujmVZBGRvbCyuIE/frqDQak9uOnIXkQ6Ksh7y64vwH3sATAG54Z7MQOHeR1JREQlWUSkozZVNnP/x9vISYzitkm5xEbqphZ7y104B/v3RyE9E2fG7ZiMbK8jiYgAKskiIh2yvbaVu+YUkhDtcOeUPBJjIryOFJKstdg3Z2LfngWDR+JccQsmPsHrWCIi31JJFhHZQxWNbdwxpxAL3Dk1j9Q4nbqwN2xbK/ZvD2Pz52MmTMP8/ApMpGYpIsFFJVlEZA/UtwS4a04RtS0B7p2WR25SjNeRQpKtq2m/g96mdZhTL8Ace6pOsBCRoKSSLCKyGy1+l3s/LmJbXSu3TcplUGoPryOFJLujsP0Ei5oqnMt/gzlkgteRRER+kEqyiMiP8LuWPyzYxrqyJn59RA6js+O9jhSS7NoVuI8/CFFROL++H9PvAK8jiYj8KJVkEZEf4FrLo5/vYMn2Bi4fm8mEPro18t5wF3yI/efjkNkL55rbMakZXkcSEdktlWQRkR/w3BdlzN1cyzmj0jjugJ5exwk51nWxrz+Pff9VGHYQzmU3YeJ0JV5EQoNKsojI93htdQX/WlvJCQekcNaIVK/jhBzb0oL7zP/BsoWYo47FnHMZJkLH5YlI6FBJFhHZxexN1Ty3vIwj+yRyyZhMnb7QQbamCvfRe+HrjZgzL8ZM+4lmKCIhRyVZROQ7FhXV8edFxYzOjufaw3NwVO46xBZtwX3kHqivxbnyFszocV5HEhHZKyrJIiL/sbqkkYc+2c4AXyw3H9mLqAgV5I6wBUtxn/wDxPbAuelBTJ8BXkcSEdlrKskiIsDmqmbu/biI9Pgobp+US48ox+tIIcWd+y72haegVx+cGbdhfGleRxIR2ScqySLS7RXXtXLnnEJ6RDncNSWPpFi9Ne4p6wawLz+Lnf0mjBqL88sbMbG62YqIhD59EohIt1bV5OeOOYUEXMs903qTHh/ldaSQYZubcP/6R1iRj5l6EubMizCOTrAQkfCgkiwi3VZDa4C75hZS1eTnnmm96Z0c43WkkGGrKnAfvQcKt2B+dhnO5BO8jiQi0qlUkkWkW2oNuNz/cRFbq1u4dVIug9O0RGBP2a2b2k+waGpqX3888hCvI4mIdDqVZBHpdgKu5aFPtlNQ2sT147M5OCfB60ghw67Ix/3LQxCfgHPzg5jcfl5HEhHpEirJItKtWGt5LL+YRUX1XHJIBkf1S/Y6Ukiw1mI/ehP70jPQewDO1bdiUnxexxIR6TIqySLSrfxjRTmzN9Vw5ohUThqikrcnbCCAffEv2HnvwkHjcC6+HhMT63UsEZEupZIsIt3GG2sreWV1BccMTOFno3SO756wTY24T/0BCpZhjvkp5tQLMI7OkBaR8KeSLCLdwtyvanhmWSmH5yVy2dhMjG43vVu2orR9g96OQsx5V+FMPMbrSCIi+41KsoiEvSXb6nn48x2MyozjhgnZRDgqyLtjN29oP+KtrQ3n2jsxw0Z7HUlEZL9SSRaRsLa2rJHfL9hGv54x3HJUL6IitFRgd+zShbjP/C8kpuDccC8mp7fXkURE9juVZBEJW19Xt3DPvCJS4yK5fXIecVG6G9yPsdZiP3gN++pz0H8wzlW/wySleB1LRMQTKskiEpZK6lu5c04h0REOd03JIyVWb3c/xvr92JlPYBd8iBl7JOYX12CidQdCEem+9KkhImGnutnPnXMKaQm43D+tN5kJ0V5HCmq2sR73id/D2hWYE87E/ORnOsFCRLo9lWQRCSuNbQHunltEeaOfu6fk0benzvP9MbasGPfhu6GsGHPhtTjjp3odSUQkKKgki0jYaAu4PDB/G5urmvntxFyGZsR5HSmo2Y1rcR+7HwIBnOvuxgwe4XUkEZGgoZIsImEh4Fr+d+EOVhY3cu3h2YzNTfA6UlBz8+djn/1/4EvDmXE7JquX15FERIKKSrKIhDxrLU8uLmHh1jouOjiDKf2TvY4UtKy12Hdewr7xTxg0DOeK32ISk7yOJSISdFSSRSTkvbCqnA82VnPqMB8nD/V5HSdo2bY27N8fxX4+FzNuEub8GZioKK9jiYgEJZVkEQlp73xZxaxVFUwbkMz5o9O9jhO0bH0t7uMPwPrVmJN/hjnhLN2aW0TkR6gki0jImr+llr8sKeGw3ASuPDRLpe8H2OJtuI/cDZVlmEtuwDnsKK8jiYgEPZVkEQlJy7bX86eF2xmW0YMbJuQQ4aggfx/7ZUH7FWRj2m8xPXCY15FEREKCSrKIhJwvy5v4/YJt9E6J4XdH5RITqRtffB934Rzs3x+F9CycGbdhMrK9jiQiEjJUkkUkpBTWtHDPvCJSYiO5Y3Ie8dERXkcKOtZa7JszsW/PgiGjcC6/GROvI/FERDpCJVlEQkZZQxt3zCkkwsCdU/Lo2UNvYbuyba3YZ/8fdvECzBHTMedegYnUnEREOkrvnCISEmpbAtw5p5CmNpf7pvUmOzHa60hBx9bV4P75Pti0DnPaBZhjTtVmRhGRvaSSLCJBr6nN5Z65hZTUt3HnlDz6+2K9jhR0/IVbcO+/EWqq2pdXHDLe60giIiFNJVlEglpbwPL7BdvYWNnMb47sxYjMOK8jBR37ZQGVj90PkZE4v74f0+8AryOJiIQ8lWQRCVqutTz82Q6+2NHA1YdlMS4v0etIQceuXYH76D1EZORgr74Vk5rhdSQRkbCgkiwiQclay1+XlDD/61rOH53O9IEpXkcKOt8UZNKy8N3zKJV+1+tIIiJhQ4eLikhQem5xIe+sr+bkIT05dZjP6zhBx675AveReyA9G+fG+3BSNCMRkc6kK8kiEnTe31DFX/JLmNQviV8cnKETGnZhC5a1n2KR1Qvn+nswicleRxIRCTsqySISVD7dWssT+SWM79uTGeMycFSQd2ILluL++X7Iyv1PQU7yOpKISFjScgsRCRorihv43093MDitB/ccP4RIRwX5u+yqJe1XkLNzcW5QQRYR6UoqySISFDZWNHP/x9volRjNbZNyiY3S7aa/y65cjPvY/ZDTG+eGezEJKsgiIl1JJVlEPLettpW75xaSFONwx5RcEmJUkL/LrsjHfewB6NW3fYlFvI7CExHpalqTLCKeqmhs4845WwG4c0pvUuOiPE4UXOzyRbhP/B7y+uH86i5MfILXkUREugWVZBHxTH1LgLvmFFHb4nLftN70Sor2OlJQsV98jvvkH6B3f5xf3YmJU0EWEdlftNxCRDzR4ne5Z14R2+pa+d1RvRiYGut1pKBil32G++Tv/1OQ71JBFhHZz1SSRWS/87uWPyzYxpflTVw/IZtRWfFeRwoqdulC3Kf+AH0H4Vx3NyZO8xER2d9UkkVkv3Kt5ZHPd7BkewOXH5rJhN46peG77JJP/v+CfO2dmB5xXkcSEemWtCZZRPYbay1/W1bKvM21nDsqjWMH9fQ6UlBxF3+C/etD0H8wzrV3YGJVkEVEvKKSLCL7zWtrKnljXRUnDO7JGSNSvY4TVNz8+din/xcGDMG55nYVZBERj6kki8h+8e+N1fx9eRkT+yRxySEZGN1u+lvuoo+xT/8fDBqKM+N2TGwPryOJiHR7Kski0uU+L6zjsfxiDsqO55rDs3FUkL/lfj4P+8yf4IDhODNuw8TolA8RkWCgkiwiXaqgpJGHPtnOQF8sN0/sRVSECvI33M/mYp/9fyrIIiJBSCVZRLrMV5XN3PdxEZkJUdw2OY/YSB2o8w134UfYvz0Mg0fiXH0bJibG60giIvIdKski0iV21LVy19xCekQ53Dklj6SYCK8jBQ3304+wzz0MQ0bhXHWrCrKISBBSSRaRTlfV5OfOOYUELNw7JY/0+CivIwWN/4+9Ow+MqrrbOP6cmwRCIAlZgABhDcQVlRgVcIOCiFgUW7FKFRV33KlWQawrQqtYFUFQKVqrrVZrWwtSRdwAF1xQcQsBRAIBzAYhIckk97x/jC82uISBJGcm8/381XFumGd+HZzHm3Pu9Ze+LPvnB6UDDpV3+U0yrSjIABCO9qgkr1y5UvPnz5fv+xo6dKhGjx5d7/mioiLNmjVLFRUV8n1fY8eOVU5OjmprazVnzhytW7dOvu/ruOOO02mnndYkbwRAeNhRU6fbXt2gsqpa3TG0u7olUwL/n//mS8GCfFB/eRMmU5ABIIw1WJJ939e8efM0ZcoUpaWladKkScrNzVVmZuauY5577jkNHDhQw4cPV0FBgaZNm6acnBy9/fbbqq2t1YwZM1RdXa2JEyfq6KOPVseOHZv0TQFwo7rW112vF2jDtmpNGdxN2elcyuz/+W8skn1itnRwTrAgx7VyHQkA8BMa3EWTn5+vjIwMderUSbGxsRo0aJBWrFhR7xhjjCorKyVJlZWVSkn57i5aVVVVqqurU01NjWJjY5WQwAXygZaozreasWyTPtu6U1cP7KL+ndu6jhQ2/NdeDBbkfrkUZACIEA2eSS4pKVFa2nd3xkpLS9Pq1avrHTNmzBjdeeedWrRokaqrq3XzzTdLkgYMGKD33ntPF198sWpqanTuueeqXbt2jfwWALhmrdXsdzfrnYIduji3k47rmeQ6UtjwX10o+9ScYEG+bJJMHOuzASASNMrGvWXLlmnw4MEaNWqU8vLyNHPmTM2YMUP5+fnyPE9z585VRUWFfve736lfv37q1KlTvZ9fvHixFi9eLEmaPn260tPTGyNWyGJjY529diRiXqFpyfN6aOlXWrxmm84/spvOHdijUf7MljCvyoXPqvypOWp1xDFqf/2dTXoGuSXMqzkxr9Awr9Axs9CE47waLMmpqakqLi7e9bi4uFipqan1jlmyZIkmT54sScrOzlYgEFB5ebmWLl2qww47TLGxsUpOTtZ+++2nNWvWfK8kDxs2TMOGDdv1uKioaJ/e1N5KT0939tqRiHmFpqXO65+fF+svH3yjEX3b69Q+CY32HiN9Xv4rL8j+7RHpsKNUO/5aFW/b3qSvF+nzam7MKzTMK3TMLDSu5tWlS5cffa7BNclZWVkqLCzU1q1bVVtbq+XLlys3N7feMenp6Vq1apUkqaCgQIFAQElJSfX+eVVVlVavXq2uXbvuy3sBEEaWrN2m+R98o0HdE3VxbicZbjctSfIX/+vbgjxA3iW/lYlliQUARJoGzyTHxMRo/PjxPqtjHQAAIABJREFUmjp1qnzf15AhQ9StWzc9/fTTysrKUm5ursaNG6e5c+dqwYIFkqQJEybIGKMRI0Zo9uzZmjhxoqy1GjJkiHr0aJxfxQJw672NOzTz7UIdkpGgiYM6K8ajIEuS//K/ZJ+ZJ+UMlHfR9TKxXI4eACKRsdZa1yF2t2nTJievy69GQsO8QtOS5vX51kr9bskGdUturTuHdVNCXOPfTS8S5+W/9Lzs3+dLOYPkXXRdsxbkSJyXS8wrNMwrdMwsNOG43IJTHABC8lVple54vUDpCXG6ZUhmkxTkSOT/9x+yzz4mc/jRMhf+hjPIABDh+Lc4gD22ZUeNbn21QK1jPN36s0wlx/OvEEnyX3xO9h+PyxxxrMwFE2Vi+A8HAIh0fMMB2CNlVbW6dckGBep83XVCD3Vqxw0xJMlf+HfZ55+gIANAC9Pg1S0AoDJQp9tf3aCiylpNGZypHu1bu44UFvwFzwQL8pHHU5ABoIXhTDKAn1RT5+uu1zdqXWm1bjo+Uwd04NbykuT/52+y/3pKZsAQmfOvkvEoyADQklCSAfyoOt/q3mWF+mRLpa4Z2Fm5XbmtvCT5//6r7At/lRk4ROY8CjIAtESUZAA/yFqruSu26K0N5Rqf01FDeie7juSctVb233+V/c/fZAYNlTn3CgoyALRQlGQAP+ipj4v03/wynX5Qmk49ILXhH2jhggX5Kdn/PC1z9DCZcVfIeGzrAICWipIM4Hte+KJEz6wq1rCsZJ19aLrrOM5Za2X/+aTswmdkjh0uc/YECjIAtHCUZAD1vL5umx59f6uOymynCUdmyJjovt20tVb2+SdkX3yWggwAUYSSDGCXDzbt0P1vFergjm103TFdFONRkO0//iy76DmZ40bI/PpSCjIARAlKMgBJ0pdFOzX9jY3q3r61Jh+fqVYx0V0GrbWyzz0m+9/nZQafJHPWJRRkAIgilGQA+npbte54dYNS2sTq1iHd1LZVdF+xwVor++x82Zf+KTNkZLAgR/myEwCINpRkIMp9UxHQrUs2KNYzuu1n3dS+TXT/a8FaK/vMn2QX/0vmZz+XOfMiCjIARKHo/jYEotz2qlrdumSDqgK+pp7QXRmJrVxHcspaK/v0o7KvvCAzdJTMry6kIANAlGKBHRCldgZ83f5agbZWBHTT4Ez1Sol3Hckpa63s3x4JFuRhp1CQASDKUZKBKBSos5r+5katKanSdcd00UEdE1xHcspaK/vUXNkl/5E54VSZMy6gIANAlGO5BRBlfGt131ubtLKwQlcOyNBRmYmuIzllfV/2r3NlX3tRZvhpMqefR0EGAFCSgWhirdWj723R0vXlOvewDhqW1d51JKes78s+OUf2jUUyJ/5C5pfnUpABAJIoyUBUeWZVsRbklWn0Aan6xUFpruM4ZX1f9i+zZd98SeakX8qcNo6CDADYhZIMRIkX80r11MdF+lnvJJ3bv4PrOE5Z35d9Ypbs0pdlRo6RGX02BRkAUA8lGYgCy9Zv19wVW3RE17a6/KjO8qK4EFrfl/3zTNllr8icfIbMqb+mIAMAvoeSDLRwKwsrdO/yTTqgQxtdf0xXxXrRWwitXyf7+IOyy1+R+fmZMqecRUEGAPwgSjLQgq0u3qlpb2xU16TWuun4TLWOjd6rPlq/TvaxB2TfelVm1FnyTjnLdSQAQBijJAMtVMH2at3+aoGSWsfoliGZatc6xnUkZ6xfJzv/ftm3X5M5Zay8UWe6jgQACHOUZKAFKq4M6NZXNshIuu1n3ZSWEOc6kjO2rk72T/fJvvu6zKm/lvfzX7mOBACIAJRkoIUpr67TrUs2aEeNr6kndFeXpFauIzkTLMh/lH33DZnRZ8s7+QzXkQAAESJ6FygCLVB1ra87XivQpvKAJh/fVVmp8a4jOWPr6mTn3RssyL84l4IMAAgJZ5KBFqLWt/r9mxu1uninfntMVx2S0dZ1JGdsba3sozNk318mc/p58k78hetIAIAIQ0kGWgDfWs18q1Dvb6rQhCMzNLB7outIztjaWvmP3iO9v1xmzPnyhp/mOhIAIAJRkoEIZ63Vnz7Yqte+2q5fH5quE/u2dx3JGVtbK/+Ru6UP3pI54wJ5J5zqOhIAIEJRkoEI99xnJXrhi1KN2i9FYw5Kcx3HGVsbkP/w3dKHb8v86kJ5w05xHQkAEMEoyUAEeym/TE+s/EbH9UzS+MM7Ru3d42xtQP7cP0gr35E582J5Q3/uOhIAIMJRkoEI9daGcj307mbldG6rqwZ0lhetBTkQkD/399JH78qMvUTekJNdRwIAtACUZCACfbKlQjOWblLftHjdcFxXxcVEcUGeM136eIXM2EvlDRnpOhIAoIWgJAMRZm1Jlaa+tlEZiXG6eXA3xcdG5+XObaBG/kPTpU/ek/n1ZfIGn+Q6EgCgBaEkAxGksLxGt766QW1bebr1Z92U2DrGdSQnbKBG/uy7pFUfyJwzQd5xI1xHAgC0MJRkIEKU7KzVLUs2yLfSbT/rpvSEONeRnLA11fJn3SV99qHMuCvkHTvcdSQAQAsUnb+nBSLMjpo63bZkg7ZV1ep3gzOVmdzadSQnggV5qvT5Splzr6QgAwCaDCUZCHPVtb6mvlaggu3VmnRcprLT27iO5IStrpb/4J3S5x8FC/IxJ7iOBABowSjJQBir863uWbZJn3+zU9cO6qLDOrd1HcmJYEG+Q/riY5nzrpJ39DDXkQAALRxrkoEwZa3Vg+9s1rsFO3TJEZ10TI8k15GcsNVV8mfeIeWtkjn/GnkDh7iOBACIApRkIEz9eeU3WrJ2m87sl6aR2Smu4zhhq6vkP3C7tPozmfHXyBtAQQYANA9KMhCGnv+sWP/4rEQn9W2vM/ulu47jhL+zUv4Dt0mrP5e54Fp5Rx3vOhIAIIpQkoEws2TtNj324Tc6unuiLsrtJBOFt5u2VZUqu/fmYEG+cKK8I49zHQkAEGUoyUAYebegXDPfLtShGQm6dlBnxXjRWZD9+2+TvzZP5qLfyDviWNeRAABRiJIMhInPtlbq7qWblJUarxuP66q4mOi7+IzdWSn//luldXlKnni7dux3iOtIAIAoFX3fwkAY+qq0Sne+VqAObeN08+BMJcRF3+2mbWWF/Ptukb5aLe/i3yr+6J+5jgQAiGKcSQYc27KjRre+WqD4WE+3Dumm5Pjo+2u5qyB/vUbexdfL5AxyHQkAEOWi79sYCCNlO2t1y5INCtT5mnZCD3VsF+c6UrOzlTvk33drsCBfcoNM/wGuIwEAQEkGXKkM1Om2VzeopLJWtw/tru7tW7uO1OxsxQ75f/ydVPCVvEtvkDmMggwACA+sSQYcqKnzddfrG7W+rFo3HNtV+3do4zpSs6tXkC+7kYIMAAgrlGSgmdX5Vvcu26RPtlTqqoGddXjXdq4jNTtbUS7/3puljV/JmzBJ5tAjXUcCAKAellsAzchaqzkrNuutDTt04eEdNbhXsutIzc7u2B4syIUF8ibcJNPvcNeRAAD4Hkoy0Iye/KhIL+Vv0+kHpWnU/qmu4zQ7W/5tQd5cIO/yyTIHU5ABAOGJkgw0k2c+3Ki/f1qs4X2Sdfah6a7jNLtgQZ4ibdkk74opMgf1dx0JAIAfRUkGmlhloE4v5pXpzyu/0cBu7XTpERkyJrpuN23Lt8mfMUXaWijviptkDqQgAwDCGyUZaCJfb6vWi3mlWrJ2u6pqfQ3okaKJAzsoxouygry9LLjE4ptCeVfeLHPAoa4jAQDQIEoy0IjqfKt3Csq1MK9Mn2ypVKxndEyPRI3MTtGg/TJVXFzsOmKzsttL5d8zRSreIu8KCjIAIHJQkoFGULazVi/ll2lRfpmKK2vVISFW5xzaQSf0Sd51m+moW2KxrTS4xKJ4q7yrbpHZr5/rSAAA7DFKMrCXrLX6ominFn5ZpuUbtqvWlw7LSNAluZ2U27Vd1C2r+F+2rCRYkEu++bYgH+w6EgAAIaEkAyGqrvX1+lfbtTCvVOtKq5UQ52lE3xSdlN1emUnRd2vp3dmy4uASi7JieVffIpNNQQYARB5KMrCHCstr9GJeqRav3aaKGl89klvrsiM76fieyWoTx80rJcmWFsu/5yZpW6m8q2+V6Xug60gAAOwVSjLwE+p8qw8LK7Tgy1J9UFihGCMN6Jaok7NTdGDHNlG3zvin2JIi+TNukraVybvmFpk+FGQAQOSiJAM/YHt1nRavKdOi1WXasiOglDaxOrNfmob3aa+0hDjX8cKOLfkmeAa5fJu8a26V6XOA60gAAOwTSjLwP/KLq7Qwr1Rvrt+umjqrgzq20bjDOmhAt0TFRvFGvJ9ii78JnkHesV3eNbfJZO3vOhIAAPuMkoyoF6jztXR9uRbmlSqvuEqtY4yG9ErWyOz26pkS7zpeWLPFW4NnkCvKgwW5936uIwEA0CgoyYha31QEtGh1mV7OL9O26jp1SWylCw/vqJ/1TlbbVjGu44U9W7QlWJArK+Rde7tMr2zXkQAAaDSUZEQVa60+2lyphXmlWrFxhyTpiK7tNDI7RYdkJMhjI94e2VWQd1bIm3i7TM++riMBANCoKMmIChU1dVqydpteXF2mjdtrlNQ6RqcdkKoRfVPUsR0b8UJhv9kcLMhVO+VNvEOmRx/XkQAAaHSUZLRo68uqtTCvVK+t26aqWqu+afG6ZmBnHd0jUa1iuLZxqIIFebJUVfVtQc5yHQkAgCZBSUaLU+tbvb0huBHv0607FecZHdszSSOz26tvWhvX8SKW3VoYPINcUy3vN3fIdKcgAwBaLkoyWoySnbV6aXWZ/ptfppKdterULk7n9u+gYVntldSajXj7wm7dJP/um6TaGnm/uVOmWy/XkQAAaFKUZEQ0a60+27pTC/JK9faGctVZKadzW004MkM5Xdoqhmsb7zO7ZVNwiUVtbbAgZ1KQAQAt3x6V5JUrV2r+/PnyfV9Dhw7V6NGj6z1fVFSkWbNmqaKiQr7va+zYscrJyZEkrV+/Xg8//LB27twpY4ymTZumVq1aNf47QVTZGfD1+lfbtDCvTOvLqtW2laef75eik7JT1DmRz1djsZsL5N8zRar7/4Lc03UkAACaRYMl2fd9zZs3T1OmTFFaWpomTZqk3NxcZWZm7jrmueee08CBAzV8+HAVFBRo2rRpysnJUV1dnWbOnKkrrrhCPXv2VHl5uWJjOXmNvVewvVov5pVpydptqgz46pXSWpcflaHjeyapdSwb8RqTLSyQP2OK5NfJu26qTNceriMBANBsGmys+fn5ysjIUKdOnSRJgwYN0ooVK+qVZGOMKisrJUmVlZVKSUmRJH300Ufq3r27evbsKUlKTExs7PyIAnW+1Xsbd2hhXqlWbq5UrCcN6hbciLd/hzYyXNu40dnCDd8WZF/eb6bKdO3uOhIAAM2qwZJcUlKitLS0XY/T0tK0evXqeseMGTNGd955pxYtWqTq6mrdfPPNkqTCwkIZYzR16lRt375dgwYN0qmnnvq911i8eLEWL14sSZo+fbrS09P36U3trdjYWGevHYmael6llQH959PNev6TzdpSXq0O7VrpooHdNeqgDKW1jbwlFZHy+ardsE6l994szxilTH1Isd16OskRKfMKF8wrNMwrNMwrdMwsNOE4r0ZZ+7Bs2TINHjxYo0aNUl5enmbOnKkZM2aorq5OX3zxhaZNm6bWrVvr9ttvV+/evdWvX796Pz9s2DANGzZs1+OioqLGiBWy9PR0Z68diZpqXnlFwY14S9eXq9a36tcpQecd1kVHZSYqxjOyO7eraGejv2yTi4TPl934tfwZN0meJ+83U1XWpp3E38eIwLxCw7xCw7xCx8xC42peXbp0+dHnGizJqampKi4u3vW4uLhYqamp9Y5ZsmSJJk+eLEnKzs5WIBBQeXm50tLSdMABBygpKUmS1L9/f61bt+57JRmorvW1dP12LcwrU35JleJjPZ2QlayR2Snq3r6163hRwRZ8Jf/emyUvRt51d8pkZDb8QwAAtFAN7nTKyspSYWGhtm7dqtraWi1fvly5ubn1jklPT9eqVaskSQUFBQoEAkpKStKhhx6qDRs2qLq6WnV1dfr888/rrWUGtuyo0eMfbtUFz+frgbc3q6rW18W5nTT/F1m69MgMCnIzsQXrgmuQY2KCm/QoyACAKNfgmeSYmBiNHz9eU6dOle/7GjJkiLp166ann35aWVlZys3N1bhx4zR37lwtWLBAkjRhwgQZY9SuXTudfPLJmjRpkowx6t+//65LwyF6+dZqZWGFFuaV6r2NFTJGOiqznUZmp6hfpwQ24jUzu2Gd/HunSLGtggW504//6gkAgGhhrLXWdYjdbdq0ycnrsn4oNKHOa0d1nV5Zu00vri5VYXlAyfExOrFPe53Yt73SE+KaMGl4CMfPl/16bXCJRavWwSUWHcOnIIfjvMIZ8woN8woN8wodMwtNRK5JBvbV2pIqLcwr1etfbVdNndX+6W10Vr90DeqeqLgYrm3sil2/JliQ4+ODl3nr2Nl1JAAAwgYlGU0iUGf11oZyLcwr1eff7FSrGKPjeibp5OwU9U6Ndx0v6tn1+fLv/Z0U3ya4xKJDhutIAACEFUoyGlVxZUCLVpfppfwylVXVKaNdnMbndNTQ3slq1zrGdTxIsl+tlv/H30lt2gZvNU1BBgDgeyjJ2GfWWn2ypVIL88r0TkG5rJUO79JWI7NT1L9LW3lsxAsbdt23BTmhbfAMcnon15EAAAhLlGTstYqaWi3MK9XCvFJt2FajxFaeTtk/VSf1ba+MxMi7I15LZ9d+Kf++W6S2icGCnNbRdSQAAMIWJRkh27CtWgvzSvXaV6tVWVOnrNTWunJAho7tkaTWsWzEC0d2zRfy779VapcU3KSX1sF1JAAAwholGXukzrd6t2CHFuaV6uMtlYr1jIZmp2tojwRlp8VzbeMwZtd8ETyDnJgcPIOcSkEGAKAhlGT8pLKdtXopv0yL8stUXFmrDgmxOufQDhrWJ1l9MjO4BmSYs/mfyb/vNim5ffAMcmq660gAAEQESjK+x1qrL4p2amFemZZ/vV21vnRoRoIuye2k3K7tFONx1jgS2NWfyb//Nik5JXgGOSXNdSQAACIGJRm7VNf6euOr7VqQV6p1pdVKiPM0om+KTspur8yk1q7jIQQ271P5D9wmtU8L3kmvPQUZAIBQUJKhwvIavZhXqlfWbtOOGl89klvr0iM6aXCvZLWJYyNepLFfrpI/83YpJS24xKJ9qutIAABEHEpylKrzrT4srNDCvFK9v6lCMUYa0C1RJ2en6MCObdiIF6Hsl5/If+B2KbVDcIlFcorrSAAARCRKcpTZXl2nxWvKtGh1mbbsCCglPkZn9kvT8D7tlZYQ5zoe9oH9/CP5D94hpXUKLrFIoiADALC3KMlRIr+4SgvzSvXm+u2qqbM6sEMbnXNoBw3olqi4GM4aR7pdBTk9Q95v7qAgAwCwjyjJLVigzteyr8u1MK9UXxZVqXWM0ZBeyRqZ3V49U+Jdx0MjsZ+tlP/gnVLHzvIm3iGT1N51JAAAIh4luQX6piKgRavL9HJ+mbZV16lLYitdeHhHDemdrHatYlzHQyOyn34of9bUYEH+zZ0yicmuIwEA0CJQklsIa60+2lyphXmlWrFxhyQpt2s7nZydokMyEuSxEa/Fsas+CBbkjK7yJt4pk5jkOhIAAC0GJTnCVdTU6dV127Qwr0wbt9coqXWMTjsgVSP6pqhjOzbitVT2k/flz75L6pwZXGLRjoIMAEBjoiRHqPVl1VqYV6rX1m1TVa1V37R4XT2ws47pkahWMVzbuCWzH6+Q/9A0qUv3YEFum+g6EgAALQ4lOYLU+lbvbAhuxFu1dafiPKNjeyZqZHaK+qa1cR0PzcB+9K78h6ZLmT3lXXsbBRkAgCZCSY4AJTtr9dLqMv03v0wlO2vVsW2czj2sg4ZlJSspnv8Lo4Vd+Y78Ob//tiDfLtO2netIAAC0WDSsMGWt1Wff7NTCvFK99XW56qzUv3NbXXZkJx3epZ1iPDbiRRP74dvy5/5B6tYreAY5gYIMAEBToiSHmZ0BX69/tU0v5pXpq7JqtY3zdPJ+KTqpb4q6JLVyHQ8O2A/ekv/wH6TuWfKuuU0moa3rSAAAtHiU5DCxcXuNXswr1ZK121QR8NUrpbUuPypDx/VMUnwsG/GilX1/ufxH7pZ69JF39a0UZAAAmgkl2aE63+q9TTu08MtSrdxcqRgjDeqeqJOzU7R/hzYyXNs4qtn3lsp/5B6pV3awILdJcB0JAICoQUl2YFtVrV5es03/XV2qrRW1SmsTq7GHpGt4n/ZKacP/JZD8FUtlH71H6r2fvKtvkYmnIAMA0JxoZM0oryi4EW/p+nIFfKuDOyXovJyOOiozUbFsxMO3/HffkJ13r9R7f3lX/46CDACAA5TkJlZd62vp+u1amFem/JIqxcd6GpaVrJHZKerevrXreAgz/juvy877o9Rnf3lX3SITz/WvAQBwgZLcRLbsqNGi1WV6ec02lVfXKTOplS7O7aQhvZOUEBfjOh7CkP/2a7J/uk/qe6C8K2+mIAMA4BAluRH51mplYYUW5pXqvY0VMkY6KrOdRmanqF+nBDbi4Uf5b70qO/9+KfugYEFuHe86EgAAUY2S3Ah2VNfplbXb9OLqUhWWB5QcH6PTD0rTiX3bq0PbONfxEOb85a/IPvaAtF8/eVfcLNOaZTgAALhGSd4H60qrtDCvVK+v267qOqv90tvorH7pGtQ9UXExXNsYDfOXvSL7+APS/ofIu3wKBRkAgDBBSQ5RoM7qrQ3lWphXqs+/2alWMUbH9UzSydkp6p3Kr8ix5/ylL8v++UHpgEPlXX6TTCsKMgAA4YKSvIeKKwNatLpML+WXqayqThnt4nR+TgcN7d1eia3ZiIfQ+G++FCzIB/aXd/lkCjIAAGGGkvwTrLVatbVSC/PK9PaGclkrHd6lrUZmp6h/l7by2IiHveC/sUj2idnSwTnyJkyWiWvlOhIAANgNJfkHVAbq9Nq67Xoxr1Rfb6tRu1aeTtk/VSf1ba+MRAoN9p7/+iLZv8yWDj5c3oRJFGQAAMIUJfl/fFVSqadWbNaStdu1s9ZXVmprXTkgQ8f2SFLrWDbiYd9ULvpHsCD3y5V32SSZOK58AgBAuKIkS6rzrW5/dYNWbq5UrGd0TPdEjdwvRdlp8VzbGI3Cf2ORyp+YLR1yhLxLb6QgAwAQ5ijJkmI8o86JrXRkrw46ukuc2sczFjQe++Uq2SfnqFXOQNVeeB0FGQCACEAb/NalR2YoPT1dRUVFrqOgBbEl38if+3upY2clT7xNJTurXEcCAAB7gIW2QBOxgRr5D02XAjXyJtwkr20715EAAMAeoiQDTcBaK/vkQ9JXq+WNv1amc6brSAAAIASUZKAJ2NdflF32iszPfyXTf4DrOAAAIESUZKCR2dWfyf7tEalfrsyos1zHAQAAe4GSDDQiW1osf850Ka2jvAsnynj8FQMAIBLxDQ40EhsIBAtydVXwdtMJbNQDACBSUZKBRmL/9rC09kt5518t07WH6zgAAGAfUJKBRuC/sUj2jf/KnPRLmcOPdh0HAADsI0oysI/smi9kn3pYOrC/zOizXccBAACNgJIM7AO7rTS4DjklTd7F18l4Ma4jAQCARkBJBvaSrQ3In/N7qbJC3uWTZdomuo4EAAAaCSUZ2Ev2mXlS/mcy514pk9nLdRwAANCIKMnAXvCXLZZ9daHM8NPkHXmc6zgAAKCRUZKBENl1q2X/8pB0wKEyvxjnOg4AAGgClGQgBHZ7mfyHpknJKfIuul4mho16AAC0RJRkYA/Z2lr5c/8g7dgu77JJMolJriMBAIAmQkkG9pB9dr6Ut0rmnMtlemS5jgMAAJoQJRnYA/5br8q+8oLM0FHyBg5xHQcAADQxSjLQALt+jewTs6Tsg2VOP991HAAA0AwoycBPsOXbgxv12iXJu+S3MrGxriMBAIBmQEkGfoStq5P/yN3SttLgRr2k9q4jAQCAZkJJBn6E/cefpc8/kjl7gkyvvq7jAACAZkRJBn6A/+4bsi89LzNkpLyjh7qOAwAAmhklGdiN3bBO9vEHpD4Hypxxges4AADAAUoy8D9sRbn82XdJCe3kXXqDTGyc60gAAMABSjLwLevXyX/4Hqm0WN6lN8okp7iOBAAAHKEkA9+y//yL9NmHMmMvkcna33UcAADgECUZkGTfWyr74nMyx50o77gTXccBAACOUZIR9ezG9fIfe0DqvZ/MmRe7jgMAAMIAJRlRzVbuCG7Ui28j77IbZeLYqAcAACjJiGLW9+U/eq9UvDV4JYv2aa4jAQCAMEFJRtSyL/xV+uQ9mTMvkulzoOs4AAAgjFCSEZXsh2/L/udpmaOHyRx/kus4AAAgzMTuyUErV67U/Pnz5fu+hg4dqtGjR9d7vqioSLNmzVJFRYV839fYsWOVk5NT7/lrr71WY8aM0SmnnNK47wAIkS3cIP9Pf5R69pX59aUyxriOBAAAwkyDJdn3fc2bN09TpkxRWlqaJk2apNzcXGVmZu465rnnntPAgQM1fPhwFRQUaNq0afVK8uOPP67+/fs3zTsAQmArK4Ib9eJafbtRr5XrSAAAIAw1uNwiPz9fGRkZ6tSpk2JjYzVo0CCtWLGi3jHGGFVWVkqSKisrlZLy3Z3K3n33XXXs2LFeqQZcsL4fPIO8tVDeJTfIpHZwHQkAAISpBktySUmJ0tK+2/WflpamkpKSeseMGTNGb775pi699FJNmzZN48ePlyRVVVXpX//6l8aMGdPIsYHQ2QXPSB+9K3PGBTL7Hew6DgAACGN7tCa5IcuWLdPgwYM1atQo5eXlaebMmZoxY4aeeeYZnXzyyYqPj//Jn1+8eLEWL14sSZo+fbrS09MbI1bIYmNjnb12JIqkeVWvWKayfz+l+MGsl+4SAAAbuUlEQVQjlHTGeU7WIUfSvMIB8woN8woN8woN8wodMwtNOM6rwZKcmpqq4uLiXY+Li4uVmppa75glS5Zo8uTJkqTs7GwFAgGVl5crPz9f77zzjp588klVVFTIGKNWrVppxIgR9X5+2LBhGjZs2K7HRUVF+/Sm9lZ6erqz145EkTIvu3mj/D/eInXvrZoxF9T7PDenSJlXuGBeoWFeoWFeoWFeoWNmoXE1ry5duvzocw2W5KysLBUWFmrr1q1KTU3V8uXLddVVV9U7Jj09XatWrdLgwYNVUFCgQCCgpKQk3X777buOeeaZZxQfH/+9ggw0JVtVGdyoFxMjb8JkmVatXUcCAAARoMGSHBMTo/Hjx2vq1KnyfV9DhgxRt27d9PTTTysrK0u5ubkaN26c5s6dqwULFkiSJkyYwGW14Jy1Vv78+6XNG+Vde5tMWkfXkQAAQITYozXJOTk59S7pJkm/+tWvdv3vzMxM3XHHHT/5Z5xxxhl7EQ/Ye/bFZ6UP3pIZM17mgENdxwEAABGEO+6hRbKr3pf9519kjjxO5oRTXccBAAARhpKMFsduLZT/yD1S154y465k6Q8AAAgZJRktiq3aGdyoZzx5EybJtGajHgAACB0lGS2GtVb28ZnSpg3yLr5OpkOG60gAACBCUZLRYtiXnpd9b6nMaefIHNjfdRwAABDBKMloEexnH8o+92fp8EEyI37hOg4AAIhwlGREPPvNZvkP3yN1zpR33tVs1AMAAPuMkoyIZqur5T80TbK+vMsny8S3cR0JAAC0AJRkRCxrrewTD0oFX8m78DcyHX/8/usAAAChoCQjYtlX/i37zusyp/5apl+u6zgAAKAFoSQjItkvPpb9+3yp/wCZk053HQcAALQwlGREHFv8jfy5f5A6dZU3/hoZj48xAABoXLQLRBRb8+1Gvbra4B314hNcRwIAAC0QJRkRw1or+5fZ0vp8eRdMlMnIdB0JAAC0UJRkRAz76gLZt16VGXWWzKFHuo4DAABaMEoyIoLNWyX7zDzp0CNlfv4r13EAAEALR0lG2LMlRfLn/F5Kz5A3/lo26gEAgCZH20BYs4GA/DnTpZqa4Ea9hLauIwEAgChASUbYstbKPjVHWpcnb/zVMl26u44EAACiBCUZYcu+8V/ZpS/LjDxDJmeQ6zgAACCKUJIRlmz+57J/fVg6+HCZU89yHQcAAEQZSjLCji0rDq5DTk2Xd+FvZLwY15EAAECUoSQjrNjaQPBKFlU75V1+k0zbdq4jAQCAKERJRlixf3tEWvOFvPOukunaw3UcAAAQpSjJCBv+my/Jvr5IZsQvZXKPcR0HAABEMUoywoJd+2Xwcm8HHiZz2tmu4wAAgChHSYZzdnup/IemS8mp8i66jo16AADAOUoynLK1tcGNepXl8iZMlmmX5DoSAAAAJRlu2b//SVr9mcy4K2W693YdBwAAQBIlGQ75y1+RXfIfmRNOlXfU8a7jAAAA7EJJhhN2fb7sE7Ol/Q+R+eV5ruMAAADUQ0lGs7Pl2+TPvktKai/v4utlYtioBwAAwgslGc3K1tXJn/sHqXx7cKNeYrLrSAAAAN9DSUazss8+Jn35icw5l8v0yHIdBwAA4AdRktFs/Ldfk138L5mho+QNHOI6DgAAwI+iJKNZ2K/XyD7xoJR9kMzp57uOAwAA8JMoyWhydsd2+bOnSQmJ8i75rUxsrOtIAAAAP4mSjCZl6+rkP3y3tK1E3oRJMkkpriMBAAA0iJKMJmWff0L6/COZX18m0yvbdRwAAIA9QklGk/FXLJX97z9kBp8k75gTXMcBAADYY5RkNAlb8JXsY/dLWfvL/OpC13EAAABCQklGo7MV5cE76rVpK+/SG2Vi41xHAgAACAklGY3K+nXyH50hlRTJu+xGmfapriMBAACEjJKMRmX/9ZS06gOZsRfLZO3vOg4AAMBeoSSj0dj3l8su/LvMscPlHTfCdRwAAIC9RklGo7Abv5Y//z6pV7bMWZe4jgMAALBPKMnYZ7ZyR3CjXut4eZdNkoljox4AAIhslGTsE+v78uf9USreErySRUqa60gAAAD7jJKMfWL/8zfp4xUyv7pQpu+BruMAAAA0Ckoy9lrVu2/KvvA3mUFDZQaPdB0HAACg0VCSsVdsYYG233eb1KOPzNmXyRjjOhIAAECjoSQjZHZnZXCjXlwreRMmycS1ch0JAACgUVGSERLr+/L/dJ+0dZPaX3+nTGoH15EAAAAaHSUZIbEL/y6tfFtmzHi1OjjHdRwAAIAmQUnGHrMfr5D991MyAwbLDB3lOg4AAECToSRjj9gtm+Q/eq/UrZfM2ZezUQ8AALRolGQ0yFZVyp81VYrxgnfUa93adSQAAIAmRUnGT7LWyn/sAWnzRnkXXS+T3sl1JAAAgCZHScZPsov+Ib2/XOaX58oceJjrOAAAAM2CkowfZVd9IPv8n2WOOFZm+GjXcQAAAJoNJRk/yH6zWf4j90hdusuceyUb9QAAQFShJON7bHVV8I56krwJk2VaxztOBAAA0LwoyajHWiv7+Exp43p5F10n07Gz60gAAADNjpKMeuzL/5Rd8abMaefIcEc9AAAQpSjJ2MV+tlL22celwwfJjPil6zgAAADOUJIhSbJFW+Q/crfUOVPeeVezUQ8AAEQ1SjJkq6uDG/Xq/OBGvfg2riMBAAA4RUmOctZa2b/Mkgq+knfhRJlOXVxHAgAAcI6SHOXskv/Ivv2azClnyRxyhOs4AAAAYYGSHMXsl6tkn5knHXaUzMgzXMcBAAAIG5TkKGVLvpE/9/dSx87yxl8r4/FRAAAA+H80oyhkAzXyZ0+TAjXyJtwk0ybBdSQAAICwQkmOMsGNeg9J6/PlXXCtTOdM15EAAADCDiU5ytjXXpRd/orMz8+UOWyA6zgAAABhKXZPDlq5cqXmz58v3/c1dOhQjR49ut7zRUVFmjVrlioqKuT7vsaOHaucnBx9/PHHevLJJ1VbW6vY2Fidc845Ovjgg5vkjaBhdvVnsk8/Ih1yhMyoM13HAQAACFsNlmTf9zVv3jxNmTJFaWlpmjRpknJzc5WZ+d2v6Z977jkNHDhQw4cPV0FBgaZNm6acnBwlJibqhhtuUGpqqr7++mtNnTpVc+fObdI3hB9mS4vlz5kupXUKLrNgox4AAMCParAp5efnKyMjQ506dVJsbKwGDRqkFStW1DvGGKPKykpJUmVlpVJSUiRJvXr1UmpqqiSpW7duqqmpUSAQaOz3gAbYQED+Q9Ok6mp5l0+WSWjnOhIAAEBYa/BMcklJidLS0nY9TktL0+rVq+sdM2bMGN15551atGiRqqurdfPNN3/vz3nnnXfUu3dvxcXFfe+5xYsXa/HixZKk6dOnKz09PeQ30hhiY2OdvXZT2j57unauy1Pyb+9S/CE5jfbnttR5NRXmFRrmFRrmFRrmFRrmFTpmFppwnNcerUluyLJlyzR48GCNGjVKeXl5mjlzpmbMmCHv21/pb9iwQU8++aRuuummH/z5YcOGadiwYbseFxUVNUaskKWnpzt77abiv7FI9uV/y4wcox19D9aORnx/LXFeTYl5hYZ5hYZ5hYZ5hYZ5hY6ZhcbVvLp06fKjzzW43CI1NVXFxcW7HhcXF+9aQvH/lixZooEDB0qSsrOzFQgEVF5evuv4e+65R5dffrkyMjL26g1g79g1X8g+9bB0UH+ZU8e6jgMAABAxGizJWVlZKiws1NatW1VbW6vly5crNze33jHp6elatWqVJKmgoECBQEBJSUmqqKjQ9OnTNXbsWO2///5N8w7wg2xZifyHpkup6fIuuk7Gi3EdCQAAIGI0uNwiJiZG48eP19SpU+X7voYMGaJu3brp6aefVlZWlnJzczVu3DjNnTtXCxYskCRNmDBBxhgtWrRImzdv1rPPPqtnn31WkjRlyhQlJyc37buKcrY2ELzl9M4KedfcItM20XUkAACAiGKstdZ1iN1t2rTJyeu2lPVD/pNzZF9bKHPx9fKOOLbJXqelzKu5MK/QMK/QMK/QMK/QMK/QMbPQROSaZEQWf+nLwYJ84mlNWpABAABaMkpyC2LX5ck++ZB0wKEyp41zHQcAACBiUZJbCLu9VP7saVJyqryLr5eJYaMeAADA3qIktwC2tlb+3D9IleXyJkyWaZfkOhIAAEBEoyS3APbZ+VLepzLjrpTp3tt1HAAAgIhHSY5w/luvyr7ygsywU+UddbzrOAAAAC0CJTmC2fVrZJ+YJe3XT+b081zHAQAAaDEoyRHKlm+XP/suKTFJ3iW/ZaMeAABAI6IkRyBbVyf/4T9I28vkXTZJJpE7GAIAADQmSnIEsv94XPriY5mzJ8j07Os6DgAAQItDSY4w/juvy770T5khJ8s7eqjrOAAAAC0SJTmC2A3rZP88U+p7oMwZF7iOAwAA0GJRkiOE3fHtRr2ERHmX3iATG+s6EgAAQItFSY4A1q+T/8g9UlmxvMtulElKcR0JAACgRaMkRwD7/F+kz1bKjL1Upvd+ruMAAAC0eJTkMGffWyq76DmZ40fIO3a46zgAAABRgZIcxuzG9fIfe0DK2l/mzItcxwEAAIgalOQwZSt2yJ81VYpv8+1GvTjXkQAAAKIGJTkMWb9O/qMzpJKiYEFun+Y6EgAAQFShJIch+++/SqvelznzIpk+B7qOAwAAEHUoyWHGfvCW7IJnZI45Qeb4Ea7jAAAARCVKchixhRvk/+k+qVe2zNhLZIxxHQkAACAqUZLDhK2skD/rLqlVK3mX3igT18p1JAAAgKhFSQ4D1vfl/+mPUtHm4Ea91HTXkQAAAKIaJTkM2P88LX30rswZF8hkH+w6DgAAQNSjJDtmP1oh+8JfZQb+TGbIya7jAAAAQJRkp+zmjfLnzZB69JE5+zI26gEAAIQJSrIjtqpS/uy7pJhYeZdNkmnV2nUkAAAAfIuS7EBwo9590paN8i75rUxaB9eRAAAA8D8oyQ7YF5+VPnxb5vTzZfY/xHUcAAAA7IaS3MzsJ+/L/utJmSOPlxl2ius4AAAA+AGU5GZkt26S/+g9UteeMuOuYKMeAABAmKIkNxNbtVP+7GmS8eRNmCTTmo16AAAA4YqS3AystbKPPSBt2iDv4utkOmS4jgQAAICfQEluBva//5B9f5nML8fJHNjfdRwAAAA0gJLcxOxnH8r+4wmZ3GNkhp/mOg4AAAD2ACW5CdlvNst/+B6pSzeZ865iox4AAECEoCQ3EVtdHdyoZ315EybLtI53HQkAAAB7iJLcBKy1sn9+UNr4lbyLrpPp2Nl1JAAAAISAktwE7OJ/y777usypv5Y5+HDXcQAAABAiSnIjs59/JPvsfClnoMzIMa7jAAAAYC9QkhuRLd4q/+G7pU5d5Z1/NRv1AAAAIhQluZHYmm836tXVBjfqxSe4jgQAAIC9REluBNZa2b/Mlr5eI++C38hkdHUdCQAAAPuAktwI7JIFsm+9KnPKWJlDj3AdBwAAAPuIkryPbN4q2b/Pkw49UubkM1zHAQAAQCOgJO8DW1Ikf87vpQ4Z8sZfK+MxTgAAgJaAVreXbKBG/pzpUqBG3oSbZBLauo4EAACARkJJ3gvWWtkn50jr8oJnkDtnuo4EAACARkRJ3gv29UWyyxbLnHyGTP8BruMAAACgkVGSQ2TzP5P92yNSv1yZU85yHQcAAABNgJIcAltWHNyol9ZB3oUTZbwY15EAAADQBCjJe8gGAsGCXLUzeEe9hHauIwEAAKCJUJL3kP3bI9KaL+Sdf7VM1x6u4wAAAKAJUZL3gP/mS7JvLJI56Zcyhx/tOg4AAACaGCW5AXbtl7JPzZEO7C8z+mzXcQAAANAMKMk/wW4rlf/QdKl9mryLr2OjHgAAQJSgJP8IW/vtRr3KHfIunyzTNtF1JAAAADQTSvKPsM/8Scr/TObcK2Uye7mOAwAAgGZESf4B/rJXZF9dIDP8NHlHHuc6DgAAAJoZJXk39qvVsn+ZLR1wqMwvxrmOAwAAAAcoyf/DLyuR/9A0KTlF3kXXy8SwUQ8AACAaUZK/ZWtrVXbPzVL5dnmXTZJJTHIdCQAAAI5Qkr9ln3tMgU8/lDnncpkeWa7jAAAAwCFKsiQbqJFd+6USfn6GvIFDXMcBAACAY7GuA4QDE9dK3nV3qV2HDqouK3MdBwAAAI5xJvlbJi5OJpb/ZgAAAAAlGQAAAPgeSjIAAACwG0oyAAAAsBtKMgAAALAbSjIAAACwG0oyAAAAsJs9uubZypUrNX/+fPm+r6FDh2r06NH1ni8qKtKsWbNUUVEh3/c1duxY5eTkSJKef/55LVmyRJ7n6fzzz9dhhx3W+O8CAAAAaEQNlmTf9zVv3jxNmTJFaWlpmjRpknJzc5WZmbnrmOeee04DBw7U8OHDVVBQoGnTpiknJ0cFBQVavny57r33XpWWluqOO+7Q/fffL8/jBDYAAADCV4NtNT8/XxkZGerUqZNiY2M1aNAgrVixot4xxhhVVlZKkiorK5WSkiJJWrFihQYNGqS4uDh17NhRGRkZys/Pb4K3AQAAADSeBs8kl5SUKC0tbdfjtLQ0rV69ut4xY8aM0Z133qlFixapurpaN998866f7du3767jUlNTVVJS0ljZAQAAgCbRKPdhXrZsmQYPHqxRo0YpLy9PM2fO1IwZM/b45xcvXqzFixdLkqZPn6709PTGiBWy2NhYZ68diZhXaJhXaJhXaJhXaJhXaJhX6JhZaMJxXg2W5NTUVBUXF+96XFxcrNTU1HrHLFmyRJMnT5YkZWdnKxAIqLy8/Hs/W1JS8r2flaRhw4Zp2LBhux4XFRWF/k4aQXp6urPXjkTMKzTMKzTMKzTMKzTMKzTMK3TMLDSu5tWlS5cffa7BNclZWVkqLCzU1q1bVVtbq+XLlys3N7feMenp6Vq1apUkqaCgQIFAQElJScrNzdXy5csVCAS0detWFRYWqk+fPvv4dgAAAICm1eCZ5JiYGI0fP15Tp06V7/saMmSIunXrpqefflpZWVnKzc3VuHHjNHfuXC1YsECSNGHCBBlj1K1bNw0cOFATJ06U53m64IILuLIFAAAAwp6x1lrXIXa3adMmJ6/Lr0ZCw7xCw7xCw7xCw7xCw7xCw7xCx8xCE5HLLQAAAIBoQ0kGAAAAdkNJBgAAAHZDSQYAAAB2Q0kGAAAAdkNJBgAAAHZDSQYAAAB2Q0kGAAAAdkNJBgAAAHZDSQYAAAB2Q0kGAAAAdkNJBgAAAHZjrLXWdQgAAAAgnHAm+X/ceOONriNEFOYVGuYVGuYVGuYVGuYVGuYVOmYWmnCcFyUZAAAA2A0lGQAAANhNzK233nqr6xDhpHfv3q4jRBTmFRrmFRrmFRrmFRrmFRrmFTpmFppwmxcb9wAAAIDdsNwCAAAA2A0lGQAAANhNrOsALqxcuVLz58+X7/saOnSoRo8eXe/5QCCgBx98UGvXrlViYqKuueYadezY0VFa9xqa12uvvaYnnnhCqampkqQRI0Zo6NChLqI6N3v2bH3wwQdKTk7WjBkzvve8tVbz58/Xhx9+qNatW2vChAlhtwarOTU0r08//VR/+MMfdv39O+qoo3T66ac3d8ywUVRUpFmzZqmsrEzGGA0bNkwjR46sdwyfse/sybz4jH2npqZGt9xyi2pra1VXV6cBAwbojDPOqHcM34/f2ZN58f34fb7v68Ybb1Rqaur3LvsWdp8vG2Xq6ursFVdcYTdv3mwDgYC97rrr7IYNG+ods2jRIjt37lxrrbVLly619957r4uoYWFP5vXqq6/aRx991FHC8PLpp5/aNWvW2IkTJ/7g8++//76dOnWq9X3ffvnll3bSpEnNnDC8NDSvVatW2WnTpjVzqvBVUlJi16xZY621trKy0l511VXf+/vIZ+w7ezIvPmPf8X3f7ty501prbSAQsJMmTbJffvllvWP4fvzOnsyL78fve+GFF+x99933g3/vwu3zFXXLLfLz85WRkaFOnTopNjZWgwYN0ooVK+od895772nw4MGSpAEDBmjVqlWyUbq/cU/mhe8ceOCBateu3Y8+/9577+m4446TMUbZ2dmqqKhQaWlpMyYMLw3NC/WlpKTsOivcpk0bde3aVSUlJfWO4TP2nT2ZF75jjFF8fLwkqa6uTnV1dTLG1DuG78fv7Mm8UF9xcbE++OCDHz2bHm6fr6hbblFSUqK0tLRdj9PS0rR69eofPSYmJkYJCQkqLy9XUlJSs2YNB3syL0l655139Pnnn6tz584699xzlZ6e3pwxI0ZJSUm92aSlpamkpEQpKSkOU4W3vLw8XX/99UpJSdE555yjbt26uY4UFrZu3ap169apT58+9f45n7Ef9mPzkviM/S/f93XDDTdo8+bNOvHEE9W3b996z/P9WF9D85L4fvxfjz32mM4++2zt3LnzB58Pt89X1J1JRuM7/PDDNWvWLN1zzz065JBDNGvWLNeR0EL06tVLs2fP1t13360RI0bo7rvvdh0pLFRVVWnGjBk677zzlJCQ4DpO2PupefEZq8/zPN19992aM2eO1qxZo6+//tp1pLDW0Lz4fvzO+++/r+Tk5IjaIxF1JTk1NVXFxcW7HhcXF+9aUP9Dx9TV1amyslKJiYnNmjNc7Mm8EhMTFRcXJ0kaOnSo1q5d26wZI0lqaqqKiop2Pf6heeI7CQkJu36dmZOT83/t3bFLOnEYx/EPCA4lNDoYLTc42NZBNDS0ODVJCDY4NSXEjU79B4GEgktDS9DcEk7iEA6HmKiUCNEUNIXREtddww/0d2bp9Lt+3Ps1ns/w8PBwz8Pdl1MfHx8ajUYBZxUsx3F0cnKi7e1tbW5ufvmdHvObVy96bLbl5WWlUim1223fdebjbN/Vi/k4cX9/L9u2VSgUVCqV1O12dXp66ov5bf0VuiXZMAw9PT3p+flZjuPo5uZGpmn6YjY2NlSv1yVJzWZTqVQqtOeMFqnX3+cdbdvW6urqv07zv2GaphqNhjzP02Aw0NLSUuhfg//k5eVlfB5tOBzKdd1QD2TP81StVpVIJLS7uzszhh6bWKRe9NjEaDTS29ubpD9fbuh0OkokEr4Y5uPEIvViPk7s7++rWq2qUqnIsiytr6/r6OjIF/Pb+iuU/7jXarV0fn4u13W1s7OjTCajy8tLGYYh0zT1/v6ucrmsh4cHxWIxWZaleDwedNqBmVevi4sL2batSCSiWCymg4ODLzeKsCiVSur3+3p9fdXKyoqy2awcx5EkpdNpeZ6ns7Mz3d7eKhqN6vDwUIZhBJx1cObV6/r6WrVaTZFIRNFoVPl8XslkMuCsg3N3d6fj42Otra2NB0culxs/OabH/BapFz028fj4qEqlItd15Xmetra2tLe3x3z8xiL1Yj7O1uv1dHV1pWKx+Kv7K5RLMgAAAPCT0B23AAAAAOZhSQYAAACmsCQDAAAAU1iSAQAAgCksyQAAAMAUlmQAAABgCksyAAAAMOUTObEMA6l7VNEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x864 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.style.use(\"ggplot\")\n",
    "plt.figure(figsize=(12, 12))\n",
    "plt.plot(hist[\"accuracy\"])\n",
    "plt.plot(hist[\"val_accuracy\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KCjluwQxgxzD"
   },
   "outputs": [],
   "source": [
    "# Eval\n",
    "pred_cat = model.predict(X_te)\n",
    "pred = np.argmax(pred_cat, axis=-1)\n",
    "y_te_true = np.argmax(y_te, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 490
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 449082,
     "status": "ok",
     "timestamp": 1595429138938,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "ez8I_yB9g9Lj",
    "outputId": "41d71fe4-b5c3-4f53-b3d0-7cb377f160b1"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       B-art       0.00      0.00      0.00        44\n",
      "       B-eve       0.00      0.00      0.00        37\n",
      "       B-geo       0.10      0.89      0.18      3681\n",
      "       B-gpe       0.50      0.92      0.65      1571\n",
      "       B-nat       0.00      0.00      0.00        17\n",
      "       B-org       0.50      0.56      0.53      1994\n",
      "       B-per       0.54      0.70      0.61      1735\n",
      "       B-tim       0.08      0.79      0.14      2003\n",
      "       I-art       0.00      0.00      0.00        32\n",
      "       I-eve       0.00      0.00      0.00        32\n",
      "       I-geo       0.04      0.01      0.01       718\n",
      "       I-gpe       0.00      0.00      0.00        25\n",
      "       I-nat       0.00      0.00      0.00         5\n",
      "       I-org       0.09      0.78      0.16      1712\n",
      "       I-per       0.10      0.14      0.12      1767\n",
      "       I-tim       0.04      0.01      0.01       647\n",
      "           O       0.31      0.99      0.48     88476\n",
      "         PAD       1.00      0.00      0.00    255204\n",
      "\n",
      "    accuracy                           0.27    359700\n",
      "   macro avg       0.18      0.32      0.16    359700\n",
      "weighted avg       0.80      0.27      0.13    359700\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn_crfsuite.metrics import flat_classification_report\n",
    "\n",
    "# Convert the index to tag\n",
    "pred_tag = [[idx2tag[i] for i in row] for row in pred]\n",
    "y_te_true_tag = [[idx2tag[i] for i in row] for row in y_te_true] \n",
    "\n",
    "report = flat_classification_report(y_pred=pred_tag, y_true=y_te_true_tag)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0mZT4Z4w88Ix"
   },
   "source": [
    "Evaluate some samples in the test set. (At each execution it will test on a different sample)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 454
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 446883,
     "status": "ok",
     "timestamp": 1595429138940,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "kgHXCTl0hJia",
    "outputId": "b5c4aa79-ff29-4082-91fe-33a0f24c788a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample number 1162 of 4796 (Test Set)\n",
      "Word           ||True ||Pred\n",
      "==============================\n",
      "The            : O     O\n",
      "foreign        : O     O\n",
      "minister       : O     O\n",
      "of             : O     O\n",
      "the            : O     O\n",
      "ousted         : O     O\n",
      "Taleban        : B-org B-org\n",
      "regime         : O     O\n",
      "in             : O     O\n",
      "Afghanistan    : B-geo B-geo\n",
      "has            : O     O\n",
      "decided        : O     O\n",
      "to             : O     O\n",
      "run            : O     O\n",
      "for            : O     O\n",
      "a              : O     O\n",
      "seat           : O     O\n",
      "in             : O     O\n",
      "September      : B-tim B-tim\n",
      "'s             : O     O\n",
      "parliamentary  : O     O\n",
      "elections      : O     O\n",
      ".              : O     O\n"
     ]
    }
   ],
   "source": [
    "i = np.random.randint(0,X_te.shape[0]) # choose a random number between 0 and len(X_te)\n",
    "p = model.predict(np.array([X_te[i]]))\n",
    "p = np.argmax(p, axis=-1)\n",
    "true = np.argmax(y_te[i], -1)\n",
    "\n",
    "print(\"Sample number {} of {} (Test Set)\".format(i, X_te.shape[0]))\n",
    "# Visualization\n",
    "print(\"{:15}||{:5}||{}\".format(\"Word\", \"True\", \"Pred\"))\n",
    "print(30 * \"=\")\n",
    "for w, t, pred in zip(X_te[i], true, p[0]):\n",
    "    if w != 0:\n",
    "        print(\"{:15}: {:5} {}\".format(words[w-2], idx2tag[t], idx2tag[pred]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 95,
     "referenced_widgets": [
      "0318f53fdb4a4facae3a3aa37fc7cb39",
      "23f45e15fde04197966163a60bebebff",
      "4846aa00cf29467abe816b318b4f5070",
      "95a39657947141f8a21c475e4a6463d6",
      "19577af82bc8462cbe99e1c8c4921356",
      "0f384f71fbab44efa2ad4a0d375fdeb7",
      "34b8e69db07c4befa7dfc9249fb997f8",
      "216282ad6f2345b4bc3419539a950541",
      "46737dcdfb8247feab4e91f4d04e328f",
      "2451ac832f624b279b886a790c09b72a"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 446431,
     "status": "ok",
     "timestamp": 1595429138941,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "2ERtAY-W4aGp",
    "outputId": "9aaf6bb9-a2a4-4c32-f090-c824c36ca25e"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0318f53fdb4a4facae3a3aa37fc7cb39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Textarea(value='', description='sentence', placeholder='Type your sentence here'), Butto…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ipywidgets import interact_manual\n",
    "from ipywidgets import widgets\n",
    "\n",
    "import re\n",
    "import string\n",
    "\n",
    "# Custom Tokenizer\n",
    "re_tok = re.compile(f'([{string.punctuation}“”¨«»®´·º½¾¿¡§£₤‘’])')\n",
    "def tokenize(s): return re_tok.sub(r' \\1 ', s).split()\n",
    "    \n",
    "def get_prediction(sentence):\n",
    "    test_sentence = tokenize(sentence) # Tokenization\n",
    "    # Preprocessing\n",
    "    x_test_sent = pad_sequences(sequences=[[word2idx.get(w, 0) for w in test_sentence]],\n",
    "                            padding=\"post\", value=word2idx[\"PAD\"], maxlen=MAX_LEN)\n",
    "    # Evaluation\n",
    "    p = model.predict(np.array([x_test_sent[0]]))\n",
    "    p = np.argmax(p, axis=-1)\n",
    "    # Visualization\n",
    "    print(\"{:15}||{}\".format(\"Word\", \"Prediction\"))\n",
    "    print(30 * \"=\")\n",
    "    for w, pred in zip(test_sentence, p[0]):\n",
    "        print(\"{:15}: {:5}\".format(w, idx2tag[pred]))\n",
    "\n",
    "interact_manual(get_prediction, sentence=widgets.Textarea(placeholder='Type your sentence here'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CTtuGIma5QZg"
   },
   "source": [
    "# Save the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_Wnp2jKK4j8P"
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Saving Vocab\n",
    "with open('models/word_to_index.pickle', 'wb') as handle:\n",
    "    pickle.dump(word2idx, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    " \n",
    "# Saving Vocab\n",
    "with open('models/tag_to_index.pickle', 'wb') as handle:\n",
    "    pickle.dump(tag2idx, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "    \n",
    "# Saving Model Weight\n",
    "model.save_weights('models/lstm_crf_weights.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DUjTp5E_5ULF"
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uqISik0s5qwd"
   },
   "source": [
    "[Build a deep learning model for predicting the named entities from text.](https://github.com/floydhub/named-entity-recognition-template)\n",
    "\n",
    "[A simple CRF layer for tensorflow 2 keras](https://github.com/xuxingya/tf2crf)\n",
    "\n",
    "[CRF(Conditional Random Field) Layer for TensorFlow 1.X with many powerful functions](https://github.com/howl-anderson/tf_crf_layer)\n",
    "\n",
    "[Bidirectional LSTM-CRF Models for Sequence Tagging](https://arxiv.org/pdf/1508.01991v1.pdf)\n",
    "\n",
    "[Building an Incremental Named Entity Recognizer System](http://www.stokastik.in/building-an-incremental-named-entity-recognizer-system/)\n",
    "\n",
    "[BiLSTM-CRF Sequence Tagging for E-Commerce Attribute Extraction](http://www.stokastik.in/bilstm-crf-sequence-tagging-for-e-commerce-attribute-extraction/)\n",
    "\n",
    "[BERT-Named Entity Recognizer](https://github.com/cchantra/nlp_tourism/tree/master/more_experiment/restuarant/bert-NER)\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyNsM7ZB3SfDlkRzfulzzfth",
   "collapsed_sections": [],
   "name": "ner_word_level_BiLSTM_CRF.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0318f53fdb4a4facae3a3aa37fc7cb39": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "VBoxModel",
     "state": {
      "_dom_classes": [
       "widget-interact"
      ],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "VBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "VBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4846aa00cf29467abe816b318b4f5070",
       "IPY_MODEL_95a39657947141f8a21c475e4a6463d6",
       "IPY_MODEL_19577af82bc8462cbe99e1c8c4921356"
      ],
      "layout": "IPY_MODEL_23f45e15fde04197966163a60bebebff"
     }
    },
    "0f384f71fbab44efa2ad4a0d375fdeb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "19577af82bc8462cbe99e1c8c4921356": {
     "model_module": "@jupyter-widgets/output",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_2451ac832f624b279b886a790c09b72a",
      "msg_id": "",
      "outputs": []
     }
    },
    "216282ad6f2345b4bc3419539a950541": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ButtonStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ButtonStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "button_color": null,
      "font_weight": ""
     }
    },
    "23f45e15fde04197966163a60bebebff": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2451ac832f624b279b886a790c09b72a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "34b8e69db07c4befa7dfc9249fb997f8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "46737dcdfb8247feab4e91f4d04e328f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4846aa00cf29467abe816b318b4f5070": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "TextareaModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "TextareaModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "TextareaView",
      "continuous_update": true,
      "description": "sentence",
      "description_tooltip": null,
      "disabled": false,
      "layout": "IPY_MODEL_34b8e69db07c4befa7dfc9249fb997f8",
      "placeholder": "Type your sentence here",
      "rows": null,
      "style": "IPY_MODEL_0f384f71fbab44efa2ad4a0d375fdeb7",
      "value": ""
     }
    },
    "95a39657947141f8a21c475e4a6463d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ButtonModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ButtonModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ButtonView",
      "button_style": "",
      "description": "Run Interact",
      "disabled": false,
      "icon": "",
      "layout": "IPY_MODEL_46737dcdfb8247feab4e91f4d04e328f",
      "style": "IPY_MODEL_216282ad6f2345b4bc3419539a950541",
      "tooltip": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}