nlp_torch.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "-GA2Z4Y4v1dX",
    "outputId": "c14792ba-95a9-4168-dfc0-7dd3a207d013"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from torchtext.data import Field\n",
    "from torchtext.data import TabularDataset\n",
    "from torchtext.data import Iterator, BucketIterator\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.autograd import Variable\n",
    "import tqdm\n",
    "import spacy\n",
    "from spacy.symbols import ORTH\n",
    "\n",
    "from google.colab import drive\n",
    "# Ignore warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "drive.mount('/content/gdrive')\n",
    "data_dir = 'gdrive/My Drive/Colab Notebooks/data/headlines' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "32uy01GoeC0y"
   },
   "source": [
    "I am using Google Colab notebook, so I had to provide access to my files from within the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oVA9BLBuVi_6"
   },
   "source": [
    "# Preprocessing declaration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OGTbbpBATy74"
   },
   "source": [
    "First we need to define a function to be used for tokenization. I will use Spacy library. \n",
    "\n",
    "What is also necessary is to define data fields. It is a way to declare what type of preprocessing should be done on raw text. \n",
    "\n",
    "Note that our labels are ready to be used, we don't need vocabulary mapping for them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fxsA7PdWwHRt"
   },
   "outputs": [],
   "source": [
    "tok = spacy.load('en')\n",
    "\n",
    "def spacy_tokenize(x):\n",
    "    return [token.text for token in tok.tokenizer(x)]\n",
    "\n",
    "TEXT = Field(sequential=True, tokenize=spacy_tokenize, lower=True)\n",
    "LABEL = Field(sequential=False, use_vocab=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jzTQpsI3Vn53"
   },
   "source": [
    "# Dataset classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zsw0OEcfZOis"
   },
   "source": [
    "When working with PyTorch we always need to declare our dataset classes. In this case it is appropriate to use tabular dataset from torchtext package. Each headline and corresponding class are together wrapped in Example object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2CuKXsZTzy8Q"
   },
   "outputs": [],
   "source": [
    "datafields = [(\"headline\", TEXT), (\"is_sarcastic\", LABEL)]\n",
    "\n",
    "train_dataset, validation_dataset = TabularDataset.splits(path=data_dir, train='train.csv', validation=\"val.csv\",\n",
    "                                          format='csv', skip_header=True, fields=datafields)\n",
    "\n",
    "test_dataset = TabularDataset(path=data_dir+\"/test.csv\", format='csv', skip_header=True, fields=datafields)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FKgC11Z3e0DS"
   },
   "source": [
    "We can verify if the rows are represented correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kPbfK-B21Ee6",
    "outputId": "01ec5f4f-cd91-433f-ea56-e6e754c7416e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torchtext.data.example.Example at 0x7fe3e2c9aa58>"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "RMkArw7zVNA2",
    "outputId": "0c7bf277-9e24-44af-d574-90ec0a389ee8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['headline', 'is_sarcastic'])"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0].__dict__.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "H79qt78CVdH_",
    "outputId": "2be20f10-917e-4cdd-c8f5-f76d6c4750fa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['woman', 'has', 'no', 'business']"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0].headline[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EKydulg5VudG"
   },
   "source": [
    "When working with text data and neural netoworks, we need to build a vocabulary. that is mapping from words to their corresponding integer ids.\n",
    "\n",
    "Training dataset is used for that purpose. For my model, I will use GloVe 100-dimensional word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OSecIUnlVp3Y"
   },
   "outputs": [],
   "source": [
    "TEXT.build_vocab(train_dataset, vectors=\"glove.6B.100d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xa0Pse8AWBBO"
   },
   "source": [
    "For text data, PyTorch uses the convention of iterators, that enable us to iterate over the datasets.\n",
    "\n",
    "In each iteration over the dataset we get a batch of headlines and their labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qAsV3ZhhZMiZ"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "train_iterator, val_iterator = BucketIterator.splits((train_dataset, validation_dataset), batch_sizes=(64, 64), device=device,\n",
    "                                                     sort_key=lambda x: len(x.headline), sort_within_batch=False, repeat=False)\n",
    "\n",
    "test_iterator = Iterator(test_dataset, batch_size=64, device=device, sort=False, sort_within_batch=False, repeat=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HQ0NWUpSzYCT"
   },
   "source": [
    "We can see that each batch consists of 64 examples. The size of headlines tensor is determined by length of the longest sentence in the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "kRu5VHEJqHZa",
    "outputId": "4089b5c9-0e93-4320-ffa0-44dd3ef8c36b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[torchtext.data.batch.Batch of size 64]\n",
       "\t[.headline]:[torch.cuda.LongTensor of size 17x64 (GPU 0)]\n",
       "\t[.is_sarcastic]:[torch.cuda.LongTensor of size 64 (GPU 0)]"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = next(train_iterator.__iter__()); batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "MMVObpMAqUOv",
    "outputId": "b3d20b9f-4459-4dba-df73-f181e77007b4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['batch_size', 'dataset', 'fields', 'input_fields', 'target_fields', 'headline', 'is_sarcastic'])"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch.__dict__.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N79BHyuXWd2n"
   },
   "source": [
    "For conveniance, we can define a wrapper for each batch: while iterating over a dataset we will get a touple (headline, label)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oP3pGuYGa_ln"
   },
   "outputs": [],
   "source": [
    "class BatchWrapper:\n",
    "      def __init__(self, dl, x_var, y_vars):\n",
    "            self.dl, self.x_var, self.y_vars = dl, x_var, y_vars\n",
    "\n",
    "      def __iter__(self):\n",
    "        for batch in self.dl:\n",
    "                  x = getattr(batch, self.x_var)\n",
    "\n",
    "                  if self.y_vars is not None:\n",
    "                        y = torch.cat([getattr(batch, feat).unsqueeze(1) for feat in self.y_vars], dim=1).float()\n",
    "                  else:\n",
    "                        y = torch.zeros((1))\n",
    "\n",
    "                  yield (x, y)\n",
    "\n",
    "      def __len__(self):\n",
    "            return len(self.dl)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dv8L0QLGsp2B"
   },
   "outputs": [],
   "source": [
    "train_dataloader = BatchWrapper(train_iterator, \"headline\", [\"is_sarcastic\"])\n",
    "valid_dataloader = BatchWrapper(val_iterator, \"headline\", [\"is_sarcastic\"])\n",
    "test_dataloader = BatchWrapper(test_iterator, \"headline\", [\"is_sarcastic\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zn5kAJfRXI-v"
   },
   "source": [
    "For this task, I'll use a network that embedds the headline using embedding layer first, then the vectors are passed through LSTM and hidden layer, and then output layer. In order to get class probabilty I am using sigmoid function and return the output.\n",
    "\n",
    "What I mean by embedding layer? It is a special layer that provides a lookup for a word in embeddings space. It's a theoretical, multidimensional space, in which similar words appear close to each other. Such a representation is able to encode a lot of information about the language, it's semantics and structure. They are build on large portions of text, called corpuses - like a whole Wikipedia! I am using GloVe embeddings here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xOTo2YoWc4lg"
   },
   "outputs": [],
   "source": [
    "class SimpleLSTM(nn.Module):\n",
    "    def __init__(self, hidden_dim, emb_dim=100, recurrent_dropout=0.1):\n",
    "      \n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(len(TEXT.vocab), emb_dim)\n",
    "        self.encoder = nn.LSTM(emb_dim, hidden_dim, num_layers=1, dropout=recurrent_dropout)\n",
    "\n",
    "        self.predictor = nn.Linear(hidden_dim, 1)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "    \n",
    "    def forward(self, seq):\n",
    "        hdn, _ = self.encoder(self.embedding(seq))\n",
    "        feature = hdn[-1, :, :]\n",
    "\n",
    "        preds = self.predictor(feature).squeeze()\n",
    "        preds = self.sigmoid(preds)\n",
    "        return preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "ViAyxDaAfIEF",
    "outputId": "0bab6ee2-99b8-4d22-a91d-30472ebdb4ca"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleLSTM(\n",
       "  (embedding): Embedding(19723, 100)\n",
       "  (encoder): LSTM(100, 500, dropout=0.2)\n",
       "  (predictor): Linear(in_features=500, out_features=1, bias=True)\n",
       "  (sigmoid): Sigmoid()\n",
       "  (relu): ReLU(inplace=True)\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = SimpleLSTM(hidden_dim=500, recurrent_dropout=0.2)\n",
    "\n",
    "weight_matrix = TEXT.vocab.vectors\n",
    "model.embedding.weight.data.copy_(weight_matrix)\n",
    "model.cuda()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oy_Ccw5pfkun"
   },
   "source": [
    "We are set up now! Time to write training loop, decide on training parameters like optimizer, learning rate and number of epochs. Depending on the task we are working on, appropriate loss function must be used. In case of binary classification, we can stick to Binary Cross Entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "4IwbHZ_pcUb_",
    "outputId": "6a3de0be-86f3-42f8-d12f-98dc50d5fae3"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.09it/s]\n",
      "  2%|▏         | 4/233 [00:00<00:06, 36.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 1, Training loss: 0.2304, Validation loss: 0.127\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.78it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 2, Training loss: 0.2205, Validation loss: 0.1345\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.17it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 46.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 3, Training loss: 0.1989, Validation loss: 0.122\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.75it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 4, Training loss: 0.1717, Validation loss: 0.122\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.46it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:05, 44.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 5, Training loss: 0.1633, Validation loss: 0.1195\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.58it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 52.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 6, Training loss: 0.1581, Validation loss: 0.1171\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.83it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 7, Training loss: 0.1526, Validation loss: 0.1167\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.28it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 8, Training loss: 0.1483, Validation loss: 0.1176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.51it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 9, Training loss: 0.1457, Validation loss: 0.1144\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.98it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 51.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 10, Training loss: 0.1428, Validation loss: 0.1124\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.32it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 53.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 11, Training loss: 0.1392, Validation loss: 0.1132\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.88it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 48.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 12, Training loss: 0.1346, Validation loss: 0.1141\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.92it/s]\n",
      "  2%|▏         | 4/233 [00:00<00:05, 38.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 13, Training loss: 0.1339, Validation loss: 0.1092\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.84it/s]\n",
      "  2%|▏         | 4/233 [00:00<00:05, 39.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 14, Training loss: 0.1308, Validation loss: 0.1125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.50it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 53.29it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 15, Training loss: 0.1291, Validation loss: 0.1122\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.93it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 45.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 16, Training loss: 0.1249, Validation loss: 0.1055\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.72it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 53.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 17, Training loss: 0.1198, Validation loss: 0.1104\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.17it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 46.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 18, Training loss: 0.1239, Validation loss: 0.1053\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.74it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:05, 45.07it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 19, Training loss: 0.1179, Validation loss: 0.1052\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.11it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 53.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 20, Training loss: 0.1127, Validation loss: 0.0963\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.09it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 21, Training loss: 0.1123, Validation loss: 0.0958\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.48it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 22, Training loss: 0.1071, Validation loss: 0.0905\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.72it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 23, Training loss: 0.1062, Validation loss: 0.0902\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.08it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 55.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 24, Training loss: 0.1042, Validation loss: 0.0897\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.30it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 25, Training loss: 0.1023, Validation loss: 0.0933\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.09it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 54.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 26, Training loss: 0.1014, Validation loss: 0.089\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.12it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 54.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 27, Training loss: 0.0981, Validation loss: 0.0911\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.88it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 28, Training loss: 0.0967, Validation loss: 0.0878\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.95it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 52.51it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 29, Training loss: 0.0937, Validation loss: 0.0839\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.34it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 30, Training loss: 0.0945, Validation loss: 0.081\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.63it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 48.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 31, Training loss: 0.0933, Validation loss: 0.0868\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.54it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 52.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 32, Training loss: 0.0867, Validation loss: 0.0892\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.03it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 33, Training loss: 0.0877, Validation loss: 0.0868\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.13it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 48.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 34, Training loss: 0.0873, Validation loss: 0.0789\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.80it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.35it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 35, Training loss: 0.0839, Validation loss: 0.0802\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.64it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 48.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 36, Training loss: 0.0824, Validation loss: 0.0776\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.88it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 37, Training loss: 0.0823, Validation loss: 0.0927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.15it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 38, Training loss: 0.0803, Validation loss: 0.0832\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.93it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 49.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 39, Training loss: 0.0783, Validation loss: 0.0852\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.34it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 46.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 40, Training loss: 0.0771, Validation loss: 0.0853\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.16it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 47.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 41, Training loss: 0.0744, Validation loss: 0.0741\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.84it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 50.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 42, Training loss: 0.0747, Validation loss: 0.0743\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.83it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 46.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 43, Training loss: 0.0726, Validation loss: 0.0806\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.43it/s]\n",
      "  2%|▏         | 4/233 [00:00<00:05, 38.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 44, Training loss: 0.0703, Validation loss: 0.088\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.95it/s]\n",
      "  2%|▏         | 5/233 [00:00<00:04, 48.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 45, Training loss: 0.0698, Validation loss: 0.0776\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.59it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 54.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 46, Training loss: 0.0666, Validation loss: 0.0736\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.72it/s]\n",
      "  2%|▏         | 4/233 [00:00<00:05, 39.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 47, Training loss: 0.0644, Validation loss: 0.0705\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.84it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 55.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 48, Training loss: 0.0645, Validation loss: 0.0733\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 62.99it/s]\n",
      "  3%|▎         | 6/233 [00:00<00:04, 51.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 49, Training loss: 0.0636, Validation loss: 0.085\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 233/233 [00:03<00:00, 63.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Epoch: 50, Training loss: 0.0593, Validation loss: 0.0742\n"
     ]
    }
   ],
   "source": [
    "optimizer = optim.Adam(model.parameters(), lr=2e-5)\n",
    "loss_func = nn.BCELoss()\n",
    "\n",
    "epochs = 50\n",
    "\n",
    "train_epoch_losses = []\n",
    "validation_epoch_losses = []\n",
    "\n",
    "for epoch in range(1, epochs + 1):\n",
    "    running_loss = 0.0\n",
    "    running_corrects = 0\n",
    "    model.train()\n",
    "\n",
    "    for x, y in tqdm.tqdm(train_dataloader):\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        y_pred = model(x)\n",
    "        loss = loss_func(y_pred, y)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        running_loss += loss.item() * x.size(0)\n",
    "\n",
    "    epoch_loss = running_loss / len(train_dataset)\n",
    "    train_epoch_losses.append(epoch_loss)\n",
    "\n",
    "    val_loss = 0.0\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "      for x, y in valid_dataloader:\n",
    "          y_pred = model(x)\n",
    "          loss = loss_func(y_pred, y)\n",
    "          val_loss += loss.item() * x.size(0)\n",
    "\n",
    "    val_loss /= len(validation_dataset)\n",
    "    validation_epoch_losses.append(val_loss)\n",
    "    print(f'\\n Epoch: {epoch}, Training loss: {round(epoch_loss, 4)}, Validation loss: {round(val_loss, 4)}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "colab_type": "code",
    "id": "wAXBt1S3tUGA",
    "outputId": "c2c61509-2409-4731-876a-d4d0c1cd94d8"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3hUZfbA8e9JhzRSqaEjJSEQEorS\nRREVQWxUQVfFgov7c3VF19VVF9e2iih2KTbQRVEsiKgUdSEQinSkk4SSUJIQ0pP398edYIgkmZAy\nycz5PM88M3PrezHeM/ct5xVjDEoppVyPm6MLoJRSyjE0ACillIvSAKCUUi5KA4BSSrkoDQBKKeWi\nPBxdgMoIDQ01rVu3dnQxlFKqXlm/fv1xY0xY6eX1KgC0bt2ahIQERxdDKaXqFRE5eL7lWgWklFIu\nSgOAUkq5KA0ASinloupVG4BSqvbl5+eTlJRETk6Oo4uiKuDj40OLFi3w9PS0a3sNAEqpciUlJeHv\n70/r1q0REUcXR5XBGMOJEydISkqiTZs2du2jVUBKqXLl5OQQEhKiN/86TkQICQmp1JOaBgClVIX0\n5l8/VPa/k0sEgKXbjvJR/CFHF0MppeoUlwgAn29MZvrX20k9nevooiilKiktLY3XXnvtgva96qqr\nSEtLs3v7f/7zn7zwwgsXdK76yCUCwINXdCSnoIiZP+x2dFGUUpVUXgAoKCgod99vvvmGRo0a1USx\nnIJLBIC2YX6M7RXB/LWH2Jea6ejiKKUqYdq0aezdu5fu3bvz4IMPsmLFCvr378+IESPo0qULANde\ney2xsbFERkby1ltvnd23devWHD9+nAMHDtC5c2fuuOMOIiMjGTp0KNnZ2eWed9OmTfTp04fo6GhG\njRrFqVOnAJg5cyZdunQhOjqaMWPGALBy5Uq6d+9O9+7diYmJ4fTp0zX0r1G97OoGKiLDgJcBd+Ad\nY8wzpdbfD9wOFACpwJ+MMQdFpDvwOhAAFALTjTEf2/aZCwwE0m2HucUYs6nKV1SG+4ZcxGcbknl+\n6S5enxBbU6dRyqk98eU2th/OqNZjdmkWwOPXRJa5/plnnmHr1q1s2mTdHlasWMGGDRvYunXr2e6O\ns2fPJjg4mOzsbHr27Mn1119PSEjIOcfZvXs38+fP5+233+amm27i008/ZcKECWWed+LEibzyyisM\nHDiQxx57jCeeeIIZM2bwzDPPsH//fry9vc9WL73wwgvMmjWLvn37kpmZiY+PT1X/WWpFhU8AIuIO\nzAKuBLoAY0WkS6nNNgJxxphoYCHwnG15FjDRGBMJDANmiEjJ57EHjTHdba8au/kDhPl7M3lAW5Zs\nPcqGQ6dq8lRKqRrWq1evc/q6z5w5k27dutGnTx8SExPZvfuP1b1t2rShe/fuAMTGxnLgwIEyj5+e\nnk5aWhoDBw4EYNKkSaxatQqA6Ohoxo8fzwcffICHh/Ubum/fvtx///3MnDmTtLS0s8vrOntK2QvY\nY4zZByAiC4CRwPbiDYwxy0tsvwaYYFv+W4ltDotIChAG2N8qU43u6N+WD9Yc4t/f7OCTOy/Wrm1K\nVVJ5v9Rrk6+v79nPK1as4Pvvv2f16tU0bNiQQYMGnbcvvLe399nP7u7uFVYBleXrr79m1apVfPnl\nl0yfPp0tW7Ywbdo0rr76ar755hv69u3L0qVL6dSp0wUdvzbZ0wbQHEgs8T3JtqwstwFLSi8UkV6A\nF7C3xOLpIrJZRF4SEe/S+1Q3X28P/nJZB9YdOMX3O1Jq+nRKqWrg7+9fbp16eno6QUFBNGzYkJ07\nd7JmzZoqnzMwMJCgoCB++uknAN5//30GDhxIUVERiYmJDB48mGeffZb09HQyMzPZu3cvXbt25aGH\nHqJnz57s3LmzymWoDdXaCCwiE4A44PlSy5sC7wO3GmOKbIsfBjoBPYFg4KEyjjlZRBJEJCE1NbXK\nZRzdM4K2Yb48++1OCgqLKt5BKeVQISEh9O3bl6ioKB588ME/rB82bBgFBQV07tyZadOm0adPn2o5\n77x583jwwQeJjo5m06ZNPPbYYxQWFjJhwgS6du1KTEwMU6dOpVGjRsyYMYOoqCiio6Px9PTkyiuv\nrJYy1DQxxpS/gcjFwD+NMVfYvj8MYIz5d6ntLgNeAQYaY1JKLA8AVgBPG2MWlnGOQcADxpjh5ZUl\nLi7OVMeEMN9uPcpdH6zn39d1ZWyvllU+nlLObMeOHXTu3NnRxVB2Ot9/LxFZb4yJK72tPU8A64AO\nItJGRLyAMcDiUgePAd4ERpS6+XsBi4D3St/8bU8FiFURfy2w1Y6yVIsrIhsT2yqIl5b9RlZe+f2I\nlVLKWVUYAIwxBcC9wFJgB/CJMWabiDwpIiNsmz0P+AH/FZFNIlIcIG4CBgC32JZvsnUNBfhQRLYA\nW4BQ4F/Vd1nlExEeuaoTKadzefen/bV1WqWUqlPs6qtkjPkG+KbUssdKfL6sjP0+AD4oY92l9hez\n+sW2CmZol8a8uWofkwe2xdvD3ZHFUUqpWucSI4HLckNsCzJzC/g1Mb3ijZVSysm4dADo1SYYEViz\n74Sji6KUUrXOpQNAo4ZedGoSQPx+DQBKKdfj0gEAoHebYNYfPEVegY4JUMpZ+Pn5AXD48GFuuOGG\n824zaNAgKupWPmPGDLKyss5+r2x66bLUlbTTLh8A+rQNISe/iM1JDslOoZSqQc2aNWPhwvMOP7JL\n6QDgbOmlXT4A9GoTDED8/pMOLolS6nymTZvGrFmzzn4v/vWcmZnJkCFD6NGjB127duWLL774w74H\nDhwgKioKgOzsbMaMGUPnzp0ZNWrUObmA7r77buLi4oiMjOTxxx8HrARzhw8fZvDgwQwePBj4Pb00\nwIsvvkhUVBRRUVHMmDHj7PnqU9rp+pGyrgYF+3rRqYk/a/adYMrg9o4ujlJ125JpcHRL9R6zSVe4\n8pkyV48ePZq//OUvTJkyBYBPPvmEpUuX4uPjw6JFiwgICOD48eP06dOHESNGlJnk8fXXX6dhw4bs\n2LGDzZs306NHj7Prpk+fTnBwMIWFhQwZMoTNmzczdepUXnzxRZYvX05oaOg5x1q/fj1z5swhPj4e\nYwy9e/dm4MCBBAUF1au00y7/BABWO0DCgVPka24gpeqcmJgYUlJSOHz4ML/++itBQUFERERgjOGR\nRx4hOjqayy67jOTkZI4dO1bmcVatWnX2RhwdHU10dPTZdZ988gk9evQgJiaGbdu2sX379rIOA8DP\nP//MqFGj8PX1xc/Pj+uuu+5s4rj6lHba5Z8AAHq3DWHe6oNsTkontlWQo4ujVN1Vzi/1mnTjjTey\ncOFCjh49yujRowH48MMPSU1NZf369Xh6etK6devzpoGuyP79+3nhhRdYt24dQUFB3HLLLRd0nGL1\nKe20PgFQsh1Au4MqVReNHj2aBQsWsHDhQm688UbA+vUcHh6Op6cny5cv5+DBg+UeY8CAAXz00UcA\nbN26lc2bNwOQkZGBr68vgYGBHDt2jCVLfs9mX1Yq6v79+/P555+TlZXFmTNnWLRoEf3796/0dTk6\n7bQ+AQChft50CPdjzb6T3DPI0aVRSpUWGRnJ6dOnad68OU2bNgVg/PjxXHPNNXTt2pW4uLgKfwnf\nfffd3HrrrXTu3JnOnTsTG2tNDdutWzdiYmLo1KkTERER9O3b9+w+kydPZtiwYTRr1ozly3+f96pH\njx7ccsst9OrVC4Dbb7+dmJiYcqt7yjJv3jzuuususrKyaNu2LXPmzDmbdjo9PR1jzNm00//4xz9Y\nvnw5bm5uREZGVjntdIXpoOuS6koHfT7/+Hwrn21IYtPjQ/F01wcjpYppOuj6pbrTQbuE3m2DOZNX\nyNZkzQuklHINGgBsercJAXQ8gFLKdWgAsAnz96ZdmC/xmhhOqT+oT1XFrqyy/500AJTQp20I6w6c\n0rmClSrBx8eHEydOaBCo44wxnDhxolKDw7QXUAm924bwYfwhth/JILqF8+T7UKoqWrRoQVJSEqmp\nqY4uiqqAj48PLVq0sHt7DQAl9LGNB1iz74QGAKVsPD09adOmjaOLoWqAVgGVEB7gQ9tQX+L3aUOw\nUsr5aQAopXfbYNbuP0lhkdZ3KqWcm10BQESGicguEdkjItPOs/5+EdkuIptF5AcRaVVi3SQR2W17\nTSqxPFZEttiOOVPKSuFXy/q0DeF0bgE7jmQ4uihKKVWjKgwAIuIOzAKuBLoAY0WkS6nNNgJxxpho\nYCHwnG3fYOBxoDfQC3hcRIqzrb0O3AF0sL2GVflqqkHxeACdJ1gp5ezseQLoBewxxuwzxuQBC4CR\nJTcwxiw3xhRPm7MGKG6GvgJYZow5aYw5BSwDholIUyDAGLPGWH3L3gOurYbrqbImgT60DmnIGm0H\nUEo5OXsCQHMgscT3JNuystwGFKfTK2vf5rbPFR5TRCaLSIKIJNRWN7TebUJYd+AkRdoOoJRyYtXa\nCCwiE4A44PnqOqYx5i1jTJwxJi4sLKy6Dluu3m2DSc/OZ+fRqk23ppRSdZk9ASAZiCjxvYVt2TlE\n5DLg78AIY0xuBfsm83s1UZnHdJSYllYzxZZknSheKeW87AkA64AOItJGRLyAMcDikhuISAzwJtbN\nP6XEqqXAUBEJsjX+DgWWGmOOABki0sfW+2ci8McZnR2kVXBD/L092KKZQZVSTqzCkcDGmAIRuRfr\nZu4OzDbGbBORJ4EEY8xirCofP+C/tt6ch4wxI4wxJ0XkKawgAvCkMaa4dfUeYC7QAKvN4PdpeBzM\nzU3o0iyArcnaFVQp5bzsSgVhjPkG+KbUssdKfL6snH1nA7PPszwBiLK7pLWsa/NA3l9zkILCIjx0\nghillBPSO1sZopoHkltQxJ7UTEcXRSmlaoQGgDJENQ8EYEuStgMopZyTBoAytA31xdfLXaeIVEo5\nLQ0AZTjbEHxYG4KVUs5JA0A5opoHsv1whmYGVUo5JQ0A5YhqFkh2fiH7tCFYKeWENACUo2sLW0Ow\ntgMopZyQBoBytAvzw8fTTQOAUsopaQAoh7ub0KVpANt0RLBSyglpAKhA1+aBbDucrqmhlVJORwNA\nBSKbB3Imr5B9x884uihKKVWtNABUoKttRPC2w9oOoJRyLhoAKtAh3A9vDzdNCaGUcjoaACrg4e5G\np6YBbNUnAKWUk9EAYIeuza2eQNoQrJRyJhoA7BDVLJDTuQUcPJnl6KIopVS10QBgh+LU0JoZVCnl\nTDQA2OGixv54ubtpAFBKORUNAHbw8nCjYxN/bQhWSjkVDQB2imoeyNbkDIzRhmCllHOwKwCIyDAR\n2SUie0Rk2nnWDxCRDSJSICI3lFg+WEQ2lXjliMi1tnVzRWR/iXXdq++yql9U8wDSs/NJPJnt6KIo\npVS18KhoAxFxB2YBlwNJwDoRWWyM2V5is0PALcADJfc1xiwHutuOEwzsAb4rscmDxpiFVbmA2lI8\nInjr4XRahjR0cGmUUqrq7HkC6AXsMcbsM8bkAQuAkSU3MMYcMMZsBorKOc4NwBJjTL3sS3lRY388\n3ERTQyulnIY9AaA5kFjie5JtWWWNAeaXWjZdRDaLyEsi4n2+nURksogkiEhCamrqBZy2evh4unNR\nY3/tCaSUchq10ggsIk2BrsDSEosfBjoBPYFg4KHz7WuMecsYE2eMiQsLC6vxspana/NAtiana0Ow\nUsop2BMAkoGIEt9b2JZVxk3AImNMfvECY8wRY8kF5mBVNdVpUc0DOJWVT3KaNgQrpeo/ewLAOqCD\niLQRES+sqpzFlTzPWEpV/9ieChARAa4FtlbymLWueESwZgZVSjmDCgOAMaYAuBer+mYH8IkxZpuI\nPCkiIwBEpKeIJAE3Am+KyLbi/UWkNdYTxMpSh/5QRLYAW4BQ4F9Vv5ya1blpAKF+Xry+ci+FmhhO\nKVXPSX2qz46LizMJCQkOLcOXvx7mz/M38ujVnbm9f1uHlkUppewhIuuNMXGll+tI4EoaHt2UIZ3C\n+c93v5Go2UGVUvWYBoBKEhGeujYKN4FHFm3RHkFKqXpLA8AFaNaoAQ9d2Ymfdh9n0cbKdohSSqm6\nQQPABZrQuxU9Wjbiqa+2cyIz19HFUUqpStMAcIHc3IRnro8mM7eAp77aXvEOSilVx2gAqIKLGvtz\nz6D2fL7pMMt3pTi6OEopVSkaAKronsHtaB/ux6OLtnImt8DRxVFKKbtpAKgibw93nrmuK8lp2fx9\n0RYdIKaUqjc0AFSDuNbB3H/5RXy+6TD3LdhIfmF5WbGVUqpuqHBCGGWfqUM64OXhxjNLdpKTX8ir\n43rg4+nu6GIppVSZ9AmgGt01sB1PjYzk+x0p3DZvnbYJKKXqNA0A1ezmi1vzwo3dWL33BBNnryU9\nO7/inZRSygE0ANSAG2Jb8Oq4HmxOSmPc22t0oJhSqk7SAFBDruralLdujmNPSiZjNQgopeogDQA1\naHCncObc2pNDJ7MY/048p87kObpISil1lgaAGnZJu1DenhjHvuNnmPBuPOlZ2iaglKobNADUgv4d\nwnjz5lh2H8tk4ux4MnI0CCilHE8DQC0Z3DGc18b3YPuRDCbNXkumdhFVSjmYBoBadFmXxrwytgeb\nk9K5dc5asvI0CCilHEcDQC0bFtWEl8d0Z/3BU9w6Z522CSilHMauACAiw0Rkl4jsEZFp51k/QEQ2\niEiBiNxQal2hiGyyvRaXWN5GROJtx/xYRLyqfjn1w/DoZrw0ujsbDp1i5Kyf+e3YaUcXSSnlgioM\nACLiDswCrgS6AGNFpEupzQ4BtwAfnecQ2caY7rbXiBLLnwVeMsa0B04Bt11A+eutkd2bM/+OPmTm\nFnLtrF9YsuWIo4uklHIx9jwB9AL2GGP2GWPygAXAyJIbGGMOGGM2A3alwRQRAS4FFtoWzQOutbvU\nTiKudTBf/bkfHZv4c/eHG3h+6U5NJ62UqjX2BIDmQGKJ70m2ZfbyEZEEEVkjIsU3+RAgzRhT3Apa\n5jFFZLJt/4TU1NRKnLZ+aBLow4LJfRjTM4JZy/dy27x1mj9IKVUraqMRuJUxJg4YB8wQkXaV2dkY\n85YxJs4YExcWFlYzJXQwbw93nrk+mumjovhlz3FGvvoz328/hjH6NKCUqjn2BIBkIKLE9xa2ZXYx\nxiTb3vcBK4AY4ATQSESK5yOo1DGd1fjerVgwuQ9FBm5/L4GrZv7MV5sPa7WQUqpG2BMA1gEdbL12\nvIAxwOIK9gFARIJExNv2ORToC2w31k/b5UBxj6FJwBeVLbwzim0VzI9/HciLN3Ujt6CQez/ayOUv\nreTT9Uk605hSqlqJPdUMInIVMANwB2YbY6aLyJNAgjFmsYj0BBYBQUAOcNQYEykilwBvYjUOuwEz\njDHv2o7ZFqtBORjYCEwwxpSbMjMuLs4kJCRc4KXWP4VFhm+3HuWVH3ez8+hpIoIbMPXSDlzXowXu\nbuLo4iml6gkRWW+rij93eX2qZ3a1AFDMGMMPO1KY+eNuNiel06mJPw8N68SgjmFYHaqUUqpsZQUA\nHQlcD4gIl3VpzBdT+vLquBiy8wu5de46xr69hl8T0xxdPKVUPaUBoB4REYZHN2PZ/w3kiRGR7D6W\nychZvzDlww0cOpHl6OIppeoZDQD1kJeHG5Muac2KBwcx9dL2/LgzhStmrOK91Qco0h5DSik7aQCo\nx/x9PLl/aEd+fGAgPdsE89gX27h5djzJadmOLppSqh7QAOAEmgY2YN6tPXl6VFc2HUpj2Eur+CQh\nUQeSKaXKpQHASYgI43q35Nu/DKBLswD+tnAzt89LICUjx9FFU0rVURoAnExEcEPm39GHx4Z34ec9\nx7ny5Z/YcOiUo4ullKqDNAA4ITc34U/92vD11H74+Xgw9q01fL1Z000rpc6lAcCJtQ/3Z9E9fYlq\nHsiUjzbwxsq92i6glDpLA4CTC/b14sPbezM8uinPLNnJI4u2aE4hpRQAHhVvouo7H093Zo6JoXWI\nL68u30PSqWxmje9BgI+no4umlHIg1w4Ah+Jh93dw6aPg5Dl13NyEB67oSMuQhjzy2RaGz/yZ3m2C\naRHUkIjgBkQENyQiqCHh/t64aaI5pVyC6waAzBT4eDycSYXW/aDdYEeXqFbcFBdBi0YNmPHDblbt\nTuVYxrkJWH083RjfuxVTh3QgsIE+ISjlzFwzABgDX0yBnAxoEATxb7pMAAC4pH0ol7QPBSAnv5Dk\ntGwST2aRdCqbDQdPMfuX/Xy2IYn7h3ZkbM8IPNy1qUgpZ+Sa6aDj34IlD8KVz0HWCVj5HEzdAMFt\nq35sJ7DtcDpPfrmd+P0nuaixH/8Y3oX+HZxzOk6lXIGmgy6WsgO+exQ6DIVekyHuT+DmDmvfcXTJ\n6ozIZoEsmNyHNybEkpNfxM3vruW2uetYu/+kTk+plBNxrSqg/BxYeBv4BMDIWVbDr38TiBwFG9+H\nwY+At5+jS1kniAjDopowuFMYc385wCs/7uGHnSkENfTk0k6NubxLY/p3CMXX27X+hJRyJq71f+8P\nT0DKNhj3X/AL/31577tgy3/h1/nQ6w7Hla8O8vZw586B7RjXuyWrfjvOsu1HWbb9KJ9uSMLLw41+\n7UMZ37slQzo3dnRRlVKV5DptALu/hw+vh153wlXP/XH9W4MhLxPuiQc316sZq4z8wiLWHTjJsu3H\n+G7bMZLTsrl7UDseGNpR5ypWqg5y7TaAM8fh87shvAtc/sT5t+l9Fxz/DfYtr92y1UOe7m5c0i6U\nx6+J5McHBjK2V0teX7GXW+euIy0rz9HFU0rZya4AICLDRGSXiOwRkWnnWT9ARDaISIGI3FBieXcR\nWS0i20Rks4iMLrFurojsF5FNtlf36rmkUoyBL+6FnHS4/h3wbHD+7SKvBd9wq0uospu3hzv/vq4r\nT4/qyuq9xxnx6i/sPJrh6GIppexQYQAQEXdgFnAl0AUYKyJdSm12CLgF+KjU8ixgojEmEhgGzBCR\nRiXWP2iM6W57bbrAa6joAiD2Frj6BWgcWfZ2Ht5Wj6DdS+HE3hopijMb17slCyZfTE5+Ide99j/N\nPqpUPWDPE0AvYI8xZp8xJg9YAIwsuYEx5oAxZjNQVGr5b8aY3bbPh4EUoPY7lHccBj0mVrxd3K3g\n5glr3675Mjmh2FZBfPXnfnRuGsCUjzbw3Lc7NfuoUnWYPQGgOZBY4nuSbVmliEgvwAso+fN6uq1q\n6CUR8S5jv8kikiAiCampqZU9beWc7RL6AeSertlzOanwAB/m39GHsb1a8tqKvTz06Wa7xg4UFBbx\n1ebDpGfl10IplVJQS43AItIUeB+41RhT/JTwMNAJ6AkEAw+db19jzFvGmDhjTFxYWC08PPS+C/JO\nw6b5NX8uJ+Xl4cbTo6KYOqQDnyQkMXX+RvIKyk5BnZKRw7h34rn3o41MmrOWrLyCWiytUq7LnnEA\nyUBEie8tbMvsIiIBwNfA340xa4qXG2OKK4lzRWQO8IC9x6xRLWKheRzEvwFhHW0LS/yCFTdrvVdD\nhxSvvhAR7r/8Ivy9PZj+zQ7O5BXw+vhYGni5n7Pd//YeZ+r8TZzJLeC2fm2Y88t+7v1oI2/dHKs5\niJSqYfYEgHVABxFpg3XjHwOMs+fgIuIFLALeM8YsLLWuqTHmiIgIcC2wtVIlr0l97oZPb4P3Rpx/\nfYNg6H2nlUqiYXDtlq2euWNAW/x8PHhk0RYmzVnLu5Pi8PfxpKjI8PrKvfznu120CfVl/h296dDY\nnzahvjz6+VYeWbSFZ6+PRpw8TbdSjmTXQDARuQqYAbgDs40x00XkSSDBGLNYRHpi3eiDgBzgqDEm\nUkQmAHOAbSUOd4sxZpOI/IjVICzAJuAuY0xmeeWotmRwFTEGktdDQYlUycU3otzTkDAHflsCng0h\n5ma4eAoEtar5ctVji389zP0fb6JLswBeHhPDk19uY/muVK7p1oxnrut6TkqJF5f9xswfdnPv4PY8\ncEXHco6qlLJHWQPBXGckcHVL2Qn/mwmbPwFTBFHXwSVToWm0o0tWZ/2w4xh3f7iBvIIiPN2Fx4Z3\nYUKfVn/4lW+M4ZFFW5i/NpEnRkQy6ZLWjimwUk5CA0BNSU+GNa/B+rlWKom2g+CSP0O7IU4/y9iF\nWL33BK+v3Mv9l19E94hGZW5XUFjEXR9s4Iedx5g1rgdXdW1ai6VUyrloAKhp2WlWEIh/A04fsdJO\nXHwvdL3BGmRmr4I8a5aygGYuH0Cy8wqZ8G48W5LSmXtrz7OT2CilKkcDQG0pyIOtn8L/XrEyj/rZ\nxhb4hYNvKDQMgYa2dzd3SN0Jx7ZBynY4th1O7IaiAmu+gmHPQEg7R1+RQ6Vl5XHjG6tJPJXFOxN7\n0q+DBgGlKksDQG0zBvb+CKtfhYOroSC7/O0DW0LjLtaTg7sXrJ4FhblWu0L/v7p0t9PjmblMeCee\nfcfP8OaEWAZ3Cq94J6XUWRoAHC0vC7KOW1NQnjlhvRfmQmhHCO9sTVJT0umjsOwx2PwxBEbAFU9D\n52tctlro1Jk8bp4dz66jp5k1rgdDI5s4ukhK1RsaAOqrg/+Dbx6EY1uh7WAY+BC07FO1QFCQZwUf\nb//qK2ctSM/OZ+LstWxLTuflMTFcHa0Nw0rZQwNAfVZYAAnvwo/TITcdglpD9GjrVV4bQX62ldk0\ndSek7vr9/aQtHdOIV6H72Fq5hOpyOiefW+esY8OhU/znpm6Mimnh6CIpVedpAHAGuZmw40vYvAD2\nrQQMtOgF3UZbbQgn98KJPbbXXkhP4mwaC3GD4LYQ1slKcZG4Fg78BEP/ZXVbrUjyesg6CT6NoEEj\n8Am0Pnt41eQVn9eZ3AJun5fAmv0nuKN/W9qE+hLU0ItgXy+CfT0J9vUmsIGnzk6mlI0GAGeTnmyb\nx3gBpO74fbl3IIS2h+B2ENLeekII72x9LtkdtSAXFt0J2xZB3/vgsifOX610+ih8+zBs++z85fBs\nCF1GwsjXanUqzey8Qv48fwPf70g57/pwf29m39KTqOaBtVYmpeoqDQDOyhirG2lepnWTbxhif/tA\nUSEs+Rusewe6T4BrXgZ3W0qGoiJYPxu+f8IKFgMesNogctKsMQ85tteJvfDrfBjwN7j07zV3nWXI\nzivkVFYeJ8/knX0/eSaPdwJcv5YAABvISURBVH7aT0ZOPnNv7UVsq6BaL5dSdUlZAcCeZHCqLhOB\nJlEXtq+bO1z1AviGwYp/Wz2Tbpxj3dS/+gskrYM2A2H4S2W3NRhjHWfVc9CkK3QpI4FesX0rra6x\nQ/9VItvqhWvg5U4DrwY0a3TuVJ9DI5sw/u013PxuPO9MiuOSdjp+QKnS9AlAWda+bfU2CmkHJ/dD\ngyAY9m/oemPFTxQFuTD3amsg2+3fW+MZzmfLQlh0FxTlW08qEz6DZjUzFTRY8wyMfyeeQyezeEPH\nDygXVtYTgCZcV5Zed8ANs606/5jxcO86iL7JvuokD2+46X3w9oMFY63G4tJWv2al2I7oBXcsB09f\nmHeN1c21hoQH+PDxnRfTPtyPye8nsGSLzlOsVEkaANTvoq6DaYkw4pXKz3MQ0BRGfwAZh60bfVGh\ntbyoyBrQtvRhayDbhM+geQ/40xLwawzvXwe7v6/+a7EJ9vXiozv60LV5IFM+2sBnG5Jq7FxK1Tda\nBaSq1/p58OVUK4XFkMfgi3utbqtxt8FVz1vtBcXOHIf3R0HKDrj+bStnUg0p7jq6et8JQny96NTU\nn05NAujYxJ/OTQLo0NgPH0/3ig+kVD2kvYBU7fn6r1bPosZR1gjmSx+F/g+cvzopOw0+Gg1Ja+Ga\nmdDj5horVk5+IZ8kJLI1OZ2dR0/z27HT5ORbcxW7CfRtH8oNsS0Y2qXJH6auVKo+0wCgak9BHrw3\nEhLXWF1Le0wsf/u8M/DxBCt5nqcvuHlYTwpuHuDuab33vx9ib6nWYhYWGQ6eOMOuo6f5NSmdL389\nTHJaNv7eHlwd3ZQbYlsQ2ypIp6VU9Z4GAFW78rKs9oDQ9vZtX5AL8W9C5jGr/aCowOotVFQAR7dY\nKSzuWQPBbWqsyEVFhvj9J1m4PoklW4+QlVdI65CGXBvTnGu6NaNdmF+NnVupmqQBQNVfGYfhlTho\n0x/GfVwrpzyTW8CSrUdZuD6R+P0nMQa6NA1geLemXBPdjIhg103PreofDQCqfvtlJiz7B4xdAB2v\nrNVTH8vI4evNR/hy82E2HkoDoHtEIy7rHE7nplZDcvNGDbSqSNVZVQoAIjIMeBlwB94xxjxTav0A\nYAYQDYwxxiwssW4S8Kjt67+MMfNsy2OBuUAD4BvgPlNBYTQAuLDCfHijH+RnwZS14Nmg4n1qQOLJ\nLL7ecoQvfz3MtsMZZ5f7eXtwUWM/OjYJICaiEdfGNMfLQ3tZq7rhggOAiLgDvwGXA0nAOmCsMWZ7\niW1aAwHAA8Di4gAgIsFAAhCHlZZyPRBrjDklImuBqUA8VgCYaYxZUl5ZNAC4uP0/wbzh1pwIgx9x\ndGk4nZPPb8dOs/PoaXYd/f09PTuf9uF+PDkyUlNQqDqhKrmAegF7jDH7bAdaAIwEzgYAY8wB27qi\nUvteASwzxpy0rV8GDBORFUCAMWaNbfl7wLVAuQFAubg2/a3UFD/PqHguhFrg7+NJbKtgYlv9PmjO\nGMPyXSk8vngb496OZ0S3Zjx6dWfCA3wcWFKlzs+eZ9TmQGKJ70m2ZfYoa9/mts8VHlNEJotIgogk\npKam2nla5bQuf8qaM3nJQ1YiujpGRLi0U2OW/d9Apg7pwLdbj3Lpf1Yy++f9FBSW/n2klGPV+UpK\nY8xbxpg4Y0xcWFiYo4ujHC2gKQx+GPYsg51fO7o0ZfLxdOf+yy/iu/8bQGyrIJ78ajvDX/mZd37a\nx86jGdSnzhfKedlTBZQMRJT43sK2zB7JwKBS+66wLW9Rarm9x1SurtedsPFD+HYatLsUvOpul8zW\nob7MvbUnS7cd5cVlv/Gvr63Je8L9venXPpR+HULp1z5Uq4iUQ9gTANYBHUSkDdZNegwwzs7jLwWe\nFpHiGTmGAg8bY06KSIaI9MFqBJ4IvFK5oiuX5e4BV78Ac66Ez+6ANgOsxHJ+jcG/Mfg1qVNBQUQY\nFtWUYVFNOZyWzc97jvPT7uOs+C2VzzZav3su69yYaVd2pH24v4NLq1yJvd1Ar8Lq5ukOzDbGTBeR\nJ4EEY8xiEekJLAKCgBzgqDEm0rbvn4DiLhvTjTFzbMvj+L0b6BLgz9oNVFXK0r9D/BvWaOHSgttZ\ng8ZCO9R+uexUVGTYfiSD77YfY/bP+8nOL2R0zwj+clkHwv31iUBVHx0IppxTUZE1k1nmMcg8CqeP\nwekjVmAQN5i4GMI7ObqUFTqRmcsrP+7hgzUH8fJwY/KAttzRvy2+3tZDelGR4UhGDvtTz7D/xBny\nCoqIadmIqGaBOt5AVUgDgHItqbusCWeKCmHSl2XPUnahjAFTdG5662qw//gZnl+6k2+2HCXM35vY\nlkEcOHGG/cfPkFvwx15E3h5udGvRiNjWQcS1CiKuVTCBDT2rtUyq/tMAoFzP8T1WECjIgYlfQNPo\n6jluyk744h7ITIGb3rMmuKlm6w+e4j/f7eJoeg5tQn2tV5gvbUKsdzcR1h88xfqDp0g4eIptyekU\nFBl8PN14/7be9GxdyQl9atLB/0GzGIeN3lYaAJSrOrkP5l4DeZkw8XPrRnShigqtCe1/nA5evuDZ\nEM6kWimvu4+tvjJfgOy8Qn5NSuPhz7ZwOiefxff2o1mjOnDDPboV3ugLPe+wGu6VQ+icwMo1BbeF\nW78GnwCYNxKS1l/YcY7vhtlXWNNbdrgcpsTDnSutOY4/vwuWTLPyFZXl2HZrXuS0xLK3qYIGXu70\naRvC2xPjyM0vYvL7CWTnFdbIuSplsy17a8Js68lJ1Sn6BKBcQ1qilUfo9DErKHj5WpPYe/mCl7/1\n7tcYAprZXs2td88GsOZ1+PEp8PCBq/8DUdf/PrtZYYGVpXTNa9C6P9w4F3xt+X9yM2HbZ9Y0mcm2\nv1tPX2sgW++7rMluasCPO49x27wEroluxstjujsuS2lREcyIgsAIa9rPiF4wYWHF+6lqp1VASqUn\nw6rnrLmI8zKtG3TeGdvnDMhJ/+M+Hj5WG8JFV8I1M8C/yfmP/esC+PI+8A2DoU/B3uWw9VPr2KEd\nIXYStOoLy5+G3Uut6TKHv2TdFGvAayv28Ny3u3hoWCfu7tcCDq2Bpt2gQaMaOd95FSfvu/5dq2fW\nd4/ChE+h/WW1VwYFaABQqmL5OdaNKuOw7ZVsfW/R89xf/WU5vBEWTICMJPBoAFHXQY9J1k2+eF9j\nYOdXVi6jjGRrmsshj0PD6m20Ncbw9w9+JGzXh9zjtxLvnOPW00fMeOvpozYS6X1xL2xbBA/stnpL\nzeoNHt5w1y/WYD5VazQAKFUbzpyw5kJu3Q98AsveLvc0rHjGql5qEGQ1kEaOqp4yHN4E8W9gtn6K\nFOaxysRw0dA7aJLyE2xZiCkqIKPl5axrMoaVuRdhBIIbehHs60WQr/Ue7OtFm1BfGnpd4I06Pwde\nuMiavOe6N61l2xfDJzfD1S9Cz9uq51qr24//sv79xn1c7V18HUkDgFJ10ZHN8OVU6+mh2zi46jnw\nvsB0EMf3WMc6+Iv1a7/7OI51nsTVHx3Fz9uDHq2COJJ4gL5pnzPObRnBksl204Z/u93Bzzmt/5Bc\nNdTPi38M78KIbs0q345QfLOf8Bm0H2ItMwbmDofUnTB1Q/kB0hF2fg0LbFlurnsHom90bHmqkQYA\npeqqwnxY+Sz89B9o1BKue7vybQPpSfDuFdaMaf3/CjETztb3rztwktvmrsPH053IZgFENguka2Mv\nemYsIyjhZcQUUnjXatLx5eSZPE6eySP1dC5v/bSPXxPTGHBRGP8aGUXLkErkV/p4AhyKh/t3nFvd\nc3gTvDUILvmz1VZSV2Qchtcvsf79CwugIBumrHOaqioNAErVdQdXw2eTrbaBgX+D/g/YdwPKOgmz\nh1ntFbd8ZTX2llJYZHB3O8+v+MMb4Z3LIPI6uP7tP+zzwZqDPPftTgqN4b4hF3F7/zZ4ulfQezz7\nlFX9E3cbXPnMH9d/fg9s+a81tWdwm3PXHd8Nv863us0W5tle+VCUb332awxjPrLaEqpLUSG8NxKS\nN8Cdq+D4b7BgLIx4BXpMrL7zVOTUAWsE+0VXVPuhdRyAUnVdq4vh7p+tBucV/7aynZ7cV/4+uZnw\n4Q3WzWPs/PPe/IHz3/zBGhg34EHY8gls/+IP+0y6pDXf/3UgAzqE8ey3O7nmlZ+J33eCoqJyfjhu\nX2zdrMuqQrn0H+DmYY2pACtgrHsH3h4Cr8bBzy9BeqLVK6uo0LrZNwi2eljt+R42fVj+v0ll/fwS\nHPgJrnoeQttb7RbNY2Hlc1CQW73nKs/iqTB/DKT+Vmun1CcApeqizf+Fr++3uqD2mmxV65TuKVSQ\nCx+Nhv0rYfQH0OnqCztXYb71FJB2yBrg5hd+3s2WbjvK419s42hGDqF+3gzoEMrAjmH0ax9KiF+J\nX+Rzh1tPI/cmlN1zasWzsOJpaH+5Vf7CPAiPtEZUd73JSutdmjFWOc+kwJ83VM84isS11tNT5LVW\nd9Xi8u79Ed4fBVc+D70nV/08FTm80aoaA+v6Sz2NVZVWASlV32QchuXTrclvfAKsINDrTvD0sX4Z\nf3qb1c1y5Cyrzr8qUnbCmwOsBtsxH5V5487MLWDJliOs2n2cn3enciorHxHo2jyQi9uG0ITj3LL2\nGta3uZPN7e7Cw13wdHfjknYhtArx/f1AeVnwWh9rnETXm6wbf5Poirva7voW5o+Ga9+oevqNnHR4\no5/1+a6fz22ULm6wPrEbpm6q+fkl/nur9XQTdT1smAf3xEPYRdV2eA0AStVXx7bB9/+E3d9Zo2ov\nfRQS4630Cpc/BX2nVs95/vcqfPd3GPmaNV6gAoVFhq3J6az8LZVVv6WyMTGN22UxD3vOZ2Duixw0\n5w6a690mmBvjIriqaxOre2l+tlUVVJlf8sbAG/2tJ6Mp8RfeVdMYWwD9HP60FCJ6/nGbg6thzjC4\n/Enoe9+Fnccepw7AzBirYfySqTCjK3QaXq1PARoAlKrv9q200k4c+dX63vc+6+ZUXYqKrJG7R7fA\n3f+DRhEV71OCMQZe74vxbED2xKUUFBkKiwwZ2fl8veUI/01I5MCJLHy93Bke3Ywb41oQ2yqo8l1M\nty2C/95ipd240LETGz+0Mrpe+qjVBlKWD66H5PVw32brKawmfPMgJMyBv2yx5rz+7h9W0sEpa6tt\nQiMNAEo5g6IiK7/Q6SNw8b0VV5lU1qkD8HpfK8X1zV9Yx087ZKV0PvQ/61dx3hnofSf0vN3Kp1Ts\n2HZ4/eIy682NMSQcPMUn6xL5essRsvIKaR3SkFExLRgV09z+bqZFhbZRxT5w10+V/zdISoC5V1sj\nvCd+Uf5TRPIGeHswDHoYBk2r3HnsceYEvBQJXa+3qvIAMlPh5WjofA1c91a1nEYDgFLKPuvnWnmN\nWve3eiFlWPMW4xMIEX2s6pf9K62eORdPsRqpfQKsaqpfZsJfd4FfWLmnOJNbwNdbjrBoQzKr950A\noGfrIK7r0YKrujYlsEEF1UKb5ltZWMd+DB2H2X9tpw7CO0Os5H+3//B74r7yLBgP+1fBfb9We8oO\nVjxj9fiashbCOv6+vJqfAjQAKKXsY4xVxZK4Flr2gVaXQMuLIbwLuNl6jieusxLr7f7OCgx9psCG\n9yC8c6UzfianZfP5xmQWbUxmT0omXh5u9GsfSmyrIGIiGhEd0Qg/71LjIQrz4ZUe4BsOt39v31NA\nTro1WC7jsLWPvY2sx7Zbg8T63geXP1GpaytXXpb16z+iN4xbcO66an4K0ACglKp+yRtg1Quw62vr\nexVSKBhj2JKczmcbkvlpdyp7U88A4CZwUWN/erQKok/bEK6MamINRkuYDV/9n1WN03ZQ+QcvzIeP\nbrJ+yU/4DNoOrFzhPr0ddnwF178DnYdf0PX9wdq34ZsH4NZvrTEgpX33KKyeZY1IDm1fpVNVKQCI\nyDDgZcAdeMcY80yp9d7Ae0AscAIYbYw5ICLjgZItLNFAD2PMJhFZATQFsm3rhhpjUsorhwYApeqo\no1usFNi97wIPr2o5ZHpWPhsTT7HxUBobDp1iU2Iap3MKiAhuwJ8Hd2BUdCier8ZASHtrBHRZjLEC\nxfo5MOJV6HFz5QuTccQapHVkk9UVd+hTVRuNXFhgPcH4NYbbvjv/E0xmqtUjqMvI3xPqXaALDgAi\n4g78BlwOJAHrgLHGmO0ltrkHiDbG3CUiY4BRxpjRpY7TFfjcGNPO9n0F8IAxxu47ugYApVxXUZFh\n+a4UXv5hN5uT0okIbsDM1muI2f6s1ZWzZZ/z77h6Fix9BPr9H1z2zwsvQEEufP8ErJlljVm4Yc6F\n/zLf+iks/BOM/rD8J4qlf7cmG6riU0BVAsDFwD+NMVfYvj8MYIz5d4ltltq2WS0iHsBRIMyUOLiI\nPG3tZv5u+74CDQBKqUoyxgoEM77fze6kY6z2+QtnQqPZ0P9tsnLzyMnOITcvh9ycHJqeXMv1+x/D\ndB6B241zf2/DqIpd31oN0AV51qQ+3UZXvM+5FwBvDbR6U01ZV36ZMlNgRnSVnwLKCgD2pLprDpSc\nyDQJ6F3WNsaYAhFJB0KA4yW2GQ2MLLXfHBEpBD4F/mXOE41EZDIwGaBly5Z2FFcp5cxEhEs7NWZw\nx3CW70ph8eJRTDw+j8afReIhRX/YflNRO+7ZdRNXL9nJ6J4RtA+/wHTbxToOsya1+fR2WDQZ9q2w\nkt7Zm956/0prLMc1MysOSH7h1twJa16zxitUsS2gtFrJdSoivYEsY8zWEovHG2OSRcQfKwDcjNWO\ncA5jzFvAW2A9AdRGeZVSdV9xIDBtnubEsmCkqAAPLx88vbzx9PLBw8uHIndvMr0uptuvGcz55QBv\n/7Sf2FZBjO4ZwfDophc+4U1gc5j0Jax63uoNtWeZNbNb9/Fl39SNseYc+O5Rq+4/2s4nh773WWMe\nqrsLKrVUBSQiLwGpxpinyzjHLUCcMebe8sqiVUBKqQt1PDOXzzYk8fG6RPamnsHf24PrY1swoU8r\n2of7VXyAshzeBEv+ZqXnaNYDrnzu3NQSxsDeH2yzjW20Gq2v/k/FPZeqUVXaADywGoGHAMlYjcDj\njDHbSmwzBehaohH4OmPMTbZ1bljVQ/2NMftKHLORMea4iHgC84HvjTFvlFcWDQBKqaoqHpH8wZqD\nfLPlCPmFhr7tQ7i5Tysu69wYj4rmOzj/Qa05Dr77B2QetWZ3u+yfcHIv/PCUNYo6sCUMegiix9T6\nRDNV7QZ6FTADqxvobGPMdBF5EkgwxiwWER/gfSAGOAmMKXGzHwQ8Y4zpU+J4vsAqwNN2zO+B+40x\nheWVQwOAUqo6Hc/M5eN1iXwUf4jktGyaBPhwVdemtAhqQJNAH+sV4EO4v7d9gSH3tDWz2+pZgEBh\nrlXdM+BBa3KZ6pzIphJ0IJhSSpWhsMjw484U3l9zkDX7TpBXcG5jsptAqxBf7hvSgZHd7Zgj+cRe\n+N9Mq7on7raaTyddAQ0ASillB2MMaVn5HEnP4VhGDkfScziakcPynSlsSU6nZ+sg/jkikshmdWxS\n+3JoAFBKqSooKjJ8kpDIc0t3kZaVx/jerfjr0Ito1LB6Rj7XJJ0TWCmlqsDNTRjTqyXL/zqIiRe3\n5sP4gwx+YQUfxh8kv/CP4w/qA30CUEqpC7DjSAaPL97G2v0nCfP3ZnRcBGN7t6R5owaOLtofaBWQ\nUkpVM2MMK35L5YPVB/lxVwoCXNopnPF9WjGgQxjubtU8Yc8FqkoqCKWUUuchIgzuGM7gjuEkncpi\n/tpDfLwuke93pNAiqAFtQn3JLSgir/hVaL03auhJpyb+dG4aQKcmAXRu6u+QtgR9AlBKqWqUV1DE\nd9uP8t+EJDJy8vFyd8PLw+33dw83jmfmsuPIaU6eyTu7X5MAHy5pH8LjwyMJbFjBjGiVpE8ASilV\nC7w83Bge3Yzh0c3K3c4YQ+rpXHYcPc3OIxlsP5LBl78eZv3BU7x5cyydmtTQJPQl6BOAUkrVEesP\nnuTuDzZwOqeAZ2+IZkS38oOIvbQbqFJK1XGxrYL5amo/opoHMHX+RqZ/vZ2CGuxiqgFAKaXqkHB/\nHz68vQ+TLm7F2z/tZ8K78RzPzK2Rc2kAUEqpOsbLw40nRkbxnxu7sfFQGte88jO7j52u9vNoI7BS\nStVR18e2oGMTf55buovGgT7VfnwNAEopVYdFNQ/kvT/1qpFjaxWQUkq5KA0ASinlojQAKKWUi9IA\noJRSLkoDgFJKuSgNAEop5aI0ACillIvSAKCUUi6qXmUDFZFU4OAF7h4KHK/G4tQXet2uxVWvG1z3\n2u257lbGmLDSC+tVAKgKEUk4XzpUZ6fX7Vpc9brBda+9KtetVUBKKeWiNAAopZSLcqUA8JajC+Ag\net2uxVWvG1z32i/4ul2mDUAppdS5XOkJQCmlVAkaAJRSykW5RAAQkWEisktE9ojINEeXp6aIyGwR\nSRGRrSWWBYvIMhHZbXsPcmQZa4KIRIjIchHZLiLbROQ+23KnvnYR8RGRtSLyq+26n7AtbyMi8ba/\n949FxMvRZa0JIuIuIhtF5Cvbd6e/bhE5ICJbRGSTiCTYll3w37nTBwARcQdmAVcCXYCxItLFsaWq\nMXOBYaWWTQN+MMZ0AH6wfXc2BcBfjTFdgD7AFNt/Y2e/9lzgUmNMN6A7MExE+gDPAi8ZY9oDp4Db\nHFjGmnQfsKPEd1e57sHGmO4l+v5f8N+50wcAoBewxxizzxiTBywARjq4TDXCGLMKOFlq8Uhgnu3z\nPODaWi1ULTDGHDHGbLB9Po11U2iOk1+7sWTavnraXga4FFhoW+501w0gIi2Aq4F3bN8FF7juMlzw\n37krBIDmQGKJ70m2Za6isTHmiO3zUaCxIwtT00SkNRADxOMC126rBtkEpADLgL1AmjGmwLaJs/69\nzwD+BhTZvofgGtdtgO9EZL2ITLYtu+C/c50U3oUYY4yIOG2/XxHxAz4F/mKMybB+FFqc9dqNMYVA\ndxFpBCwCOjm4SDVORIYDKcaY9SIyyNHlqWX9jDHJIhIOLBORnSVXVvbv3BWeAJKBiBLfW9iWuYpj\nItIUwPae4uDy1AgR8cS6+X9ojPnMttglrh3AGJMGLAcuBhqJSPGPO2f8e+8LjBCRA1hVupcCL+P8\n140xJtn2noIV8HtRhb9zVwgA64AOth4CXsAYYLGDy1SbFgOTbJ8nAV84sCw1wlb/+y6wwxjzYolV\nTn3tIhJm++WPiDQALsdq/1gO3GDbzOmu2xjzsDGmhTGmNdb/zz8aY8bj5NctIr4i4l/8GRgKbKUK\nf+cuMRJYRK7CqjN0B2YbY6Y7uEg1QkTmA4Ow0sMeAx4HPgc+AVpipdK+yRhTuqG4XhORfsBPwBZ+\nrxN+BKsdwGmvXUSisRr93LF+zH1ijHlSRNpi/TIOBjYCE4wxuY4rac2xVQE9YIwZ7uzXbbu+Rbav\nHsBHxpjpIhLCBf6du0QAUEop9UeuUAWklFLqPDQAKKWUi9IAoJRSLkoDgFJKuSgNAEop5aI0ACil\nlIvSAKCUUi7q/wGRiqTSD9D3ZQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_plot, = plt.plot(range(len(train_epoch_losses)), train_epoch_losses, label='train loss')\n",
    "val_plot, = plt.plot(range(len(validation_epoch_losses)), validation_epoch_losses, label='validation loss')\n",
    "plt.legend(handles=[train_plot, val_plot])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hYjJ8T9yCxSM"
   },
   "outputs": [],
   "source": [
    "def inference(test_dataloader, test_dataset, criterion):\n",
    "    test_loss = 0.0\n",
    "    acc = 0.0\n",
    "\n",
    "    for x, y in tqdm.tqdm(test_dataloader):\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "          output = model(x)\n",
    "          loss = criterion(output, y)\n",
    "          test_loss += loss.item() * x.size(0)\n",
    "\n",
    "          output = output.cpu().data.numpy()\n",
    "          output_binary = np.zeros_like(output, dtype='int')\n",
    "          output_binary[output >= 0.5] = 1\n",
    "              \n",
    "          y = y.cpu().data.numpy().flatten()\n",
    "          acc += len(np.where(output_binary == y)[0])\n",
    "\n",
    "    return test_loss / len(test_dataset), acc / len(test_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "e-EprjK2F9MT",
    "outputId": "50f1954a-eea4-4e55-e0fd-e71362e48432"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 125/125 [00:00<00:00, 196.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing loss: 0.12399046846334386, testing accuracy: 0.859792006014284\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_accuracy = inference(test_dataloader, test_dataset, loss_func)\n",
    "print(f'Testing loss: {test_loss}, testing accuracy: {test_accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UkD7wYkeT4lu"
   },
   "source": [
    "We can see that our model gives a better accuracy that traditional machine learning models. But it is also crucial to notice that when I've tried to train it for longer or used higher learning rate it started to overfit. It might be the case that there is no enough training data. In this case it might be beneficial to stick to traditional machine learning methods: logistic regression or Naive Bayes. They give similar results for this data. They are fast, easy to train and interprete. What also might be the reason is that GloVe embeddings are not the best for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IeyDnZIwUqmE"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "nlp-torch.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}