02_intro_neural_networks/01_introduction_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Networks \n",
    "Author: Marieme Ngom, adapting materials from Bethany Lusch, Asad Khan, Prasanna Balaprakash, Taylor Childers, Corey Adams, Kyle Felker, and Tanwi Mallick.\n",
    "\n",
    "This tutorial will serve as a gentle introduction to neural networks and deep learning through a hands-on classification problem using the MNIST dataset. \n",
    "\n",
    "In particular, we will introduce neural networks and how to train and improve their learning capabilities.  We will use the PyTorch Python library.\n",
    "\n",
    "The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) contains thousands of examples of handwritten numbers, with each digit labeled 0-9.\n",
    "<img src=\"images/mnist_task.png\"  align=\"left\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import torch\n",
    "import torchvision\n",
    "from torch import nn\n",
    "\n",
    "import numpy \n",
    "import matplotlib.pyplot as plt\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST dataset\n",
    "\n",
    "We will now download the dataset that contains handwritten digits. MNIST is a popular dataset, so we can download it via the PyTorch library. Note:\n",
    "- x is for the inputs (images of handwritten digits) and y is for the labels or outputs (digits 0-9)\n",
    "- We are given \"training\" and \"test\" datasets. Training datasets are used to fit the model. Test datasets are saved until the end, when we are satisfied with our model, to estimate how well our model generalizes to new data.\n",
    "\n",
    "Note that downloading it the first time might take some time.\n",
    "The data is split as follows:\n",
    "- 60,000 training examples, 10,000 test examples\n",
    "- inputs: 1 x 28 x 28 pixels\n",
    "- outputs (labels): one integer per example "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data = torchvision.datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=torchvision.transforms.ToTensor()\n",
    ")\n",
    "\n",
    "test_data = torchvision.datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=torchvision.transforms.ToTensor()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = int(0.8 * len(training_data))  # 80% for training\n",
    "val_size = len(training_data) - train_size  # Remaining 20% for validation\n",
    "training_data, validation_data = torch.utils.data.random_split(training_data, [train_size, val_size], generator=torch.Generator().manual_seed(55))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNIST data loaded: train: 48000  examples, validation:  12000 examples, test: 10000 examples\n",
      "Input shape torch.Size([1, 28, 28])\n"
     ]
    }
   ],
   "source": [
    "print('MNIST data loaded: train:',len(training_data),' examples, validation: ', len(validation_data), 'examples, test:',len(test_data), 'examples')\n",
    "print('Input shape', training_data[0][0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look. Here are the first 10 training digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAbGUlEQVR4nO2deZQU1fXHPw8BZQICsrigwC8uMwE0OCgiByXighs6BIgRFQy4oWQi4A4qO4omkZMIghkFATcEEQFxZYtxQVQ0oqISYXAXBERERev3R/V9VTPT09N7Vxf3c86cGbq7qt6luqru+767GMdxUBRFURRFCTO1cj0ARVEURVGUTKMOj6IoiqIooUcdHkVRFEVRQo86PIqiKIqihB51eBRFURRFCT3q8CiKoiiKEnpSdniMMSONMbPSMZigojbmP2G3D9TGsBB2G8NuH6iNQSUuh8cY09cY85oxZocx5jNjzFPGmC6ZHlw8GGPGGGPeNsbsNsaMTGE/QbaxtTFmqTFmpzHmPWPMKUnuJ5A2GmOaG2MeMsZ8aozZZox50RhzXBL7CaR9EP5zCGCM6WyMedUY860x5q1kxxVUG40xLSNj8v84xphhSewr1DYG1T7QZ0aC+wmyje2NMSsjz4xNxpiba9qmRofHGDMUuAsYD+wPtAQmA+emOuA08SFwHbAo2R3kgY0PAW8ATYDhwGPGmGaJ7CDgNtYHVgEdgP2AGcAiY0z9eHcQcPsg5OfQGLMf8CRwB9AImAg8aYxpnOB+Amuj4zgbHcepLz/AkcAvwNxE9hN2G4NsXwR9ZsRBHtj4ILAC95nRFbjSGHNOzC0cx6n2B2gI7AD6xPjMSGCW799zgM+BbZHBtPW9dyawFvgW+AS4JvJ6U2AhsBXYAqwEasUaW5RxzAJGJrJNPtgIHAH8ADTwvbYSuCIsNlYznu1AhzDYtyecQ+Bs4J1Kr60DBobFxihjuRVYmuA2obYxn+xDnxl5a2Nk251Am0rHvzHWNjUpPMcD+wCP1/A5P08BhwPNgdeB2b73yoDLHcdpALQDXoi8PgzYBDTD9SRvAhwAY8xkY8zkBI6fKEG3sS2w3nGcb32vrYm8Hi9Bt7ECxpj2QF3cmVg8BN2+PeUcmij/bpfAePPBRiKfM0A/XDUyEcJuY97YlwJBt3FPud/cBfQzxtQxxhRGxvxcrAHWrsGAJsDXjuPsruFzFsdx7pO/I+uj3xhjGjqOsw34CWhjjFnjOM43wDeRj/4EHAi0chznQ1wvT/Z3ZbzHTpKg21gf12P2sw1oEe94Cb6NFmPMvsBMYFTkWPEQdPv2hHP4EnCQMeZ84DGgL3AoUBDveAm+jX664N6gH4t3rBHCbmM+2ZcsQbdxT7jfgKsMPQBcA+wFjHYcZ1WsDWpSeDYDTY0xNTlGABhj9jLG3GaM+cgYsx34OPJW08jvXrjS1gZjzHJjzPGR1+/Anc0/Y4xZb4y5IZ7jpYmg27gD2LfSa/viSoPxEnQb5bj1cONAXnYcZ0ICmwbdvtCfQ8dxNuOu7Q8FvgBOx51tbYpn+wiBtrES/YG5juPsSHC7sNuYT/YlS9BtDP39xrgxg0uA0bhK1CFAd2NMbCcpjnW874De8azjARcB7wL/hytnN8KVpw6rtE0dYAhQHmV/7YAvgZPjWcfzbZfKemxgbcRdj91FxfXYFSS+HhtYGyOf3xt4GlcGTXQtPtD27SnnsNK2tYGNQPew2QjUw50xd0vi/yXUNuaLfZHt9JmRvzYeA3xT6bWrgYWxtoup8DiuFHULcLcxpsQYU2Dc9bIzjDETo2zSADdYajOulD1e3jDG1DXGXBCRuH7CDUr9JfLe2caYw4wxBvci+1neq4nIePbBVatqG2P2McbsFc+2+WCj4zjrgDeBWyO29QSOIoGsiaDbaIypgyubfw/0dxwnrnOfL/btCecwsu3RkTHtC9yJe1N7Okw2RuiJK8kvTWCbPcLGfLBPnxn5byNuQoQxbtp8LWPMAcB5wFs1GRaPt3cB8Bqux/c5bjpf5yheXn3gCVzpbANuwJsDHIYbhLoE9yLajpuG3CWy3RBcCew7XAn8Zt+x7wHuiTG26ZFj+H8uTsJrD7KNrYFluA7B+8ApidoXZBtxUwod3Kj7Hb6fE8Jg355wDiPvP4R709oGPAI0D5uNkc88DYxJxrY9xcYg24c+M8JiY7fIvrZFxnYvUBDLHhPZUFEURVEUJbRoLy1FURRFUUKPOjyKoiiKooQedXgURVEURQk96vAoiqIoihJ61OFRFEVRFCX0xKyiaIzJ6xQux3Eq9/apgtoYfNTG8NsHamM+oDaG3z4Ir42q8CiKoiiKEnrU4VEURVEUJfSow6MoiqIoSuhRh0eJi4EDBzJw4ECWLVvGsmXLGDRokC3XvW7dOtatW8fAgQNzPUxFURRFiYo6PIqiKIqihJ6YvbTCGqntJ1M2dunSBYD+/fsDMGDAAPtev379AJg9e3bKx8m0jVdffTUAt99+OwB77eU1FXYb3CKN3Pj555959913AbjqqqsAePHFF5M9tCXTNk6YMAGAF154AYBnn30WgLfffpt27doB8MorrwCwdu1a5s51mw4vWrQo2UNWQTND1MZ8IGg2jhkzBoCbbroJgIMOOgiAL774Iul9Bula7NChQ9TXv/76azZs2JDUPoN2DjOBZmkpiqIoirLHEhiF55BDDgHg448/BuCDDz4AoFOnTmzdujWpfWbLk61d2y1ndPDBBwNQVFTEvffeC3gzjl9++cV+ftu2bQBccMEFADz99NNJHzvTNo4fPx6A6667Ltp+ZQxV3vvkk08AOPfccwF48803kx1CRm3s0KED//nPfwD44YcfADj//PMB6N27t/3c73//ewAaNGjAd999B3gKz4gRIwD48MMPkxkCEKxZZSbYk2eVftTG9NGyZUurIB944IGAd7/98ssvk95vrq/FoqIiAGbOnElxcbGMSY4NwFdffcUtt9wCwLx58wBX9YmHIJ3DTFGdjTELD2aTs846C/BO7FdffQV4D6EgI46OOGk10bBhQwAuu+wyIDWHJwjI+Lt3725fa9GiBeAt6aXi8GSSNm3aUKdOHQD7u2XLlgD86U9/sp+TZa+//OUvDBo0CIA//OEPAHTs2BGAk046KWmZOUgcdthhAFxzzTWA5/g9//zz9nzu2rUrN4NTlAiXXHKJdXTCgDg6a9euBdxnoTg47733HgCvv/46ACUlJUydOhWA0047Dag4QcslEvYg989TTz0V8O6XlTn99NMBWLp0KQA//vhjlX1JSEXjxo1ZvHgxgA0tSARd0lIURVEUJfQEZklrwYIFAJx55pmAt5Qisl0yZFq6+93vfgd4AXMnnXQSAFu3bmXNmjWAt+wxceLEavdz5JFHWg8+UTJtY6NGjQBvqbF+/fr2vVmzZgHw97//HXDPoSg7wu7duwEYMmQIU6ZMSWoMmbRx165dbNmyBcCm1T/33HMA/PTTT9GOY2cdf/vb3wC4+OKLAdiyZQunnHIKkPjyVq5l9L333htwr7+77roL8JaZ/feIX//61wAJK1nZktEvv/xyAHseunXrRuPGjWMdE8Ber/369eOtt95K6tjZXio44IADADjhhBPs+GW2f8QRR9CpUycAjj322Arb7dixwyZVyHbxEqTlkBkzZtiwACFfl7SaNWvGq6++CngKs+M4VlmW3zt37gRcNeidd96xnwPo06cPjz/+eI3HyvQ5vP766wFvzHEcC4Ann3wSgHPOOQdw1Zwbb7wR8NRmwCbGHH/88QBs3769yj41aFlRFEVRlD2WQMTwTJkyxcbwrF69GoCxY8fmckhx8fzzzwMVA5IBSktLeeihhwBv9h+LBQsW0KNHDwDef//99A4yRSRg/I477gA8NWvvvfemvLwc8GaJ5513HgsXLgQ8ZUgCugcPHsyMGTMAb5aSS+rVqwdArVq17Iz+qaeeqnE7x3GsalVaWgrARx99BLiq5IUXXgjAyJEj0z3kjCAqjgQ+FhcXW3VKgrfl/6qsrMxelxdddFG2hxoXEicgauvu3butPTITlBgJgKOPPhqAo446CoATTzwxaYUnW0g5ixNPPBHwVI3qqHx/KigosEpdogqPkhkuvPBCq+yI4vH444/bc135nvnee+9ZhU9iWubNm2fjYbIdFyr3+eLiYq644oqonykvL6egoACAb775BoBDDz3U2nvyyScDbmwWwPDhw2nVqlWV/cgqQt26dRMfZ8JbZICzzjqLWrVcsUmWE/yBS0FCvmSPPvqoHfOOHTsAr77OK6+8wooVKwBXboaqNx0/hx9+uM0CilcGzDbjxo0DvKWtFi1aVFmme/nll7nzzjuBqg5rYWEhvXr1Atzsg1wjY6ldu7Z1spNl0qRJgCuxDhs2DPDq9sTjROWKAw88kL/+9a8ANhtk6tSpDB06FPACkyWYErzAyKA6POJwi8PTu3dvu1wejUceeQSAtm3bAokv1WUTeZj16dMH8AI6x44dy/3331/tdv/4xz8AL1xgx44dcSdYBBGxu169evZhKcG8kkGZL/Ts2ROAG264wS5NSTjHhAkTYk4OJbFnyJAhACxfvtxeq9l2eJo1awa4z4DKLFmyBIC+ffva56eEcPTs2dOGOjRv3hyAadOmxTzW5MmTgfiz0vzokpaiKIqiKKEnpwqPBNU1btzYeqvivQWV++67D3Bn86LaSBDVE088AbizysoBVe+++64NMhMlyI8sFX366acAdvknaNRUHVpslJnWr371K/vepZdeCngev5zzXNC0aVP7d7oUp5tuuoljjjkG8GTZICo8kn4/ceJEq3SNHj0agFGjRlX5vFTNNsakVGsoG0jdJ+Hzzz+P+rk2bdoAcPbZZwPe9/aZZ57J4OhSQ5ZSZXYsS+pTpkyJaud5550HeMqOcN999/Hf//43k0PNKL/5zW8AtzaWqCLLli0D8kfhkaVv/zL/zTffDCSu8ovS8f7771vFKwjINSWB5Vu3bq1SU2/16tVxnTNZAhs6dChz5sxJekyq8CiKoiiKEnpyqvBIqlm9evXs2t+mTZtyOaQaKSwstH9LMT2ZSQlSJAk8NeeJJ57g1ltvrXa/++yzD0DUIK18QlILpfqwpKwDdO7cGfDS+VPx1FNFAuQgdnxVIqxfv55HH30UgGuvvRZwlQR/kGwQkIKXffv2tbFG0ZQdKagp8TqO49hzl+/IrFMCsiXOJ8iFTiW+UQKsa0LUq8oMHz48bWPKBf5SJRK7Iwpl0JEYG0m39sftJBu/KYqf4ziUlJQA2NISuURUGfntp3Xr1oCrqEoAfeUSOdu3b+fhhx8GvDg0UY2SRRUeRVEURVFCT04UHukA619bltS6oCJdw/2F9aTIl6xBiprRokUL63VLXE+8SKpzWVmZ7UeVj6xfv77a9yReKZcKj2TnSDmEdLFx40bAyyQZNmyYLWiYa2QmJZl0a9eu5corr6z28zLjbNCgAeBm6AWhpEAsli9fDngp2127drUF3fxUVqqkxEJYGDdunC0rIAqm9MPL97YgEnfmOA6bN28G4Ntvv83lkOKiVatW9vspWU2S/ZqO7FxjjI0nDSqi7Egcp7Sx8bNq1SrAVb0SfX7WRE4cHqnEKNVdly9fXmHpI2g0bdrUyvryICsrK7MXm5zEwYMH222kbk2iSJ8tqWuQ70QLogtSYF26EUdKlrSkzk0uketMrjGpXzFixIioPc6kP5GkPwulpaV8//33mRxqysj/vzzc/cHpwhlnnEH79u0BbwkrVup6PiFO7SWXXGKvM2mOG+R7bDxIQoAfeTgGGVnGmjt3Lk2aNAHS6+jI/gsLC21/rSAg50tKIJSVldm6dEcccUSVz0sQvji00Soop4ouaSmKoiiKEnqyLiM0adKE4447DvCClMaMGZPtYSTEnDlz7IxQ0h+jVZOUQoTLli3jgQceqPBe69atq+0L1rVrVxvsuv/++6dr2AkjakS8gdOSChsrVTlar7ZY/duyhfTJchzH9m6RpZ5UkAC96tKhc4HMKqWat3QljiYX16lTxxaUFGVIZHKR44OMBIjLOY3Wo65Tp042WPmFF14AvHIQ+YqodlK0rmnTpla9yndlB9zgcgmsl/vsBx98YMuEBBEJ3ZBwjYKCgqRTz6MhJT+ka/jOnTuzXnAwFpKIc+ihhwLed7MyUnRXxp7JZXNVeBRFURRFCT1ZV3gGDBhgA38lBT3ovWscx7GBf//617+qvC+zeUkFlDL1fq6//voq6c+33XYb4Ka3/+9//wO8YLZsIQGpAwcOtHFKv/3tb+PaVnpI+YNCn332WSD4gZES6zFz5sy403zzle7duwNevIp0efcjas60adPo27cv4M20/C0J8gX5HvqReB5/GYkgzYhTQYKw/XFX0jYkng7aQad37972eyz30dGjR9tWN0GjqKjIXmeisE6aNIl77703bceQtkWy/5UrVybVbiEdfPHFF4C7OiCrH3KdSVxZdUhn+2wkRGTd4ZFeGuBVVZbg33xAMm6kOSh4D3fpo+Sv9ipLYZLRBV4/qsceewxwL+aOHTtmbtAxWLlyJQDt2rVLeFuJsBfJEryGk/mEvyZPGJGHoDjVixYtsu/JEqosY0kFWP9rQX2oJIrU3vEHTAatRlIyNGnSpEqz2s2bNzN//vzcDCgDSCAreH0WP/vss1wNp0aGDx9uHRJpzCvPh3Qhletlkjx//vycOTzihJaXl9s6T7JcLPWjqkOejRKAnsk+mrqkpSiKoihK6Mm6wjNkyBBeeuklgCrdtvMBCbju1auXDRarjD8YVBSeli1b2pmyKEAyu4wmv2caWb7yV46uzKpVq6zcGI3qKrnmE7t377ZLHdLbLVrH33gRtevII49MeV/pQrpsSwCvSOC9e/e25RP8Pc9kiTZWB+58REo+ALz44otAbq69dDN48GB7XxImTZrE6tWrczSi9CHXpF+FlcQASWMOEqJulJSUWGUnWu/EVJk5c6atqixJIHK8oCCdFPxJKtF6LEoyz9FHHw14CQeZQBUeRVEURVFCT9YUHlED/AHA+Yik2pWWllar8FSHdOUWZUc6ajdp0sT+n4g6lK24pmgp4gMGDADcapixFJ4zzjijynY9e/as9vPi3Qehz4tw++232x5njRo1Snl/EuNVv359IBip3P/+978B6NKlC1CxS72sl/uLQUphUKkaHRZOPfVU+/fs2bOBzMYLZJqDDjoI8K5X8Iq8hUWdkwKSUkYA4J577snVcGpE4sQKCgp44403gPQE40pxQSlYWFJSwoMPPgh4an1QkASIgoKCCq8/88wzVgGT7+wVV1xh7z3RCkumG1V4FEVRFEUJPSZWEThjTMoV4qSYnWSGtG3b1valkg6omcJxnBp7GMRj45gxY2z/Jz9Ssn3SpEmAl3XlR2bVAwYMYMOGDYDnActMulatWraMtr+7ejykaqO0FvBnaU2ZMgVw+9NI2m48SkXDhg1jKlN//OMfgej/T7FI13mMxsEHH2w7LpeVlQHeOnyiSuQxxxxji4xJMcaTTz45rnYMNdmYyrUocUXSyV7O0dy5c21Wx4wZMwD3O92tWzfAK86YDjJ5DmtC4tRWrFgBuHFb8n2P1sk5WbJto3TcHjt2rFURxC6516SbbNlYp04dwFUFwOuNBl57n0yRyrUo3bwLCwvtuYhW/DIeioqKrN1yrkU1mTdvHoMGDUpqv5k+hxIzWLk/ZnFxsX3eSGbZxx9/bG2SrDtRLlOhOhsztqQlgxZHp02bNvbf0WrZBJlx48bZk3LVVVcB7kXXuXNnwAtMlkaM1157bZU+W61bt6Zr166AG8AMFR+oEuCV7mZpNSGpgH6Hx38h/fnPfwaiP/yk4qnYUVOPrMsuuwzwnCf/0kqu2LRpk3VcxQGVejPjx4+Pqyq0LF8tXrzYBgRLJdUg9J6SeklyDfqpXKl2+vTpaXV0goBUcpWb7IQJE9Lq6OQKf+NFWZrLlKOTbeReKandjuMEvsE0YCcQRUVFNpVelqFqQpwbWRYrKSmx31m5V0qNmyDXVvIvsVZHtKXKbDTw1SUtRVEURVFCT8YUHlEMKlcdHjVqVCBmvYmwa9cuWzRKAjkvuugim0YngcwyG/GnIldWQaIxatQoK91mG38xLJEipVs2eB643xMXRNGJtzdWcXExgK20HQSFB7xiiVIuQXq7tW/f3nYflh5q4FW1PfbYYwEYOnQo4AafSyXVfOhf1L59e9u9WHq5TZ8+PXcDyhCiwAqxer/lA1Is0h+sGuRA3kRo3LgxED3oeuzYsdkeTsLI9V9YWMgNN9wAYNPHY1VZPuGEE2z4g6wArF271t5bpEBsvicSlJaWVvgN8PPPPwPxK2GpoAqPoiiKoiihJ2MKj6TnyuxfvNt8L4YlAcqzZs2ys2LpPeUvbBYNKTwo6+wSJOtvU5FtJFj60ksvtTPhykXMwEsZ3LJlCwAdO3asEEgoSJDz+vXrgYrxQP379we8QOmgIIqjpC1LjFmPHj3sOvwnn3xiPy8KlSDvjRkzxvYvCjISEDps2DB7fUqKq8y2wkLPnj1tqQQpiyCz5XxF7jv+4N0FCxbkajhpRRI6/CozwMKFC21yQZCR+19xcbFt0yLK9tSpU6vEO8r1V15ebhVvuf9kQ/HIBNXFcj788MM2plXuQQB33303kJ3vcEaytPbff3/efvttAPbbbz/AC96VhqHZIFsZBdLUTqRmcfb8lJWVsWbNGgD7Ox3kMvslW+TKxosvvtjetCRz6ccff7Q1XF577TXAdX7BzWpLlkxmaVVGvq+LFy+2QfIS2JspcnUOr7vuOtukV+pgieOdbrJlowR3SmLIkiVLbPX23bt3p7r7mGTaRum1WPk5cfXVV/PPf/4z2d0mRLquRVmakkrusdi4cWPW+mBl+hyOGDECcJu7xkOPHj2Aij3+UqU6G3VJS1EURVGU0JORJa26detaZUcCQIPc2TZVRMYUHnjggRyNREkn06dPD1UQb/PmzYGKAa7Tpk3L1XCyQocOHezf0i0+X5H+Q7VrV7xtT5gwIePKTraQPm6VbcxHJMA43wONE0XumbLS0apVq2o/269fv7QqOzWhCo+iKIqiKKEnI250eXl5KDx0RQkTdevWBbwidffffz9LlizJ5ZAyhhQK9aekz58/P1fDSQunnXYa4Cl1khggvfkUJQhI/JUUIIzW1V46Ckg8ZLZQhUdRFEVRlNCjMoyi7CHIzEt6S4UZSY01xrB06VIgvdmRQWDy5MmAVypCUYKEXHdSfDcIZLx5aC7RlG0XtTH4ZDMtPRfoOXRRG4OPXovhtTE4rpeiKIqiKEqGiKnwKIqiKIqihAFVeBRFURRFCT3q8CiKoiiKEnrU4VEURVEUJfSow6MoiqIoSuhRh0dRFEVRlNCjDo+iKIqiKKHn/wE1l8TlN5HFAwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(numpy.reshape(training_data[i][0], (28, 28)), cmap=\"gray\")\n",
    "    plt.title('Class: '+str(training_data[i][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generalities:\n",
    "To train our classifier, we need (besides the data):\n",
    "- A model that depend on parameters $\\mathbf{\\theta}$. Here we are going to use neural networks.\n",
    "- A loss function $J(\\mathbf{\\theta})$ to measure the capabilities of the model.\n",
    "- An optimization method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Model\n",
    "Let's begin with a simple linear model: linear regression, like last week. \n",
    "We add one complication: each example is a vector (flattened image), so the \"slope\" multiplication becomes a dot product. If the target output is a vector as well, then the multiplication becomes matrix multiplication. \n",
    "\n",
    "Note, like before, we consider multiple examples at once, adding another dimension to the input. \n",
    "\n",
    "\n",
    " <img src=\"images/LinearModel_1.png\"  align=\"center\"/>\n",
    " \n",
    "\n",
    "\n",
    "The linear layers in PyTorch perform a basic $xW + b$. These \"fully connected\" layers connect each input to each output with some weight parameter. We wouldn't expect a simple linear model $f(x) = xW+b$ directly outputting the class label and minimizing mean squared error to work well - the model would output labels like 3.55 and 2.11 instead of skipping to integers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need:\n",
    "- A loss function $J(\\theta)$ where $\\theta$ is the list of parameters (here W and b). Last week, we used mean squared error (MSE), but this week let's make two changes that make more sense for classification:\n",
    "    - Change the output to be a length-10 vector of class probabilities (0 to 1, adding to 1).\n",
    "    - Cross entropy as the loss function, which is typical for classification. You can read more [here](https://gombru.github.io/2018/05/23/cross_entropy_loss/). \n",
    "\n",
    "- An optimization method or optimizer such as the stochastic gradient descent (sgd) method, the Adam optimizer, RMSprop, Adagrad etc. Let's start with stochastic gradient descent (sgd), like last week. For far more information about more advanced optimizers than basic SGD, with some cool animations, see https://ruder.io/optimizing-gradient-descent/ or https://distill.pub/2017/momentum/.\n",
    "\n",
    "- A learning rate. As we learned last week, the learning rate controls how far we move during each step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearClassifier(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        # First, we need to convert the input image to a vector by using \n",
    "        # nn.Flatten(). For MNIST, it means the second dimension 28*28 becomes 784.\n",
    "        self.flatten = nn.Flatten()\n",
    "        \n",
    "        # Here, we add a fully connected (\"dense\") layer that has 28 x 28 = 784 input nodes \n",
    "        #(one for each pixel in the input image) and 10 output nodes (for probabilities of each class).\n",
    "        self.layer_1 = nn.Linear(28*28, 10)\n",
    "        \n",
    "    def forward(self, x):\n",
    "\n",
    "        x = self.flatten(x)\n",
    "        x = self.layer_1(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LinearClassifier(\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      "  (layer_1): Linear(in_features=784, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "linear_model = LinearClassifier()\n",
    "print(linear_model)\n",
    "\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(linear_model.parameters(), lr=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning\n",
    "Now we are ready to train our first model. A training step is comprised of:\n",
    "- A forward pass: the input is passed through the network\n",
    "- Backpropagation: A backward pass to compute the gradient $\\frac{\\partial J}{\\partial \\mathbf{W}}$ of the loss function with respect to the parameters of the network.\n",
    "- Weight updates $\\mathbf{W} = \\mathbf{W} - \\alpha \\frac{\\partial J}{\\partial \\mathbf{W}} $ where $\\alpha$ is the learning rate.\n",
    "\n",
    "How many steps do we take?\n",
    "- The batch size corresponds to the number of training examples in one pass (forward + backward). A smaller batch size allows the model to learn from individual examples but takes longer to train. A larger batch size requires fewer steps but may result in the model not capturing the nuances in the data. The higher the batch size, the more memory you will require.  \n",
    "- An epoch means one pass through the whole training data (looping over the batches). Using few epochs can lead to underfitting and using too many can lead to overfitting.\n",
    "- The choice of batch size and learning rate are important for performance, generalization and accuracy in deep learning.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "\n",
    "# The dataloader makes our dataset iterable \n",
    "train_dataloader = torch.utils.data.DataLoader(training_data, batch_size=batch_size)\n",
    "val_dataloader = torch.utils.data.DataLoader(validation_data, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_one_epoch(dataloader, model, loss_fn, optimizer):\n",
    "    model.train()\n",
    "    for batch, (X, y) in enumerate(dataloader):\n",
    "        # forward pass\n",
    "        pred = model(X)\n",
    "        loss = loss_fn(pred, y)\n",
    "        \n",
    "        # backward pass calculates gradients\n",
    "        loss.backward()\n",
    "        \n",
    "        # take one step with these gradients\n",
    "        optimizer.step()\n",
    "        \n",
    "        # resets the gradients \n",
    "        optimizer.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(dataloader, model, loss_fn):\n",
    "    # Set the model to evaluation mode - some NN pieces behave differently during training\n",
    "    # Unnecessary in this situation but added for best practices\n",
    "    model.eval()\n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    loss, correct = 0, 0\n",
    "\n",
    "    # We can save computation and memory by not calculating gradients here - we aren't optimizing \n",
    "    with torch.no_grad():\n",
    "        # loop over all of the batches\n",
    "        for X, y in dataloader:\n",
    "            pred = model(X)\n",
    "            loss += loss_fn(pred, y).item()\n",
    "            # how many are correct in this batch? Tracking for accuracy \n",
    "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    loss /= num_batches\n",
    "    correct /= size\n",
    "    \n",
    "    accuracy = 100*correct\n",
    "    return accuracy, loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: training loss: 0.5016719663143158, accuracy: 87.59791666666666\n",
      "Epoch 0: val. loss: 0.4940067529678345, val. accuracy: 87.55833333333334\n",
      "Epoch 1: training loss: 0.42167428183555605, accuracy: 89.01875\n",
      "Epoch 1: val. loss: 0.4124584182145748, val. accuracy: 88.75833333333333\n",
      "Epoch 2: training loss: 0.3877562354803085, accuracy: 89.64791666666667\n",
      "Epoch 2: val. loss: 0.3779958865426956, val. accuracy: 89.46666666666667\n",
      "Epoch 3: training loss: 0.36779386941591896, accuracy: 90.10416666666666\n",
      "Epoch 3: val. loss: 0.3578740727711231, val. accuracy: 89.93333333333334\n",
      "Epoch 4: training loss: 0.354208150545756, accuracy: 90.39166666666667\n",
      "Epoch 4: val. loss: 0.3443111534765426, val. accuracy: 90.29166666666667\n",
      "CPU times: user 34.5 s, sys: 199 ms, total: 34.7 s\n",
      "Wall time: 34.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "epochs = 5\n",
    "train_acc_all = []\n",
    "val_acc_all = []\n",
    "for j in range(epochs):\n",
    "    train_one_epoch(train_dataloader, linear_model, loss_fn, optimizer)\n",
    "    \n",
    "    # checking on the training loss and accuracy once per epoch\n",
    "    acc, loss = evaluate(train_dataloader, linear_model, loss_fn)\n",
    "    train_acc_all.append(acc)\n",
    "    print(f\"Epoch {j}: training loss: {loss}, accuracy: {acc}\")\n",
    "    \n",
    "    # checking on the validation loss and accuracy once per epoch\n",
    "    val_acc, val_loss = evaluate(val_dataloader, linear_model, loss_fn)\n",
    "    val_acc_all.append(val_acc)\n",
    "    print(f\"Epoch {j}: val. loss: {val_loss}, val. accuracy: {val_acc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9e487651d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJNCAYAAAB0hdJBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVhV1eLG8e9mVpxHRMR5AmcQ01Iz0yxN06w0K4fS6vZr7t5snsvKyuY5U29BmmNamZlXzUrAAVGcFRlFBBGZh7N+fxzz3kpLTdznwPt5Hh88cPbmBfTwPmuvvZZljEFEREREXIOH3QFERERE5L9UzkRERERciMqZiIiIiAtRORMRERFxISpnIiIiIi5E5UxERETEhXjZHeBcadCggWnRooXdMURERET+0oYNGw4bYxqe7GOVppy1aNGC2NhYu2OIiIiI/CXLsg6c6mMVelnTsqy7LcvaalnWNsuy7jn+vnqWZa2wLGv38bd1T3FsuWVZm4//WVKROUVERERcRYWVM8uyOgGTgQigKzDMsqw2wFRgpTGmLbDy+OOTKTTGdDv+Z3hF5RQRERFxJRU5ctYRWG+MKTDGlAGrgVHACGDW8efMAq6qwAwiIiIibqUi55xtBZ6zLKs+UAhcAcQCjY0x6cefcxBofIrj/SzLigXKgGnGmEVnGqC0tJSUlBSKiorOPL3Yxs/Pj6CgILy9ve2OIiIict5VWDkzxmy3LOtF4DsgH9gMlP/uOcayrFPtvN7cGJNqWVYr4AfLsuKNMXv/9wmWZU0BpgAEBwf/4QQpKSnUrFmTFi1aYFnW3/+ipMIZY8jKyiIlJYWWLVvaHUdEROS8q9AbAowxHxtjwowx/YAjwC4gw7KsJgDH3x46xbGpx9/uA/4DdD/Jcz4wxoQbY8IbNvzj3ahFRUXUr19fxcyNWJZF/fr1NdopIiJVVkXfrdno+NtgnPPNPgeWAOOPP2U8sPgkx9W1LMv3+N8bABcCCWeZ4WwOExvpZyYiIlVZRe8QMN+yrATgK+AOY0wOMA0YZFnWbuDS44+xLCvcsqyPjh/XEYi1LCsOWIVzztlZlTM7ZWVl0a1bN7p160ZAQABNmzY98bikpOS0zjFx4kR27tz5p895++23+eyzz85FZAAyMjLw8vLio48++usni4iIyDllGXOqKV/uJTw83Px+Edrt27fTsWNHmxL91pNPPkmNGjV44IEHfvN+YwzGGDw8XGcnrTfffJO5c+fi4+PDypUrbcngSj87ERGRc82yrA3GmPCTfcx1GkEVsmfPHkJCQhg3bhyhoaGkp6czZcoUwsPDCQ0N5emnnz7x3IsuuojNmzdTVlZGnTp1mDp1Kl27dqV3794cOuScrvfoo48yY8aME8+fOnUqERERtG/fnp9++gmA/Px8rr76akJCQhg9ejTh4eFs3rz5pPkiIyOZMWMG+/btIz09/cT7ly1bRo8ePejatSuDBw8G4NixY4wfP54uXbrQpUsXFi0645tqRURE5H+onNlkx44d3HvvvSQkJNC0aVOmTZtGbGwscXFxrFixgoSEP17FPXr0KP379ycuLo7evXvzySefnPTcxhiio6N5+eWXTxS9N998k4CAABISEnjsscfYtGnTSY9NTEwkOzubsLAwrrnmGubOnQvAwYMHuf3221m4cCFxcXFERUUBzhHBhg0bsmXLFuLi4ujfv/+5+PaIiIhUWZVmb82/8tRX20hIyz2n5wwJrMUTV4ae1bGtW7cmPPy/o5mRkZF8/PHHlJWVkZaWRkJCAiEhIb85plq1alx++eUAhIWFsXbt2pOee9SoUSeek5iYCMCPP/7Igw8+CEDXrl0JDT157qioKK677joAxowZwz/+8Q/uvvtufv75ZwYMGEDz5s0BqFevHgDff//9idEyy7KoW/eku3GJiIjIaaoy5czV+Pv7n/j77t27ef3114mOjqZOnTrccMMNJ11KwsfH58TfPT09KSsrO+m5fX19//I5pxIZGcnhw4eZNcu5iUNaWhr79u07o3OIiIjI2asy5exsR7jOh9zcXGrWrEmtWrVIT09n+fLlDBky5Jx+jgsvvJC5c+fSt29f4uPjT3rZNCEhgbKyMlJTU0+875FHHiEqKoqbb76Zu+++mwMHDtC8eXOys7OpV68egwYN4u2332b69OkYY8jJydHomYiIyN+gOWcuoEePHoSEhNChQwduuukmLrzwwnP+Oe68805SU1MJCQnhqaeeIiQkhNq1a//mOZGRkYwcOfI377v66quJjIykcePGvPvuu4wYMYKuXbsybtw4AJ544gkyMjLo1KkT3bp1O3GpdeLEiae84UBEREROTUtpVBFlZWWUlZXh5+fH7t27GTx4MLt378bLyzUHT/WzExGRyuzPltJwzd/Mcs7l5eUxcOBAysrKMMbw/vvvu2wxExERqcr027mKqFOnDhs2bLA7hoiIiPwFzTkTERERcSEqZyIiIiIuROVMRERExIWonImIiIj8D4fD3pUsVM4q0IABA1i+fPlv3jdjxgxuv/32Pz2uRo0agHN1/tGjR5/0ORdffDG/Xzrk92bMmEFBQcGJx1dccQU5OTmnE/20dOvWjTFjxpyz84mIiNjF4TCs2ZXJPz7bwM2zYmzNonJWgcaOHXtig/BfRUVFMXbs2NM6PjAwkC+//PKsP//vy9nXX39NnTp1zvp8/2v79u2Ul5ezdu1a8vPzz8k5RUREzreM3CLe+mE3/V5exU2fRPPz3izaNKph6+iZylkFGj16NMuWLaOkpASAxMRE0tLS6Nu374l1x3r06EHnzp1ZvHjxH45PTEykU6dOABQWFjJmzBg6duzIyJEjKSwsPPG822+/nfDwcEJDQ3niiScAeOONN0hLS2PAgAEMGDAAgBYtWnD48GEAXn31VTp16kSnTp2YMWPGic/XsWNHJk+eTGhoKIMHD/7N5/lfkZGR3HjjjQwePPg32ffs2cOll15K165d6dGjB3v37gXgxRdfpHPnznTt2pWpU6f+re+riIjI31HuMKzcnsEts2LpM+0Hpn+3i+B61XljbHd+eXggjwwNwcPDsi+gMaZS/AkLCzO/l5CQ8If3nW9Dhw41ixYtMsYY88ILL5j777/fGGNMaWmpOXr0qDHGmMzMTNO6dWvjcDiMMcb4+/sbY4zZv3+/CQ0NNcYY88orr5iJEycaY4yJi4sznp6eJiYmxhhjTFZWljHGmLKyMtO/f38TFxdnjDGmefPmJjMz80SWXx/HxsaaTp06mby8PHPs2DETEhJiNm7caPbv3288PT3Npk2bjDHGXHPNNWbOnDkn/bratWtnDhw4YJYvX26GDRt24v0RERFmwYIFxhhjCgsLTX5+vvn6669N7969TX5+/m/y/hlX+NmJiEjlkpydb175bqe54PnvTfMHl5qwZ1aYad9sN/sz8857FiDWnKLTVJ1FaL+ZCgfjz+05AzrD5dP+9Cm/XtocMWIEUVFRfPzxx4CzFD/88MOsWbMGDw8PUlNTycjIICAg4KTnWbNmDXfddRcAXbp0oUuXLic+NnfuXD744APKyspIT08nISHhNx//vR9//JGRI0fi7+8PwKhRo1i7di3Dhw+nZcuWdOvWDYCwsDASExP/cHxsbCwNGjQgODiYpk2bMmnSJLKzs/H29iY1NfXE/px+fn4AfP/990ycOJHq1asDUK9evT/9nomIiJwrpeUOVm7PIDI6mTW7MwHo17YhT1wZwsCOjfH2dL2LiFWnnNlkxIgR3HvvvWzcuJGCggLCwsIA+Oyzz8jMzGTDhg14e3vTokULioqKzvj8+/fvZ/r06cTExFC3bl0mTJhwVuf5la+v74m/e3p6nvSyZmRkJDt27KBFixYA5ObmMn/+fN0cICIiLuNAVj5RMcnMi03hcF4xAbX8uHNAG67t2YygutXtjvenqk45+4sRropSo0YNBgwYwKRJk35zI8DRo0dp1KgR3t7erFq1igMHDvzpefr168fnn3/OJZdcwtatW9myZQvgLEb+/v7Url2bjIwMvvnmGy6++GIAatasybFjx2jQoMFvztW3b18mTJjA1KlTMcawcOFC5syZc1pfj8PhYO7cucTHxxMYGAjAqlWreOaZZ5g8eTJBQUEsWrSIq666iuLiYsrLyxk0aBBPP/0048aNo3r16mRnZ2v0TEREzrnisnKWb8sgKjqJn/Zm4elhMaB9I8ZGNKN/u4Z4ueAo2clUnXJmo7FjxzJy5Mjf3Lk5btw4rrzySjp37kx4eDgdOnT403PcfvvtTJw4kY4dO9KxY8cTI3Bdu3ale/fudOjQgWbNmnHhhReeOGbKlCkMGTKEwMBAVq1adeL9PXr0YMKECURERABwyy230L1795Newvy9tWvX0rRp0xPFDJzFMSEhgfT0dObMmcOtt97K448/jre3N/PmzWPIkCFs3ryZ8PBwfHx8uOKKK3j++ed57733ALjtttv++psoIiJyCnsO5REVncT8jSkcKSglqG417h/UjmvCmxFQ28/ueGfMcs5Jc3/h4eHm9+t+bd++nY4dO9qUSP4O/exEROTPFJWW83V8OpHRScQkHsHLw2JwaGPG9AzmojYN7L3b8jRYlrXBGBN+so9p5ExERETcxvb0XKKik1i4KZXcojJa1K/O1Ms7cHWPIBrW9P3rE7gBlTMRERFxafnFZSzdksbn0cnEJefg4+nB5Z0DGNMzmAta1cOyXHuU7EypnImIiIjLMcYQn3qUyOhklmxOJb+knLaNavDYsBBGdW9KXX8fuyNWmEpfzowxla5RV3aVZR6kiIicudyiUhZvTiNyfRIJ6bn4eXswtHMg1/dqRo/gulXid3qlLmd+fn5kZWVRv379KvHDrAyMMWRlZZ1YwFZERCo/Ywwbk44QGZ3Msi3pFJaWE9KkFs+MCGV4t6bUruZtd8TzqlKXs6CgIFJSUsjMzLQ7ipwBPz8/goKC7I4hIiIVLKeghAUbU4mKSWJXRh7+Pp5c1T2QsRHBdG5au8oOrFTqcubt7U3Lli3tjiEiIiLHGWNYvz+byOgkvtl6kJIyB12b1WHaqM5c2TUQf99KXU1Oi74DIiIiUuEO5xUzf0MKX8Qks+9wPjX9vBjTsxljegYTEljL7nguReVMREREKoTDYVi39zCR0UmsSMigtNzQs0Vd7hjQhis6N6Gaj6fdEf8odSMc2Q+drrYtgsqZiIiInFMZuUXMi03mi9hkkrMLqVvdm5t6t2BMz2a0bVzT7ngnlxILq1+E3d9BneYQchV42FMeVc5ERETkbyt3GFbvOsTn65NZtfMQ5Q5D71b1+edlHbgstDG+Xi44SgaQtB5WT4O9P0C1ejDwceg52bZiBipnIiIi8jek5hTyRUwy82KTST9aRIMaPkzu24oxPZvRooG/3fFOLXGdc6Rs/2qo3gAGPQ3hN4NvDbuTqZyJiIjImSktd7By+yGiYpJYvcu5XFXftg15fFgIAzs2xsfLw+aEp2AM7F8Dq1+CAz+CfyMY/ByETwQf1ymSKmciIiJyWg5k5RMVk8yXG1LIPFZMQC0/7hzQhmvCm9GsXnW7452aMbBvlbOUJf0MNZvAkBchbDx4V7M73R+onImIiMgpFZeV8922DKJikli3JwsPCy7p0IgxPYO5uH1DvDxddJQMnKVsz/fOy5cpMVCrKVwxHbrfCN6uuxONypmIiIj8wZ5DeURFJ7FgUyrZ+SU0rVON+we145rwZgTUdt1iAzhL2a5vnaUsbRPUDoZhM6Db9eDla3e6v6RyJiIiIgAUlZbzdXw6UdHJRCdm4+VhMSikMWMigunbpgEeHi6+nZLDATuXOS9fHtziXBJj+JvQZQx4+did7rSpnImIiFRx29NziYpOYuGmVHKLymhRvzpTL+/A1T2CaFjT9UeacDhg+xJY8zJkbIV6reCqd6HzNeDpfpumq5yJiIhUQfnFZSzdkkZkdDKbk3Pw8fRgSKcAxkQ0o3er+u6x6bijHLYtdJayzB1Qvy2M/MC5ur+n+1Yc900uIiIiZyw+5SifRyfxVVwaecVltGlUg0eHdmRUjyDq+bvJpb/yMti2wFnKDu+Chh3g6o8hdKSti8eeKypnIiIilVxuUSmLN6cRFZ3EtrRc/Lw9GNo5kLERzQhrXtc9RsnAWcri58Ka6ZC9FxqFwjWzoONw8HDhu0bPkMqZiIhIJWSMYWNSDlHRSSzdkk5haTkdm9Ti6RGhjOjWlNrV3GguVnkpxEXC2lfgSCIEdIbr/g3th1aqUvYrlTMREZFKJKeghAUbU4mKSWJXRh7VfTy5qnsgY3oG0yWotvuMkgGUFcPmz2Htq3A0CQK7w5Bp0G4IuNPXcYZUzkRERNycMYb1+7OJik7i660HKSlz0DWoNi+M6syVXQOp4etmv+5Li2DTHPhxBuSmQNNwGPYqtLm0UpeyX7nZT0tERER+dTivmPkbUvgiJpl9h/Op6efFmJ7NGNMzmJDAWnbHO3OlhbBhFqybAcfSodkFMOJNaDWgSpSyX6mciYiIuBGHw7Bu72GiopP5LuEgpeWG8OZ1+ceANgzt3IRqPm54t2JJAcR+Auteh/xD0PwiGPk+tOxXpUrZr1TORERE3EBGbhHzYpP5IjaZ5OxC6lT35sYLWjA2ohltG9e0O97ZKc6D2I/hpzchP9NZxvrPhBYX2Z3MVipnIiIiLqrcYVi96xCR0cn8sOMQ5Q5D71b1eWBwey4LDcDP2w1HyQCKciHmQ/jpLSjMhtaXQL9/QfPedidzCSpnIiIiLiY1p5AvYpKZF5tM+tEiGtTw4Za+LRnTM5iWDfztjnf2CnMg+gP4+W0oyoG2g52lrFlPu5O5FJUzERERF1Ba7mDl9kNExSSxelcmAH3bNuTxYSEM7NgYHy83Xs+r8Aj88h788i4UH4X2V0C/f0LTHnYnc0kqZyIiIjY6kJXvHCXbkELmsWIa1/Ll/wa04drwZjSrV93ueH9PQbZzlGz9+1ByDDoMg/7/giZd7U7m0lTOREREzrPisnK+25ZBVEwS6/Zk4WHBgPaNGBsRzMXtG+Ll6cajZAD5h52T/GM+gpJ8CBnhHCkL6GR3MregciYiInKe7DmUxxcxSczfmEp2fglN61TjvkHtuCY8iCa1q9kd7+/LOwQ/vQExHzvXLOt0NfR7ABp1tDuZW1E5ExERqUBFpeV8HZ9OVHQy0YnZeHlYXNqxMWN7BXNRmwZ4elSCdbxy052lLPYTKC+BztdC3/uhYTu7k7kllTMREZEKsONgLlHRySzYmEJuURkt6lfnwSEdGB0WRMOavnbHOzeOpjpX898wCxxl0HWMs5TVb213MremciYiInKO5BeXsXRLGpHRyWxOzsHH04PLOgUwtmczLmhVH4/KMEoGkJMMP77m3P/SOKDb9XDRfVCvpd3JKgWVMxERkb8pPuUokTFJLNmcRl5xGW0a1eDRoR0Z1SOIev4+dsc7d44kwtpXYfPnzsfdb4C+90GdYFtjVTYqZyIiImfhWFEpizenERmdxLa0XHy9PBjapQljI4IJb14XqzLtCZm111nK4iLBwxPCJsBF90DtILuTVUoqZyIiIqfJGMPGpByiopNYuiWdwtJyOgTU5OkRoYzo1pTa1bztjnhuHd4Da6fDlrng6Q0RU+DCu6BWoN3JKjWVMxERkb+QU1DCwk2pREUnszPjGNV9PBnRLZAxEcF0DapduUbJAA7tcJayrfPB0xcuuB363AU1G9udrEpQORMRETkJYwzr92cTFZ3E11sPUlLmoEtQbV4Y1ZkruwZSw7cS/grN2AZrXoZti8C7OvS5E3rfCTUa2p2sSqmE/7JERETOnjGGJXFpvL5yN/sy86np68V14c0YE9GM0MDadserGOlbnKVs+xLwqemc5H/BHeBf3+5kVZLKmYiIyHEHsvJ5dNFW1u4+TGhgLV4e3YWhXZpQ3aeS/rpM2wSrX4ady8C3NvR/EHrdBtXr2Z2sSquk/9pEREROX2m5gw/W7OONlbvx9vTgqeGh3HBB88qxev/JpGyA1S/C7uXgVxsufhh63QrV6tidTFA5ExGRKm7DgSM8vCCenRnHGBIawJPDQwmo7Wd3rIqRHA3/mQZ7V0K1unDJY847MP1q2Z1M/ofKmYiIVElHC0t56dsdfB6dREAtPz68KZxBIZX0bsQDPzlHyvb9B6rXh0ufhJ63gG9Nm4PJyaiciYhIlWKMYVl8Ok99lUBWXjET+7TkvsHtKt/dl8ZA4lpY/ZLzrX9DGPwshE8CH3+708mfqGT/EkVERE4tObuAxxdvZdXOTDo1rcUn43vSOaiS3YFpjHOEbPVLkPQT1AiAIdOgx3jwqW53OjkNKmciIlLplZU7+GTdfl5bsRvLgkeHdmRCnxZ4eXrYHe3cMQb2rHRevkyJhpqBcPnL0OMm8K6kc+gqKZUzERGp1OKSc3hoQTwJ6blc2rERT43oRNM61eyOde4YA7uWO0tZ2kao3QyGvurclNzL1+50chZUzkREpFI6VlTKK9/tYtbPiTSs4cu743owpFNA5dlqyRjY+bWzlKXHQZ1guPIN6DoWvHzsTid/g8qZiIhUOsu3HeSJxdvIOFbEjRc054HL2lPLr5JsSu5wwI6vnIvHZsRD3ZYw4m3ocp1zc3JxeypnIiJSaaTlFPLEkm2sSMigQ0BN3rmhBz2C69od69xwlEPCImcpy9wO9dvAyPeh02jw1K/zykQ/TRERcXvlDsOsnxJ55budlBvD1Ms7cPNFLfGuDBP+HeWwdYFz78vDO6FBe7j6YwgdCR6edqeTCqByJiIibm1r6lEeXhjPlpSj9G/XkGev6kSzepVgyYjyMoifB2unQ9YeaBQC13wKHUeARyUonXJKKmciIuKW8ovLeG3FLj5Zt596/r68MbY7V3Zp4v4T/stLIS7KWcqOJELjznDtHOgwTKWsiqjQcmZZ1t3AZMACPjTGzLAsqx7wBdACSASuNcYcOcmx44FHjz981hgzqyKzioiI+/hhRwaPLdpGak4hYyOCmTqkA7Wru/lk+LISiPsc1r4COUnQpBuMiYT2l4O7F045IxVWzizL6oSzmEUAJcC3lmUtBaYAK40x0yzLmgpMBR783bH1gCeAcMAAGyzLWnKyEiciIlVHRm4RT321ja/jD9K2UQ3m3dabni3q2R3r7ykrhk1zYO1rkJsCTcPgiunQdrBKWRVVkSNnHYH1xpgCAMuyVgOjgBHAxcefMwv4D78rZ8BlwApjTPbxY1cAQ4DICswrIiIuyuEwfLb+AC99u5PicgcPDG7HlH6t8fFy48t8pYWwcTb8OAOOpUFQBAx/HVoPVCmr4iqynG0FnrMsqz5QCFwBxAKNjTHpx59zEGh8kmObAsn/8zjl+PtERKSK2XEwl4cWxLMpKYc+revz3MjOtGzgxht3lxTAhpmw7nXIy4DgPjDyXWjZX6VMgAosZ8aY7ZZlvQh8B+QDm4Hy3z3HWJZlzvZzWJY1BedlUoKDg/9GWhERcTWFJeW8vnI3H63dR61q3rx6bVdGdm/qvhP+i/Mg9hP46Q3Iz4QWfZ1LYrTsa3cycTEVekOAMeZj4GMAy7KexzkClmFZVhNjTLplWU2AQyc5NJX/XvoECMJ5+fP35/8A+AAgPDz8rEueiIi4ljW7MnlkUTzJ2YWMDgvi4Ss6Us/fTbckKj4G0R/Cz29BQRa0GgD9/wXN+9idTFxURd+t2cgYc8iyrGCc880uAFoC44Fpx98uPsmhy4HnLcv6dVnnwcBDFZlVRETsl3msmGeXJbB4cxqtGvgTOfkCereub3ess1N0FKI/gJ/fhsIj0GaQs5Q1i7A7mbi4il7nbP7xOWelwB3GmBzLsqYBcy3Luhk4AFwLYFlWOHCbMeYWY0y2ZVnPADHHz/P0rzcHiIhI5eNwGObGJvPCNzsoLCnn7oFtuf3i1vh5u+EK+IVHYP378Ms7zoLWboizlDUNszuZuAnLmMpxNTA8PNzExsbaHUNERM7QnkPHeHjBVqITs4loWY/nR3amTaMadsc6cwXZzkK2/n0oznUuGtvvnxDYze5k4oIsy9pgjAk/2ce0Q4CIiNiiqLScd1bt4d3Ve6nu48VLV3dhdFgQHh5uNuE/P8s5nyz6AyjJg5ARzlIW0NnuZOKmVM5EROS8+2nPYR5ZtJX9h/O5qlsgjw4LoUENX7tjnZm8Q/DTmxDzMZQWODci7/dPaBxidzJxcypnIiJy3mTnl/Dcsu3M35hC8/rVmXNzBH3bNrQ71pk5dhDWveFcFqO8GDqNhn4PQMP2dieTSkLlTEREKpwxhvkbU3luWQLHisq4Y0Br7rykrXtN+M9Nc67mv+FTcJRBl+ug7/3QoI3dyaSSUTkTEZEKtS8zj0cWbuXnfVmENa/L8yM70z6gpt2xTl9OMqyb4dxqyTig6xi46D6o39ruZFJJqZyJiEiFKC4r5/3V+3hr1R58vTx4bmQnxvYMdp8J/0cOwI+vwqbPnI+7j4OL7oW6LWyNJZWfypmIiJxz0fuzeXhhPHsO5TG0SxOeGBZCo1p+dsc6PSX5sOp5WP8eWB4QNh4uvAfqNLM7mVQRKmciInLO5BSUMO2bHUTFJNO0TjVmTujJgA6N7I51+vashKX3QE4S9BgP/R+E2k3tTiVVjMqZiIj8bcYYlsSl8czSBI4UlHJrv1bcfWlbqvu4ya+ZgmxY/gjEfQ7128DEb7T3pdjGTf7XiIiIqzqQlc+ji7aydvdhugbVZtakCEIDa9sd6/QYA1vnwzcPQlEO9H3AuVaZt5tcgpVKSeVMRETOSmm5gw/X7uP173fj7enBU8NDueGC5ni6y4T/oymw7H7Y9S0E9oDhiyGgk92pRFTORETkzG04cISHF8SzM+MYl4U25snhoTSpXc3uWKfH4YDYj+H7J51LY1z2PPS6DTzcaM01qdRUzkRE5LQdLSzl5eU7+Gx9EgG1/PjgxjAGhwbYHev0Ze6EJXdC8npofQkMe01LY4jLUTkTEZG/ZIzh6/iDPPnVNrLyipnQpwX3D25PDV83+TVSVgI/vgZrp4OPP4x837nCv+Uml2ClSnGT/1UiImKXlCMFPL54Gz/sOERoYC0+Hh9Ol6A6dq2cPeMAACAASURBVMc6fckxztGyzO3OfTCHTIMabrafp1QpKmciInJSZeUOZq5L5NUVu7AseHRoRyb0aYGXp4fd0U5PcR788Aysfx9qBcLYL6D9ELtTifwllTMREfmDuOQcHloQT0J6LgM7NOKpEaEE1a1ud6zTt3sFLL3XeUdmxGQY+Dj4utF+nlKlqZyJiMgJecVlTF++k9k/J9Kghi/vjuvBkE4BWO4yNyv/MHz7EMTPhQbtYdJyCO5ldyqRM6JyJiIiACzfdpAnFm8j41gRN/Rqzj+HtKeWn7fdsU6PMbBlLnw7FYqPQf+p0Pc+8PK1O5nIGVM5ExGp4tKPFvLE4m18l5BBh4CavHNDD3oE17U71unLSXJewtzzPQT1hOFvQqOOdqcSOWsqZyIiVVS5wzD750SmL99JuTFMvbwDN1/UEm93mfDvKIfoD2Hl087Hl78EPW/RYrLi9lTORESqoK2pR3l4YTxbUo7Sr11Dnh3RieD6bjThPyPBuTxGaiy0GeRcTLZOM7tTiZwTKmciIlVIQUkZr63YxSfrEqlb3ZvXx3RjeNdA95nwX1YMa6Y7F5T1qwWjPoLOo7WYrFQqKmciIlXEDzsyeGzRNlJzChkb0YwHh3SgTnUfu2OdvqRfnKNlh3c5V/e/7AXwr293KpFzTuVMRKSSO5RbxFNfJbAsPp02jWow99beRLSsZ3es01eUCyufgpiPoHYwjJsPbS+1O5VIhVE5ExGppBwOw2fRSbz0zQ6Kyx3cP6gdt/ZvjY+Xm0z4B9j5LSy7D3LT4IJ/wIBHwLeG3alEKpTKmYhIJbTjYC4PLYhnU1IOfVrX59mrOtGqoRuVmrxD8M2DsG0BNAqBa2dDULjdqUTOC5UzEZFKpLCknDd+2M2Ha/ZR08+LV67pyqgeTd1nwr8xEBfpXOW/tAAGPAoX3g1ebjQ3TuRvUjkTEakk1uzK5NFFW0nKLmB0WBAPX9GRev5uVGqOJMJX98C+VdDsAhj+BjRsb3cqkfNO5UxExM1lHivm2WUJLN6cRssG/nw+uRd9WjewO9bpKy+D9e/BqufA8oShr0DYJPBwo7lxIueQypmIiJtyOAxzY5N54ZsdFJSUcdfAtvzj4tb4ebvRCvkH453LY6RtgnZDnMWsdpDdqURspXImIuKG9hw6xsMLthKdmE1Ei3o8P6oTbRrVtDvW6SstgjUvwbrXoVpdGD0TQkdqMVkRVM5ERNxKUWk576zaw7ur91Ldx4sXr+7MNWHN8PBwo1KT+CN8dTdk7YFu42Dws1DdjdZdE6lgKmciIm7ipz2HeWTRVvYfzueqboE8OiyEBjV87Y51+oqOworHYcOnUKc53LgIWg+wO5WIy1E5ExFxcdn5JTy3bDvzN6YQXK86sydF0K9dQ7tjnZntS2HZ/ZB/CPrcCRc/BD7+dqcScUkqZyIiLsoYw/yNqTy3LIFjRWX84+LW3DWwrXtN+D+WAV8/ANuXQONOMDYSmvawO5WIS1M5ExFxQfsy83hk4VZ+3pdFj+A6PD+qMx0Catkd6/QZA5vmwHePOif/D3wc+twFnt52JxNxeSpnIiIupLisnPdX7+OtVXvw9fLg2as6cX1EsHtN+M/a65zwn7gWml8IV74BDdrYnUrEbaiciYi4iOj92Ty8MJ49h/IY2qUJTwwLoVEtP7tjnb7yMvj5LfjPC+DpA8NmQI/xWkxW5AypnImI2OxoQSkvfLOdqJhkmtapxswJPRnQoZHdsc5M2mbnYrIHt0CHYXDFdKjVxO5UIm5J5UxExCbGGJbEpfHM0gSOFJQypV8r7rm0LdV93OiluaQAVk+Dn94C/wZw7RwIGW53KhG35kavACIilUdSVgGPLt7Kml2ZdA2qzaxJEYQG1rY71pnZt9o5t+zIfuhxEwx62rnav4j8LSpnIiLnUWm5g4/W7uf1lbvwtCyevDKEG3u3wNOdJvwXHnHehbnp31C3JYz/Clr2szuVSKWhciYicp5sTDrCwwvi2XHwGJeFNubJ4aE0qV3N7linzxhIWAxf/xMKsuDCe+DiqeDtRl+DiBtQORMRqWC5RaW89O0OPlufREAtPz64MYzBoQF2xzozuWnOUrZjKQR0gXHzILCb3alEKiWVMxGRCmKM4ZutB3lyyTYO5xUzoU8L7h/cnhq+bvTS63DAxk9hxRNQXuKcV3bBHeDpRl+DiJvR/y4RkQqQcqSAxxdv44cdhwgNrMVH48PpElTH7lhn5vBu54T/A+ucc8qGzYD6re1OJVLpqZyJiJxDZeUOPv0pkVe+2wXAo0M7MqFPC7w83Wgh1vJSWPc6rH4JvP1g+FvQ/Qaw3OimBRE3pnImInKObEnJ4aEF8WxLy+WSDo14ekQoQXWr2x3rzKRugCV3QcZWCLkKLn8Jaja2O5VIlaJyJiLyN+UVl/HKdzuZ9VMiDWr48s64HlzeKQDLnUaaSvJh1fPwyztQozGM+Rw6DLU7lUiVpHImIvI3fLftIE8s2cbB3CLG9QrmX0M6UMvP2+5YZ2bvD/DVPZBzAMImwqCnwM/NFsQVqURUzkREzkL60UKeWLyN7xIyaN+4Jm9d34Ow5m62On5BNix/BOI+h/ptYMLX0OJCu1OJVHkqZyIiZ6DcYZjzcyLTv9tFabmDfw1pz+S+rfB2pwn/xsC2BfDNg87V/vs+AP3+6Zz8LyK2UzkTETlN29KO8vCCeOJSjtK3bQOeu6ozwfXdbML/0RRYdj/s+hYCu8ONCyGgs92pROR/qJyJiPyFgpIyZny/m49/3E/d6t68PqYbw7sGuteEf4cDYj+G758CUw6XPQ+9bgMPT7uTicjvqJyJiPyJVTsO8eiiraTmFDKmZzOmXt6BOtV97I51ZjJ3wpI7IXk9tBoAV86Aui3sTiUip6ByJiJyEg6H4cVvd/D+mn20aVSDubf2JqJlPbtjnZmyEvjxNVg7HXz84ar3oOsYLSYr4uJUzkREfqeotJz75m7m6/iD3HBBMI8NC8HXy80u/yXHOEfLMrdDp6thyDSo0cjuVCJyGlTORET+R1ZeMbfMjmVzcg6PDu3IzRe1dK+5ZcV58MMzsP59qBUIY7+A9kPsTiUiZ0DlTETkuL2ZeUycGUNGbhHvjuvBkE5N7I50ZnZ/D0vvcd6R2fMWGPg4+NWyO5WInCGVMxERYP2+LKbM2YCXh0XUlAvoHuxGC8rmZ8G3UyF+LjRoD5O+heAL7E4lImdJ5UxEqrzFm1P557wtBNWrxqcTItxn7TJjIH6es5gV5UL/B6Hv/eDla3cyEfkbVM5EpMoyxvDWD3t4ZcUuerWsx/s3hrnPMhk5SbD0XtjzPTQNh+FvQuMQu1OJyDmgciYiVVJpuYOHF8Qzb0MKI7s3ZdrVnd3jjkxHOUR/CCufdj4e8iJETNZisiKViMqZiFQ5RwtL+cdnG1i3J4u7Brbl3kvbuscdmRkJzuUxUmOhzaUw7DWoE2x3KhE5x1TORKRKSTlSwKRPY9iXmc/0a7oyOizI7kh/rawY1kx3LijrWxNGfQidr9FisiKVlMqZiFQZ8SlHmTQrhqLScmZPiqBPmwZ2R/prSb84R8sO74Iu1zn3xPR3g9wictZUzkSkSvg+IYM7IzdRz9+Hz2/pRdvGNe2O9OeKcp3zymI+gtpBMG4+tL3U7lQich6onIlIpffpuv08vTSBTk1r89H4cBrV9LM70p/b+S0suw9y06DXbXDJo+Bbw+5UInKeqJyJSKVV7jA8t2w7n6zbz6CQxrw+phvVfVz4ZS8vE759ELbOh0YhcO1sCAq3O5WInGcu/ColInL2CkrKuDtqMysSMph0YUseGdoRTw8XnUBvDMRFwvKHoSQfBjwCF94DXm6y5pqInFMqZyJS6Rw6VsQts2LZmnqUJ68MYcKFLe2OdGpHEuGre2DfKmjWC658Axp1sDuViNhI5UxEKpXdGceYMDOG7PwS3r8xnEEhje2OdHLlZbD+PVj1HFgecMV0CL8ZPDzsTiYiNlM5E5FK46c9h7n13xvw8/Zk7q296RxU2+5IJ3cw3rk8RtomaDcEhr7ivCNTRASVMxGpJL7ckMLU+Vto1dCfTyb0JKiuC25eXloEa16Cda+DXx0Y/QmEjtJisiLyGypnIuLWjDG89v1u3li5mwvb1OedcWHUruZtd6w/SlwHX90FWXug2zgY/CxUr2d3KhFxQSpnIuK2isvKeWh+PAs2pXJNWBDPjeyMj5eLzdkqOgornoANM6FOc7hxIbS+xO5UIuLCVM5ExC0dLShlypxY1u/P5oHB7bhjQBvX27x8+1L4+gHIy4De/wcDHgYff7tTiYiLq9ByZlnWvcAtgAHigYlAH2A64ANsAG42xpSd5Njy48cAJBljhldkVhFxH0lZBUz4NJqU7EJeH9ONEd2a2h3pt45lwDf/hITF0LgTjPkMmobZnUpE3ESFlTPLspoCdwEhxphCy7LmAtcDTwEDjTG7LMt6GhgPfHySUxQaY7pVVD4RcU+bko5wy6xYyhyGOTdH0KtVfbsj/ZcxsGkOfPeoc/L/JY/BhXeDpwvOgRMRl1XRlzW9gGqWZZUC1YF8oMQYs+v4x1cAD3HyciYi8hvfbk3n7qjNNK7lx8yJPWnd0IX2m8zaC1/dDYlrofmFcOXr0KCt3alExA1V2MxZY0wqzsuXSUA6cBSYC3hZlvXrZnGjgWanOIWfZVmxlmX9YlnWVRWVU0RcnzGGj9bu4/bPNhISWIuF/+jjOsWsvAx+nAHv9oH0OBg2A8YvVTETkbNWkZc16wIjgJZADjAPGAeMAV6zLMsX+A4oP8UpmhtjUi3LagX8YFlWvDFm7+8+xxRgCkBwcHDFfCEiYquycgdPfZXAnF8OcHmnAF67rht+3p52x3JKj4PF/wcHt0D7oTB0OtQKtDuViLi5iryseSmw3xiTCWBZ1gKgjzHm30Df4+8bDLQ72cHHR94wxuyzLOs/QHdg7++e8wHwAUB4eLipmC9DROySX1zGnZGb+GHHIW7t14oHh3TAwxU2Ly8thP+8AD+9Bf4N4NrZ0HG4FpMVkXOiIstZEnCBZVnVgUJgIBBrWVYjY8yh4yNnDwLP/f7A46NuBcaYYsuyGgAXAi9VYFYRcTEZuUVM+jSG7em5PHtVJ264oLndkZz2r4Eld8GR/dD9Rhj8DFSra3cqEalEKqycGWPWW5b1JbARKAM24RzletayrGE457u9a4z5AeD4PLTbjDG3AB2B9y3Lchx/3jRjTEJFZRUR17LjYC4TZ8aQW1jKxxN6MqB9I7sjQeER+O4x592YdVvCTUugVX+7U4lIJWQZUzmuBoaHh5vY2Fi7Y4jI37RmVyb/+Gwj/r6efDKhJ6GBNm9ebgxsXwJf/xPyD0Of/4P+U8HHBffuFBG3YVnWBmNM+Mk+ph0CRMRlREUn8ciirbRtVIOZE3vSpHY1ewMVZMOSO2HHUgjoAtfPhUAtvygiFUvlTERs53AYpn+3k3f+s5d+7Rry9vXdqeln88Kt+Ydh9lVweCdc+pRz+yVPvWSKSMXTK42I2KqotJwH5sWxdEs6YyOCeXpEKN6eNm9efiwDZo9wTvofGwVtBtqbR0SqFJUzEbFNdn4JU2bHEnvgCFMv78Ct/VrZv3l5bhrMGg65qc7LmJr0LyLnmcqZiNhi/+F8Js6MJu1oEW9d351hXVxg8dacZJh1JeRnwg3zoXkfuxOJSBWkciYi511sYjaTZzvvro6c3Iuw5vVsTgQcSXQWs8KjcOMiaNbT7kQiUkWpnInIefVVXBr3z4ujaZ1qzJzQkxYN/O2O5Ny0fNZwKMmD8YshsLvdiUSkClM5E5HzwhjDe6v38eK3O+jZoi4f3BhOXX8fu2NB5i7niJmjFCYshYDOdicSkSpO5UxEKlxpuYPHF28lMjqZK7sG8vLoLq6xeXlGAsweDlgwfik0DrE7kYiIypmIVKxjRaXc8fkm1uzK5I4Brbl/UHvX2Lw8fYtzuQxPHxj/FTRsZ3ciERFA5UxEKlBaTiGTPo1h96E8Xry6M9f1DLY7klPqRpgzEnxqwPglUL+13YlERE5QORORCrE19Sg3z4qhoLicTyf2pG/bhnZHckqOgX+Pgmp1nCNmdVvYnUhE5DdUzkTknFu14xB3fL6ROtW8mXd7bzoE1LI7ktOBn+Cza8C/obOY1WlmdyIRkT9QORORc2rOz4k8sWQbIYG1+Hh8TxrX8rM7ktO+1RA5Bmo1dRazWk3sTiQiclIqZyJyTjgchmnf7uCDNfu4pEMj3hzbHX9fF3mJ2bMSoq6Hui2dc8xqNLI7kYjIKbnIK6eIuLOi0nLu/WIz32w9yE29m/P4sBC87N68/Fe7lsMXN0CD9nDTIvBvYHciEZE/pXImIn/L4bxiJs+OZXNyDo8O7cjNF7W0f/PyX23/CuZNhMahcONCqO4C20SJiPwFlTMROWt7DuUx8dNoMo8V8+64MIZ0CrA70n9tXQDzb4GmPWDcl867M0VE3IDKmYiclV/2ZXHrnA14e1pETelNt2YuVH7ivoBFt0GzXjBuHvjWtDuRiMhpUzkTkTO2cFMK//pyC8H1qvPpxAia1atud6T/2vRvWPx/0OIiuP4L8HGBjdVFRM6AypmInDZjDG/+sIdXV+ziglb1eP+GcGpX97Y71n/FfgJL74XWl8B1n4GPC5VGEZHTpHImIqelpMzBwwvj+XJDCqO6N2Xa1V3w8XKROzIB1r8P3/wL2l4G184GbxdZX01E5AypnInIXzpaWMrt/97AT3uzuHtgW+65tK3r3JEJsO4NWPEYdBgGo2eCl4/diUREzprKmYj8qZQjBUycGUNiVj7Tr+nK6LAguyP91prp8MMzEDoSRn0Ini50mVVE5CyonInIKW1JyWHSp7EUl5Uza1IEfVq70AKuxsB/psHqadDlOhjxDnjqJU1E3J9eyUTkpL7bdpC7ozZTv4YPUVN60aaRCy1HYQysfAp+fA263QDD3wAPT7tTiYicEypnIvIHn/y4n2eWJdClaW0+Gt+ThjV97Y70X8bA8kfgl7chbCIMfRU8XOjGBBGRv0nlTEROKHcYnlmawKc/JXJZaGNmXNedaj4uNCLlcDjvyIz5ECJuhctfBFe6MUFE5BxQORMRAApKyrgrcjPfb8/g5ota8vAVHfH0cKHi43DA0ntg4yzocycMekbFTEQqJZUzEeHQsSJumRXL1tSjPDU8lPF9Wtgd6bcc5c5V/+M+h773wyWPqZiJSKWlciZSxe3KOMbEmTFk55fwwY3hXBrS2O5Iv1Ve5twnM34eXPww9P+XipmIVGoqZyJV2Lo9h7nt3xvw8/Zk7q296RxU2+5Iv1VeCvNvhoTFMPAJ6Huf3YlERCqcyplIFTUvNpmHFsTTqqE/MydG0LRONbsj/VZZMcybCDuXwWXPQ+877E4kInJeqJyJVDHGGF5dsYs3f9jDRW0a8M4NPajl52Kr6pcWwdwbYfd3cMV0iJhsdyIRkfNG5UykCikuK+fBL7ewaHMa14YH8dzIznh7utgaYSUFEDUW9q2GK1+HsAl2JxIROa9UzkSqiJyCEqbM2UD0/mweGNyOOwa0ca3NywGK8yByDCT+CCPehu7j7E4kInLeqZyJVAFJWQVM+DSalOxCXh/TjRHdmtod6Y+KcuGzayAlxrmBeZdr7E4kImILlTORSm5j0hEmz4ql3Bj+fUsvIlrWszvSHxXmwL+vhvTNMPoTCL3K7kQiIrZROROpxL6JT+eeLzYTUNuPmRN60qphDbsj/VFBNsy5CjIS4NrZ0GGo3YlERGylciZSCRlj+HDtPl74Zgfdm9Xhw5vCqV/DhTYv/1VeprOYHd4NYz6HdoPtTiQiYjuVM5FKpqzcwZNfbePfvyQxtHMTXrm2K37eLrR5+a+OZcDs4XDkAFwfBa0vsTuRiIhLUDkTqUTyisu48/ONrNqZya39W/HgZR3wcKXNy3+VmwazroTcdBg3D1r2tTuRiIjLUDkTqSQOHi1i0qcx7Mw4xnMjOzGuV3O7I51cTrKzmOUfhhsXQPAFdicSEXEpKmcilcD29FwmfRpDbmEpH40PZ0D7RnZHOrns/TBrOBQdhZsWQVC43YlERFyOypmIm1u9K5M7PttIDV8v5t3Wh5DAWnZHOrmsvc4Rs9ICGL8YArvbnUhExCWpnIm4sc/XJ/HY4q20a1yTTyaE06S2i21e/qvMnc4RM0cpjP8KAjrbnUhExGWpnIm4IYfD8NLynby3ei/92zXk7XE9qOHrov+dMxKcd2ViwYRl0Kij3YlERFyai76ai8ipFJWWc/+8OJZtSWdcr2CeGh6Kl6ttXv6r9DiYfRV4+TpHzBq0tTuRiIjLUzkTcSPZ+SVMnh3LhgNHeOjyDkzp18r1Ni//VeoGmDMSfGvB+CVQr5XdiURE3ILKmYib2H84n4kzo0k7WsTb1/dgaJcmdkc6teRo516Z1eo6R8zquuiyHiIiLkjlTMQNxCRmM3l2LB6WReTkXoQ1d8HNy3+VuA4+vxZqNHaOmNUOsjuRiIhbUTkTcXFL4tJ4YG4cQXWrMXNiT5rX97c70qnt+w9EjnUWsvFfQc0AuxOJiLgdlTMRF2WM4Z3/7OXl5TuJaFGP928Mo66/j92xTm3P9xA1zjm37KbFUMNFF8IVEXFxKmciLqi03MGjC7fyRWwyI7oF8tLoLvh6ueDm5b/a+Q3MvQkatocbF4N/fbsTiYi4LZUzEReTW1TKHZ9tZO3uw9x5SRvuG9TOde/IBEhYAl9OdC4se8MCqO7C8+FERNyAypmIC0nNKWTSzBj2Zubx0tVduLZnM7sj/bmt82H+ZGgaBjd8CX617U4kIuL2VM5EXMTW1KNM+jSGwpJyPp0YwUVtG9gd6c/FRcGi2yG4N1z/BfjWtDuRiEiloHIm4gJWbs/gzshN1KnmzZe396F9gIsXnY1zYMmd0LIvjI0CHxe+g1RExM2onInYbM7PiTyxZBshgbX4ZHxPGtXyszvSn4v5GJbdB60HwpjPwNtFN1sXEXFTKmciNnE4DM9/vZ2PftzPpR0b8fqY7vi76ublv/rlPfj2QWg3BK6ZBd4uXiRFRNyQi/8mEKmcCkvKufeLzXy77SAT+rTgsWEheHq48B2ZAOtehxWPQ8cr4epPwMuF11wTEXFjKmci51nmsWJumR3LlpQcHh8WwqSLWtod6a+tfhlWPQuho2DUB+DpbXciEZFKS+VM5Dzac+gYE2bGcDivmPduCOOyUBff3sgYWPU8rHkJuoyBEW+Dp142REQqkl5lRc6Tn/dmceucWHy8PIia0ptuzerYHenPGQPfPwnrZkD3G+DKN8DDhXcpEBGpJFTORM6DBRtTeHD+FprX92fmhJ40q1fd7kh/zhhY/jD88g6E3wxXTAcPD7tTiYhUCSpnIhXIGMMbK/fw2ve76N2qPu/dEEbt6i4+X8vhgG/+CTEfQa/bYcgL4MrbR4mIVDIqZyIVpKTMwUML4pm/MYVRPZoybVQXfLxcfPTJ4YCld8PG2dDnLhj0tIqZiMh5pnImUgGOFpZy25wN/Lwvi3svbcddA9u49ublAI5yWHwHxEVCv3/CgEdUzEREbKByJnKOJWcXMPHTGA5k5fPqtV0Z1SPI7kh/rbwMFt4KW790lrL+/7I7kYhIlaVyJnIObU7O4ZZZMZSUOZg9qRe9W9e3O9JfKyuB+TfD9iVw6VNw0f+3d9/hVVXZ/8ffixRC6L1DgARULIgREQSpFnDAseKI3XF0rKDOqFOccZozPwVsYy+oY+8CKqEKFqSKIJKE0FtCCy09+/fHvX5FCHCTnFvzeT1PHm/O3WeftTnxZmWds8++I9wRiYjUaErORDzy2fIt3P7GYprVq80bN/QmtUWEL14OUFoEb10FmZ/A2f+C038b7ohERGo8JWciHnh+7mr+Pvl7TmzXiOeuTKd5/drhDunoSgrgzdGQPc33qIxevw53RCIigpIzkWopK3f8bdL3vPTlGs7u3pIJl55MncQoeFBr8X544zLIme17uOwpV4U7IhER8VNyJlJF+4tLue31xUxbkcv1Z3Ti3mHHRv7i5QBFe+G1S2Hdl3D+k9DjsnBHJCIiB1ByJlIFubsLuW7iApZvyueBkd258vSUcIcUmMJ8+N/FsGEBXPAsnHBRuCMSEZGDKDkTqaSVW/Zw7Uvz2bGvmGevTGfwsS3DHVJgCnbCKxfAlqVw8Ytw3MhwRyQiIhVQciZSCXOztnHTqwupkxjH2zeezvFtG4Y7pMDs3wEvj4S8H+CSV+CYYeGOSEREDkPJmUiA3pq/nvve/44uzevxwjWn0rZRnXCHFJi9eb7EbHs2jHod0oaEOyIRETmCgJIzM+sCbHDOFZnZAOBE4GXn3K5gBicSCZxzjMvI5LEZ2fRLa8YTl/ekQVKEL17+oz1bYOII2LUOLn8LOg8Id0QiInIUga7C/C5QZmapwDNAe+C1o+1kZmPMbLmZLTOz180sycwGmdki/7aJZlZhgmhmV5lZlv9L8/wlbGZn5vHYjGwuPqUdL1x9avQkZvkb4cVhkL8BRr+jxExEJEoEmpyVO+dKgV8Cjznn7gZaH2kHM2sL3AakO+eOB+KAXwETgVH+bWuBQxIvM2sC3A+cBvQC7jezxgHGKuKZH6tmbRvV4R+/PIGEuED/lwmzXevgpWGwNxeueB9Szgh3RCIiEqBAf9OUmNll+BKpSf5tgZQP4oE6/upYMrAPKHbOZfrfzwAurGC/s4EM59wO59xOf7tzAoxVxDPTVuSydEM+tw1OJTE+ShKzHat9FbOCnXDlh9DhtHBHJCIilRDob5trgNOBfzjnVptZJ+CVI+3gnNsIPASsAzYD+cBbQLyZpfubXYTvEunB2gLrD/h+g3+bSMiUl/uqZh2bJnNBz3bhDicw27J9iVnxXrjyI2h3SrgjEhGRSgooOXPOfe+cu80597r/8mJ9YcYxeAAAIABJREFU59y/j7SPv91IoBPQBqgLXA6MAsab2TfAHqCsqsGb2Q1mtsDMFuTl5VW1G5EKfbp8Cys27+aOIWnRcTkz9wffpcyyYrh6MrTpEe6IRESkCgL6jWNms8ysgf9esEXAs2Y27ii7DQFWO+fynHMlwHtAH+fcV865fs65XsDnQGYF+27k5xW1dv5tP+Oce8Y5l+6cS2/evHkgQxEJSFm5Y3xGJqkt6jHipCgo2m5ZBi8N972+ejK07B7eeEREpMoCLQc0dM7tBi7A9wiN0/AlX0eyDuhtZslmZsBgYIWZtQAws9rA74GnKtj3M+AsM2vsr8Cd5d8mEhKTlm4iK3cvdwxJi/z1MjctgYnnQVwiXD0FWhwT7ohERKQaAk3O4s2sNXAJP00IOCLn3DzgHXyVtu/8x3oGuNvMVgBLgY+dczMAzCzdzJ7z77sD+Bsw3//1gH+bSNCVlpUzYVoWx7Sqz7DjjzgpOfw2LISXR0BiPbhmMjRLDXdEIiJSTYGuEPAAvsrVF865+WbWGcg62k7OufvxPRLjQHf7vw5uuwC4/oDvXwBeCDA+Ec+8v3gjq7ft4+krTqFWJFfN1s2DVy+Euk3hqo+hUYdwRyQiIh4IKDlzzr0NvH3A9zlU/AgMkahWUlbOozOyOKFtQ846LoIXNF8zF/53CTRo7ZuV2TAK7osTEZGABDohoJ2ZvW9muf6vd80sSp4tIBK4txdsYP2OAsYO7YrvVskIlDMLXr0IGrbz3fyvxExEJKYEes/Zi8BH+B6J0Qb42L9NJGYUlZbx+IwsTu7QiAHdInT2b9Y0eO1SaNLZl5jVbxXuiERExGOBJmfNnXMvOudK/V8vARH620ukat74Zj2b8gu5c2i3yKyarfwE3rgMmnWFqydBPf0vKCISiwJNzrab2Wgzi/N/jQa2BzMwkVAqLCnjiZnZ9OrUhL6pTcMdzqG+/xDeHA2tToCrPoLkJuGOSEREgiTQ5OxafI/R2IJvKaaLgKuDFJNIyL369Vpy9xRxZyTea/bdO/D2NdD2FLjiA6jTONwRiYhIEAW6fNNa59wI51xz51wL59z5aLamxIh9RaU8OWsVZ6Q247TOEVY1W/I6vPdr6NAbRr8HSQ3CHZGIiARZdRYMHOtZFCJhNPGrNWzfV8zYs7qGO5SfW/QyfHATpPSDy9+G2vXCHZGIiIRAdZKzCLv2I1J5ewpLeObzHAZ2a07PDhF0ufCbZ+GjWyF1MPzqTUisG+6IREQkRKqTnDnPohAJkxfmrmHX/hLGDu0W7lB+8tV/Ycpd0G0YjHoNEuqEOyIREQmhI64QYGZ7qDgJM0C/MSSq5e8v4bm5OZx1XEtOaNcw3OH4zJ0A0+6HY0fAhc9DfGK4IxIRkRA7YnLmnKsfqkBEQu3ZOTnsKSxlzNAIudds9n9g5j/g+Avhl89AXKBL34qISCzRp7/USDv2FfPiF6sZfmJrjm0d5hmQzvmSss//H5x0GYx8AmrFhTcmEREJGyVnUiM9PXsVBSVljBmSFt5AnIOMP8OXj0LPK+G8R6BWdW4FFRGRaKfkTGqc3D2FTPxqDSN7tCW1RRiv3DsHn94L856EU6+Hc/+fEjMREVFyJjXPk7NWUVLmuH1wGKtm5eW+GZkLnofev4Wz/wmRtjKBiIiEhZIzqVE25xfwv3nruLBnW1KahenZYeVl8PHtsPgV6HsHDPmLEjMREfk/Ss6kRnliZjbOOW4dFKaqWVkpfHgzLH0Dzvw9DLhXiZmIiPyMkjOpMTbs3M+b89dzSXp72jdJDn0AZSXw3g2w/D0Y9Efof3foYxARkYin5ExqjMemZ2Nm3DIoNfQHLy2Gd6+FFR/D0Aeg7+2hj0FERKKCkjOpEdZs28c7izZwRe+OtG4Y4sUtSovgrasg8xM450HofVNojy8iIlFFyZnUCI9OzyIhzvjtwC6hPXBJAbxxOayaDsPHwanXhfb4IiISdZScSczLzt3LB0s2cn2/zrSonxS6Axfvg9dHweo5MOJx6HlF6I4tIiJRS8mZxLwJ0zJJSojjN/07h+6gRXvgf5fA+q/hl0/DSZeG7tgiIhLV9DhyiWk/bNnNpKWbuaZvCk3r1Q7NQQvz4ZULYP08uPA5JWYiIlIpqpxJTBufkUn92vH8ul+IqmYFO32J2Zbv4OKX4LgRoTmuiIjEDFXOJGYt25jPZ8u3cl2/TjRKTgz+Afdth4m/gK3L4NJXlZiJiEiVqHImMWtcRiYN6yRw7Rmdgn+wvbnw8kjYkQOXvQ6pQ4J/TBERiUmqnElMWrRuJzN+yOWG/p1pkJQQ3IPt2QIvDYeda+BXbykxExGRalHlTGLS+IxMmtZN5Oo+KcE9UP5G36XMvVvh8ncgpW9wjyciIjFPlTOJOd+s3sGcrG3ceGYX6tYO4t8fO9fCi+fCvjy44n0lZiIi4glVziSmOOd4eOpKmtevzejeHYN3oB05MHEEFO2GKz+AtqcE71giIlKjqHImMeXLVduZt3oHNw/oQp3EuOAcZFsWvDjMtwLAVZOUmImIiKdUOZOY8WPVrHXDJEb16hCcg+T+4LvHDAdXT4aWxwXnOCIiUmOpciYxY1ZmHovW7eKWQakkJQSharZlmW9WptVSYiYiIkGj5ExignOO8RmZtGtch4tPae/9ATYtgYnnQXxtuGYKNO/m/TFERERQciYxIuP7rSzdkM9tg9NIjPf4xzp/A7w8AhLr+xKzpl287V9EROQASs4k6pWXO8ZlZNKpWV0uOLmt9wf49B4oLYarPoTGKd73LyIicgAlZxL1Plm2hR+27OH2wWnEx3n8I501DVZ8DGfeDU1CtHi6iIjUaErOJKqVlTvGT8skrUU9fnFSG287LymEKXdB0zQ4/VZv+xYRETkMJWcS1T7+dhPZuXu5Y0hX4mqZt51/MQF2robhD0F8ord9i4iIHIaSM4lapWXlPDI9i2Na1efc41t52/mOHJgzDrpfAJ0HeNu3iIjIESg5k6j13uKNrN62j7FDu1LLy6qZc/DJ7yEuEc7+p3f9ioiIBEDJmUSl4tJyHp2exYntGjL0uJbedv7DZMiaCgPvhQatve1bRETkKJScSVR6e+F6NuwsYMzQrph5WDUr3ud7dEaL7tDrN971KyIiEiCtrSlRp7CkjMdnZNOzQyMGdG3ubeefPwT56+GaTyFO/3uIiEjoqXImUeeNb9axOb+QO8/q5m3VLC8TvnwMTvoVdDzdu35FREQqQcmZRJWC4jKemLWK0zo1oU+Xpt517JzvmWaJyTD0Ae/6FRERqSQlZxJVXv16LXl7iryvmi1/D1bPhkF/gnoeXyoVERGpBCVnEjX2FZXy5OxV9EtrRq9OTbzruHA3fHoftO4B6dd616+IiEgV6I5niRovfbmGHfuKGTu0q7cdz3oQ9m6FUa9BrThv+xYREakkVc4kKuwuLOGZz3MYdEwLTu7Q2LuOty6HeU/BKVdDu1O861dERKSKlJxJVHhh7mryC0q8rZo5B5PvhDqNYPCfvetXRESkGnRZUyLerv3FPD9nNWd3b8nxbRt61/G3r8O6r2DE45Ds4T1sIiIi1aDKmUS8Z+fksLe4lDFeVs0KdsLUP0G7XtDjcu/6FRERqSZVziSibd9bxItfrGH4Ca05plUD7zqe8Xco2AHD34da+htFREQih34rSUR7+vMcCkvKuGOIh1WzTYth/vPQ6wZofaJ3/YqIiHhAyZlErNw9hbz81RrO79GW1Bb1vOm0vAwmjYV6LWDgfd70KSIi4iFd1pSI9d+Zqygpc9w2OM27ThdNhE2L4IJnIcnDyQUiIiIeUeVMItLm/AJem7eOi3q2I6VZXW863bcNpv0VUvrBCRd706eIiIjHlJxJRHp8RjYOx62DU73rdNr9ULwXhj0EXq7LKSIi4iElZxJx1u/Yz1sL1nPpqe1p1zjZm07XzYPFr8LpN0OLY7zpU0REJAiUnEnEeWxGFmbGLQM9utesrNS3EkCDttD/d970KSIiEiSaECARZc22fby7aCNXnt6RVg2TvOl0/nOw9Tu45GWo7dGsTxERkSBR5UwiyiPTs0iIM24a0MWbDvdsgZn/gC6D4dgR3vQpIiISRErOJGJk5+7hgyUbuer0FFrU96hqNvVPUFoIw/6fJgGIiEhUUHImEWP8tCySE+L4zZkeVc1Wfw7fvQV974CmHvUpIiISZErOJCKs2LybyUs3c03fTjSpm1j9DkuLYfJd0Kgj9Btb/f5ERERCRBMCJCKMz8ikflI8v+7X2ZsOv/4vbFsJl70JCXW86VNERCQEVDmTsPtuQz5Tv9/K9Wd0pmFyQvU7zN8As/8N3YZDt3Oq35+IiEgIKTmTsBuXsZJGyQlce0aKNx1+ei84B+c+6E1/IiIiIaTkTMJq4dqdzFyZxw39O1M/yYOqWfY0WPER9L8LGnWofn8iIiIhpuRMwmp8RiZN6yZy1ekp1e+spBCm3A1NU6HPrdXvT0REJAyUnEnYzMvZztzsbdw0oAt1a3swN+XLR2FHjm9h8/ja1e9PREQkDJScSVg453g4I5MW9WszunfH6ne4YzXMeRi6/xK6DKx+fyIiImGi5EzC4ovs7Xyzegc3D0wlKSGuep05B5/8HmrFw9n/9CZAERGRMFFyJiHnq5qtpE3DJEb1al/9DldOgazPYMA90KBN9fsTEREJIyVnEnKzVuaxeN0ubhmURu34albNivfDJ/dAi+PgtBu9CVBERCSMgpqcmdkYM1tuZsvM7HUzSzKzwWa2yMyWmNlcM0utYL8UMyvwt1liZk8FM04JHecc4zIyad+kDhent6t+h3Megvx1MPxhiPPgURwiIiJhFrTkzMzaArcB6c6544E4YBTwJHC5c64H8Brwx8N0sco518P/pZJIjJj6/Va+25jPbYPSSIir5o/ftiz44lE46TLo2MebAEVERMIs2Jc144E6ZhYPJAObAAc08L/f0L9NaoDycsf4jEw6N6vLL09uW73OnIMpd0FCMgx9wJsARUREIkDQFj53zm00s4eAdUABMNU5N9XMrgemmFkBsBvofZguOpnZYn+bPzrn5gQrVgmNKcs288OWPTwyqgfx1a2aLX8fcmb5nmlWr4Un8YmIiESCYF7WbAyMBDoBbYC6ZjYaGAMMc861A14ExlWw+2agg3PuZGAs8JqZNTi4kZndYGYLzGxBXl5esIYiHigrd0yYlkVai3qcd2I1Z1QW7YHP7oPWJ0H6td4EKCIiEiGCeVlzCLDaOZfnnCsB3gP6Aic55+b527wJHHKzkHOuyDm33f96IbAK6FpBu2ecc+nOufTmzZsHaxzigY++3Uh27l7GDO1KXC2rXmezHoQ9W2D4OKhVzdmeIiIiESaYydk6oLeZJZuZAYOB74GGZvZjojUUWHHwjmbW3Mzi/K87A2lAThBjlSAqKSvnkWlZHNu6Aed0b1W9zrYuh6+fhJ5XQrt0bwIUERGJIMG852yemb0DLAJKgcXAM8AG4F0zKwd2AtcCmNkIfDM7/wz0Bx4wsxKgHLjRObcjWLFKcL23aANrtu/n2SvTqVWdqplzMPkuSGoIQ/7iVXgiIiIRxZxz4Y7BE+np6W7BggXhDkMOUlxazsCHZtGsXiIf3NwXXxG1ipa8Dh/cCCMe81XOREREopSZLXTOVXgJSCsESFC9uWA9G3cVMGZo1+olZgW7IONP0K4X9BjtXYAiIiIRJmiXNUUKS8p4YkY2p3RszJldqzlhY8bfYf92GP0e1NLfFCIiErv0W06C5rV569iyu5A7q1s127QEFjwPp/4aWp/oXYAiIiIRSMmZBEVBcRn/nbWK3p2b0Ce1WdU7Ki+HyXdCcjMY9AfvAhQREYlQSs4kKF7+ag3b9hZx51ndqtfR4pdh4wI46+++WZoiIiIxTsmZeG5vUSlPzV5Fv7RmnJrSpOod7dsO0/4CHc+AEy/xLD4REZFIpuRMPPfSF6vZub+k+lWzaff7lmoa/hBU5541ERGRKKLkTDyVX1DCM5/nMPiYFvRo36jqHa3/Bha/Ar1vghbHehegiIhIhFNyJp56fu5qdheWMmboIUuhBq6sFCaPhfpt4Mx7vAtOREQkCug5Z+KZnfuKeWHuas7p3orj21bj5v0Fz8OW7+DiiVC7nncBioiIRAFVzsQzz8zJYV9xNatme7b6HjjbZRAcN9K74ERERKKEkjPxxLa9Rbz0xRrOO7EN3VrVr3pHGX+C0kIYpkkAIiJSMyk5E088NWsVRaVl3DEkreqdrJkLS9+EvrdD0y7eBSciIhJFlJxJtW3dXcgrX6/l/JPb0qV5Fe8RKyvxrQTQqAOcMdbbAEVERKKIJgRItf13Zjal5Y7bB1ejavb1fyHvB7jsDUhM9i44ERGRKKPKmVTLxl0FvP7Nei4+pR0dm9atWif5G2HWv6HrudDtXG8DFBERiTJKzqRaHp+RjcNxy6DUqnfy2b3gyuHcB70LTEREJEopOZMqW7d9P28vWM+oUzvQrnEVL0VmT4fvP4T+d0LjFE/jExERiUZKzqTKHp2RRa1axs0Dq1g1Ky2CKXdD01Toc5u3wYmIiEQpTQiQKsnJ28t7izZwdZ9OtGqYVLVOvngUdqyCK96H+NreBigiIhKlVDmTKnlkeha14+O4aUAVn0e2cw3MeQiOO9+3GoCIiIgASs6kCjK37uGjbzdxZZ+ONK9fxYrXJ/eAxcHZ//Q2OBERkSin5EwqbcK0TJIT4vhN/ypWzX6YApmfwIB7oGFbb4MTERGJckrOpFKWb8pnyndbuPaMTjSpm1j5Dor3wye/h+bHQu+bvA9QREQkymlCgFTK+Iws6ifFc/0ZnavWwZyHIX8dXD0Z4hK8DU5ERCQGqHImAft2/S6mrdjKr/t1pmFyFRKrbdnw5aNw4ihIOcP7AEVERGKAkjMJ2LiMTBolJ3BN35TK7+wcTLkL4uvAWX/zPDYREZFYoeRMArJw7Q5mZ+bxm/5dqJ9UharZ9x9AzkwY9Eeo18L7AEVERGKEkjMJyMNTM2lWL5Gr+nSs/M5Fe+DT+6DViXDqdd4HJyIiEkOUnMlRfbVqO1+u2s6NZ3YhObEKc0hm/xv2bILh46BWnPcBioiIxBAlZ3JEzjnGZaykZYPajO5dhapZ7gr4+knoeSW0P9X7AEVERGKMkjM5ojlZ25i/Zic3D0wlKaGSVS/nYPKdULs+DP5LUOITERGJNXrOmRyWc46HMzJp0zCJS09tX/kOlr4Ja7+AXzwCdZt6H6CIiEgMUuVMDmvGD7l8u34Xtw5Oo3Z8JatmBbtg6h+hbTqcfGVwAhQREYlBqpxJhXz3mmXSoUkyF53SrvIdzPwH7N8Ol78DtfQ3gIiISKD0W1Mq9NnyLSzftJvbBqeREFfJH5NNS2D+c3Dq9dCmR3ACFBERiVFKzuQQ5eWO8RlZdG5Wl/N7tKnszr5JAMnNYOAfghOgiIhIDFNyJoeY9N1mVm7dw+1D0oivbNVs8SuwcYFviaY6jYIToIiISAxTciY/U1pWzoRpmXRtWY9fnFjJqtn+HTDtL9CxL5x4aVDiExERiXVKzuRnPlyyiZy8fYwZ0pVataxyO0/7CxTmw7CHwCq5r4iIiABKzuQAJWXlPDI9i+NaN+Ds7q0qt/P6+bBoIvS+CVoeF5wARUREagAlZ/J/3l24gXU79jN2aCWrZuVlMHks1G8NA+4JXoAiIiI1gJ5zJgAUlZbx2IxsTmrfiMHHtqjczvOfhy1L4aIXfUs1iYiISJWpciYAvDV/PRt3FTB2aFesMveL7c2FGX+HzgOh+y+DF6CIiEgNoeRMKCwp4/GZ2aR3bEz/tGaV23nqn6C0QJMAREREPKLkTPjfvHVs3V3E2LMqWTVb8wUsfQP63AbNUoMXoIiISA2i5KyG219cypOzsjm9c1P6dKlE1aysxLcSQMMO0O/O4AUoIiJSw2hCQA338ldr2ba3mKdGd63cjvOegrwVMOp1SEwOTnAiIiI1kCpnNdjeolKenr2K/l2bk57SJPAd8zfCrAeh6zlwzLDgBSgiIlIDKTmrwV6cu5qd+0sYO7SSVbPP7oPyUjj338EJTEREpAZTclZD5ReU8OycHIYc24Ie7SuxQHn2dPj+A999Zo1TghafiIhITaXkrIZ6fk4OuwtLGVOZqllpEUy5G5p09s3QFBEREc9pQkANtHNfMS98sYZzj29F9zYNA9/xy0dhxyoY/R4kJAUvQBERkRpMlbMa6OnPc9hXXMmq2c618PnDcNxISB0cvOBERERqOCVnNUzeniImfrmGX5zYhq4tK7EO5qf3gNWCs/8VvOBEREREyVlN89TsVRSVlnH7kLTAd1r5KaycAgN+Dw3bBi84ERERUXJWk2zdXcirX6/llye3o0vzeoHtVFIAn/wOmh8DvX8b3ABFREREEwJqkidmZlNW7rh9cCWqZnPGwa61cNUkiEsIXnAiIiICqHJWY2zcVcAb36zn4vR2dGga4HJL21fBFxPghEugU7/gBigiIiKAkrMa4/EZWQDcMijAqplzMOUuiE+Cs/4exMhERETkQErOaoB12/fz9oINjOrVnraN6gS20/cfwqoZMPAPUL9lcAMUERGR/6PkrAZ4ZHoWcbWMmwemBrZD0V749F5odQKcen1wgxMREZGfUXIW41bl7eX9xRsY3bsjLRsE+FT/2f+GPZtg+DiI05wRERGRUFJyFuMemZZF7fg4bhrQJbAdclfA1/+Fk6+A9r2CG5yIiIgcQslZDFu5ZQ8fL93EVX1SaFav9tF3cA4m3wW168OQvwY/QBERETmErlnFsAnTMqmbGM9v+ncObIfv3oa1c+G8CVC3aXCDExERkQqpchajlm/K55NlW7i2bwqN6yYefYeCXfDZH6DtKdDzquAHKCIiIhVS5SxGjc/IpEFSPNf1C7BqNvOfsC8PLn8LailnFxERCRf9Fo5BS9bvYtqKXH7drzMN6wSw5NLmb2H+s3DqddDm5OAHKCIiIoel5CwGjcvIpHFyAtec0enojcvLYfKdkNwUBv0x+MGJiIjIESk5izEL1uzg88w8fnNmF+rVDuCq9ZJXYcN8GPo3qNM4+AGKiIjIESk5izEPT82kWb1Erjy949Eb798BGfdDhz5w0qjgByciIiJHpeQshny5ahtf5WznpgGpJCcGUDWb/lcozIfhD4FZ8AMUERGRo1JyFiOcc4ybmknLBrW5/LQOR99hw0JYOBF63wQtuwc/QBEREQmIkrMY8XnWNhas3cktA1NJSog7cuPyMpg8Buq3ggH3hCZAERERCYiecxYDfFWzlbRtVIdLTm1/9B0WvOB7fMZFL/iWahIREZGIocpZDJi+IpdvN+Rz66BUascfpWq2Nxem/w06D4DuF4QiPBEREakEJWdRrrzcMS4jkw5NkrnwlHZH3yHjz1CyH4ZpEoCIiEgkCmpyZmZjzGy5mS0zs9fNLMnMBpvZIjNbYmZzzSz1MPvea2bZZrbSzM4OZpzR7LPlW/h+825uH5xGQtxRTufaL+Hb16HvbdAsLTQBioiISKUELTkzs7bAbUC6c+54IA4YBTwJXO6c6wG8BhzyWHozO87ftjtwDvBfMzvK9bqap6zcMX5aJp2b1+X8k9sepXGJbyWAhh2g312hCVBEREQqLdiXNeOBOmYWDyQDmwAHNPC/39C/7WAjgTecc0XOudVANtAryLFGnUlLN5G5dS93DOlKXK2jXKKc9zTkfg/nPgiJyaEJUERERCotaLM1nXMbzewhYB1QAEx1zk01s+uBKWZWAOwGelewe1vg6wO+3+DfJn6lZeU8Mi2Lbi3rc94JrY/cePcmmPUvSDsbug0LTYAiIiJSJcG8rNkYXwWsE9AGqGtmo4ExwDDnXDvgRWBcNY5xg5ktMLMFeXl5XoQdNT5YsomcbfsYM7QrtY5WNfvsD1BeCuf+W5MAREREIlwwL2sOAVY75/KccyXAe0Bf4CTn3Dx/mzeBPhXsuxE48IFd7fzbfsY594xzLt05l968eXNvo49gJWXlPDo9i+PbNuDs7i2P3HjVTFj+HpwxFpp0Ck2AIiIiUmXBTM7WAb3NLNnMDBgMfA80NLOu/jZDgRUV7PsRMMrMaptZJyAN+CaIsUaVdxZuYN2O/Ywd2hU7UiWstAim3AWNO0Hf20MXoIiIiFRZMO85m2dm7wCLgFJgMfAMvvvH3jWzcmAncC2AmY3AN7Pzz8655Wb2Fr5krhS42TlXFqxYo0lRaRmPTc+iR/tGDOzW4siNv3wMtmfD5e9CQlJoAhQREZFqMedcuGPwRHp6uluwYEG4wwi6l79aw58/XM4r1/WiX9oRLuXuXAtPnAZpQ+DSV0MWn4iIiBydmS10zqVX9J5WCIgihSVlPD4jm14pTTgjtdmRG396L1gtOOfB0AQnIiIinlByFkVe/XotuXuKGHvWUe41y/wMVk6GM38HDQNY0klEREQihpKzKLG/uJSnZq+ib2pTendueviGJQUw5W5o1g16/zZ0AYqIiIgngjYhQLw18cu1bNtbzNNDux254dzxsGstXDUJ4hNDE5yIiIh4RpWzKLCnsISnP1/FgG7NOaVj48M33L4K5k6AEy6GTv1CF6CIiIh4RslZFHjxizXs2l/C2KFdD9/IOd/lzPjacNbfQxeciIiIeErJWYTL31/Cs3NyGHpcS05s1+jwDVd8BKumw8D7oH6r0AUoIiIinlJyFuGem5vDnsLSI1fNivb6Hp3R8gQ49dehC05EREQ8pwkBEWzHvmJemLua4Se05tjWDQ7f8PP/wO6NcNGLEKdTKiIiEs1UOYtgT3++iv0lZdwxJO3wjXJ/gK+egJNHQ4fTQheciIiIBIWSswiVt6eIl79cy8iT2pDWsn7FjZzzLWyeWA+G/DW0AYqIiEhQ6BpYhHpy1iqKy8q5fcgR7jX77h1YMwfOGw91j7Kck4iIiEQFVc4i0JaSFOJHAAARUklEQVT8Ql6dt5YLTm5Lp2Z1K25UmA9T/wBtekLPq0IboIiIiASNKmcR6ImZ2ZSXO24bfIR7zWb+C/bmwmVvQK240AUnIiIiQaXKWYTZsHM/b8xfxyWntqd9k+SKG21eCt88DenXQtueoQ1QREREgkrJWYR5fEY2hnHLwNSKG5SXw+Q7oU4TGPyn0AYnIiIiQafLmhFk7fZ9vL1wA1f07kibRnUqbrTkf7DhGxj5X6hzhHU2RUREJCqpchZBHpmeRXwt47cDulTcYP8OmHY/tO8NJ10W2uBEREQkJJScRYjs3L18sHgjV57ekRYNkipuNP0BKNgFwx+GWjp1IiIisUi/4SPEI9OzSEqI48YzD1M127gQFr4Ep90IrY4PaWwiIiISOkrOIsDKLXuYtHQTV/dJoWm92oc2KC+DSWOhXksYcE/oAxQREZGQ0YSACDA+I5N6ifHc0L9zxQ0Wvgibl8CFz0PSERZAFxERkainylmYLduYz6fLt3DtGZ1olJx4aIO9eb57zTr1h+MvDH2AIiIiElJKzsJsfEYmDeskcF2/ThU3yPgzFO+HYQ+DWWiDExERkZBTchZGi9ftZPoPudzQvzMNkhIObbD2K/j2NehzCzQ/wgLoIiIiEjOUnIXRuIxMmtRN5Oo+KYe+WVbqWwmgYXvof3fIYxMREZHwUHIWJvPX7GBO1jZuPLMzdWtXMC/jm6chdzmc8yAk1g19gCIiIhIWSs7C5OGpK2levzZX9E459M3dm2HmvyDtLDhmeMhjExERkfBRchYGX2Zv4+ucHfx2QBfqJMYd2mDqH6CsGM79tyYBiIiI1DBKzkLMOcfDGZm0bpjEZb06HNogZxYsexf6jYUmh3numYiIiMQsJWchNjszj4Vrd3LzwFSSEg6qmpUWw5S7oXEn6HtHeAIUERGRsNIKASHknGNcRibtGtfhkvT2hzb46nHYlgmXvwMJh1n8XERERGKaKmchNG1FLks35HPboDQS4w/6p9+1Dmb/B445D9KGhidAERERCTslZyFSXu6rmqU0TeaCnm0PbfDpvb6b/895MPTBiYiISMRQchYiny7fworNu7l9SBrxcQf9s2dOhR8m+R4226iCy50iIiJSYyg5C4Gycsf4jExSW9RjxEkHVc1KCuCTu6FZVzj9lvAEKCIiIhFDyVkITFq6iazcvdwxJI24Wgc9t2zuBNi5BoY/DPGJYYlPREREIoeSsyArLStnwrQsjmlVn2HHt/75mztyYO54OP4i6NQ/PAGKiIhIRFFyFmTvL97I6m37GDO0K7UOrJo5B1N+B3GJcPY/whegiIiIRBQlZ0FUUlbOozOyOKFtQ846ruXP3/xhEmRnwMD7oH6r8AQoIiIiEUfJWRC9vWAD63cUMHZoV+zANTKL98En90DL46HXDeELUERERCKOVggIkqLSMh6fkcXJHRoxoFvzn785+z+wewNc9DzE6RSIiIjIT1Q5C5I3vlnPpvxC7hza7edVs7yVvmWaelwOHXqHL0ARERGJSErOgqCwpIwnZmbTq1MT+qY2/ekN52DynZBYF4b8NXwBioiISMRSchYEr369ltw9Rdx58L1my96FNXNg8P1Qr/nhOxAREZEaS8mZx/YVlfLkrFWckdqM0zofUDUr3A2f/QHanAynXB22+ERERCSy6W50j038ag3b9xUz9qyuP39j1r9g71a47HWoFReW2ERERCTyqXLmoT2FJTzzeQ4DuzWnZ4fGP72xZRnMexrSr4G2PcMXoIiIiEQ8JWceemHuGnbtL2Hs0G4/bSwvh8ljoU4jGPSn8AUnIiIiUUGXNT2Sv7+E5+bmcNZxLTmhXcOf3vj2NVg/D0Y+AclNwhegiIiIRAVVzjzy7Jwc9hSWMmboAfea7d8BGX+G9qfBSb8KX3AiIiISNZSceWDHvmJe/GI1w09szbGtG/z0xoy/QcFOGP4w1NI/tYiIiBydMgYPPD17FQUlZYwZkvbTxo0LYcGL0Os30OqE8AUnIiIiUUXJWTXl7ilk4ldrGNmjLakt6vs2lpf5VgKo1xIG3hfW+ERERCS6aEJANT05axUlZY7bBx9QNVv4EmxaDBc+D0kNDruviIiIyMFUOauGzfkF/G/eOi7s2ZaUZnV9G/dtg+kPQEo/OP7C8AYoIiIiUUfJWTU8MTMb5xy3DjqgapZxPxTv800COHBdTREREZEAKDmrog079/Pm/PVckt6e9k2SfRvXfQ1LXoXTb4bm3Y7cgYiIiEgFlJxV0WPTszEzbhmU6ttQVuqbBNCgHZz5u/AGJyIiIlFLEwKqYM22fbyzaANX9O5I64Z1fBu/eQa2LoNLXoHEuuENUERERKKWKmdV8Oj0LBLijN8O7OLbsHszzPwnpA6BY38R3uBEREQkqik5q6Ts3L18sGQjV56eQov6Sb6NU/8IZcVw7n80CUBERESqRclZJU2YlklSQhy/6d/ZtyFnNix7B84YA027hDc4ERERiXpKzirhhy27mbR0M9f0TaFpvdpQWgxT7oLGKXDGHeEOT0RERGKAJgRUwviMTOrXjufX/fxVs6+fgG2Z8Ku3IaFOeIMTERGRmKDKWYD2FpWSuXUv1/XrRKPkRNi1Hmb/B445D7qeFe7wREREJEaochagerXjyRjTn9Jy59vw2b3gHJzzr/AGJiIiIjFFlbNKiI+rRVJCHGRlwIqP4cy7oVGHcIclIiIiMUTJWWWVFMKUu6FpGpx+a7ijERERkRijy5qV9cUE2LkarvwQ4hPDHY2IiIjEGFXOKmNHDswZB90vgM4Dwh2NiIiIxCAlZ4FyDj75PcQlwtn/DHc0IiIiEqOUnAWqaA8U7IKB90KD1uGORkRERGKU7jkLVFIDuPYzwIU7EhEREYlhSs4qo5YKjSIiIhJcyjZEREREIoiSMxEREZEIouRMREREJIIE9Z4zMxsDXI/vLvrvgGuADKC+v0kL4Bvn3PkV7Fvm3wdgnXNuRDBjFREREYkEQUvOzKwtcBtwnHOuwMzeAkY55/od0OZd4MPDdFHgnOsRrPhEREREIlGwL2vGA3XMLB5IBjb9+IaZNQAGAR8EOQYRERGRqBG05Mw5txF4CFgHbAbynXNTD2hyPjDdObf7MF0kmdkCM/vazA657CkiIiISi4KWnJlZY2Ak0AloA9Q1s9EHNLkMeP0IXXR0zqUDvwImmFmXCo5xgz+BW5CXl+dh9CIiIiLhEczLmkOA1c65POdcCfAe0AfAzJoBvYDJh9vZX3nDOZcDzAJOrqDNM865dOdcevPmzb0fgYiIiEiIBTM5Wwf0NrNkMzNgMLDC/95FwCTnXGFFO5pZYzOr7X/dDOgLfB/EWEVEREQiQjDvOZsHvAMswvdIjFrAM/63R3HQJU0zSzez5/zfHgssMLNvgZnAg845JWciIiIS88y52FjIOz093S1YsCDcYYiIiIgclZkt9N9bfwitECAiIiISQZSciYiIiEQQJWciIiIiEUTJmYiIiEgEUXImIiIiEkGUnImIiIhEECVnIiIiIhFEyZmIiIhIBFFyJiIiIhJBlJyJiIiIRJCYWb7JzPKAtSE4VDNgWwiOE4lq8tihZo9fY6+5avL4a/LYoWaPPxRj7+ica17RGzGTnIWKmS043FpYsa4mjx1q9vg19po5dqjZ46/JY4eaPf5wj12XNUVEREQiiJIzERERkQii5Kzyngl3AGFUk8cONXv8GnvNVZPHX5PHDjV7/GEdu+45ExEREYkgqpyJiIiIRBAlZ4dhZueY2Uozyzazeyp4v7aZvel/f56ZpYQ+yuAIYOxXm1memS3xf10fjjiDwcxeMLNcM1t2mPfNzB71/9ssNbOeoY4xWAIY+wAzyz/gvP851DEGi5m1N7OZZva9mS03s9sraBPL5z6Q8cfk+TezJDP7xsy+9Y/9rxW0icnP+wDHHrOf9z8yszgzW2xmkyp4Lzzn3jmnr4O+gDhgFdAZSAS+BY47qM1vgaf8r0cBb4Y77hCO/Wrg8XDHGqTx9wd6AssO8/4w4BPAgN7AvHDHHMKxDwAmhTvOII29NdDT/7o+kFnBz30sn/tAxh+T599/Puv5XycA84DeB7WJ1c/7QMYes5/3B4xxLPBaRT/f4Tr3qpxVrBeQ7ZzLcc4VA28AIw9qMxKY6H/9DjDYzCyEMQZLIGOPWc65z4EdR2gyEnjZ+XwNNDKz1qGJLrgCGHvMcs5tds4t8r/eA6wA2h7ULJbPfSDjj0n+87nX/22C/+vgm7Fj8vM+wLHHNDNrBwwHnjtMk7CceyVnFWsLrD/g+w0c+kH1f22cc6VAPtA0JNEFVyBjB7jQf2nnHTNrH5rQIkKg/z6x6nT/JZBPzKx7uIMJBv9li5PxVREOVCPO/RHGDzF6/v2XtZYAuUCGc+6w5z7GPu8DGTvE9uf9BOB3QPlh3g/LuVdyJlXxMZDinDsRyOCnvyokti3Ct9zIScBjwAdhjsdzZlYPeBe4wzm3O9zxhNpRxh+z5985V+ac6wG0A3qZ2fHhjilUAhh7zH7em9l5QK5zbmG4YzmYkrOKbQQO/OugnX9bhW3MLB5oCGwPSXTBddSxO+e2O+eK/N8+B5wSotgiQSA/GzHJObf7x0sgzrkpQIKZNQtzWJ4xswR8icn/nHPvVdAkps/90cYf6+cfwDm3C5gJnHPQW7H6ef9/Djf2GP+87wuMMLM1+G7hGWRmrx7UJiznXslZxeYDaWbWycwS8d0E+NFBbT4CrvK/vgiY4fx3DEa5o479oPtsRuC7P6Wm+Ai40j9zrzeQ75zbHO6gQsHMWv14r4WZ9cL3+RETv6D843oeWOGcG3eYZjF77gMZf6yefzNrbmaN/K/rAEOBHw5qFpOf94GMPZY/751z9zrn2jnnUvD9rpvhnBt9ULOwnPv4YB8gGjnnSs3sFuAzfLMXX3DOLTezB4AFzrmP8H2QvWJm2fhuoh4Vvoi9E+DYbzOzEUApvrFfHbaAPWZmr+ObldbMzDYA9+O7SRbn3FPAFHyz9rKB/cA14YnUewGM/SLgJjMrBQqAUbHwC8qvL3AF8J3//huA+4AOEPvnnsDGH6vnvzUw0czi8CWcbznnJtWEz3sCG3vMft4fTiSce60QICIiIhJBdFlTREREJIIoORMRERGJIErORERERCKIkjMRERGRCKLkTERERCSCKDkTkZhjZmVmtuSAr3s87DvFzJZVon1dM5vmfz3X/yBLEZHD0oeEiMSiAv+SNJHgdOArM2sM7POvzycicliqnIlIjWFma8zsP2b2nZl9Y2ap/u0pZjbDv7jzdDPr4N/e0sze9y/2/a2Z9fF3FWdmz5rZcjOb6n+6+sHH6uJ/oOurwK+AhcBJ/kpeixANWUSikJIzEYlFdQ66rHnpAe/lO+dOAB4HJvi3PQZM9C/u/D/gUf/2R4HZ/sW+ewLL/dvTgCecc92BXcCFBwfgnFvlr94tBHrhWzD6OudcD+dcrqejFZGYohUCRCTmmNle51y9CravAQY553L8C31vcc41NbNtQGvnXIl/+2bnXDMzywPaHbDwM2aWAmQ459L83/8eSHDO/f0wscx3zp1qZu8CtzvnNng8XBGJMaqciUhN4w7zujKKDnhdRgX375rZU/6JA2n+y5vnAJPMbEwVjykiNYSSMxGpaS494L9f+V9/yU8LGl8OzPG/ng7cBGBmcWbWMNCDOOduBP4K/A04H5jsv6Q5vnrhi0is02xNEYlFdfzVqh996pz78XEajc1sKb7q12X+bbcCL5rZ3UAecI1/++3AM2Z2Hb4K2U3A5krEcSbwMtAPmF2lkYhIjaN7zkSkxvDfc5bunNsW7lhERA5HlzVFREREIogqZyIiIiIRRJUzERERkQii5ExEREQkgig5ExEREYkgSs5EREREIoiSMxEREZEIouRMREREJIL8f7AB8b/s1LfQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, 10 * pltsize))\n",
    "plt.plot(range(epochs), train_acc_all,label = 'Training Acc.' )\n",
    "plt.plot(range(epochs), val_acc_all, label = 'Validation Acc.' )\n",
    "plt.xlabel('Epoch #')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZWElEQVR4nO2deXgV1fnHPwcBJWIVQRGxwE8ribhhUEr9qVStC1pqEKwWWSy4oTQVELWCrQhItYvyuFCwYRGpFspapFQrAmpbF1qXiogWl6CiCCKiooKnfwzvmSG5SW6SO8udvJ/n8UnMvXfmPcydM+/5nncx1loURVEURVHSTKO4DVAURVEURQkbdXgURVEURUk96vAoiqIoipJ61OFRFEVRFCX1qMOjKIqiKErqUYdHURRFUZTUow6PoiiKoiipJ3KHxxgz1BjznDHmC2PM9KjPHxXGmP2NMfONMZ8aY94yxvSN26ZcYYzZ0xhTtmtcnxhjnjfG9IjbrlyT5msoGGOOMMYsM8Z8bIx53RjTK26bcokxZluF/3YaY+6K265c0kDGmPrnRgOZbzoYY5YYYz4yxmwwxtxtjGkc1fnjUHjeBcYBU2M4d5TcA3wJtAYuBiYZY46M16Sc0RgoB7oD+wKjgdnGmA4x2hQGab6G7JpoFgKLgf2By4EHjDEdYzUsh1hrm8t/wEHA58CcmM3KKQ1hjDSM50aq55td3At8ALQBOuM9Q66K6uSROzzW2nnW2gXApqjPHRXGmL2B3sBN1tpt1tongUVA/3gtyw3W2k+ttTdba9+01n5trV0MvAF0idu2XJH2a7iLIuBg4A5r7U5r7TLgKdI1xiC98SbbJ+I2JERSOca0PzcayHwD8H/AbGvtdmvtBmApEJlTpzE84dAR2GGtXRv42wtEeGGjxBjTGm/ML8dtSw5pUNcwgAGOituIkBgI3G/T3U+nIYwxjTSU+eZO4CJjTIExpi3QA8/piQR1eMKhObC1wt8+BvaJwZZQMcY0AWYBM6y1a+K2J4c0hGv4Kp4aMNIY08QYcyaexFwQr1m5xxjTHm9sM+K2JSwawhhTTEOYbwBW4jlxW4H1wHPAgqhOrg5POGwDvlHhb98APonBltAwxjQCZuLtOw+N2Zxck/praK39CigBzgU2ACOA2XgTUdroDzxprX0jbkNCpCGMMa2kfr7Z9bxYCswD9gZaAS2A26KyQR2ecFgLNDbGHB7427GkaMvHGGOAMrwAu967Hp5pIvXXEMBa+6K1tru1tqW19izgUOCZuO0KgQGkX/loCGNMKw1hvtkfaAfcba39wlq7CZgGnBOVAXGkpTc2xuwF7AHsYYzZK8q0tCiw1n6K58XeYozZ2xjz/8B5eGpIWpgEHAH0tNZ+HrcxuaaBXEOMMcfsugcLjDHX4mVPTI/ZrJxijDkRaEv6MpccaR9j2p8bDWG+sdZ+iJfcMmTX9dwPL+bsxahsiEPhGY2XNnkD0G/X76NjsCNsrgKa4cVIPAgMsdamwlvfFStwBV5a4YZA/Y+LYzYt16T2GgboD7yHN8bTgTOstV/Ea1LOGQjMs9amZnsgA2kfY0N4bjSE+eZ84GxgI/A68BUwLKqTGw3mVxRFURQl7WgMj6IoiqIoqUcdHkVRFEVRUo86PIqiKIqipB51eBRFURRFST3q8CiKoiiKknqqrWNgjMnrFC5rranpPTrG5FPTGNM+PtAx5gM6xvSPD3SM+UBVY1SFR1EURVGU1KMOj6IoiqIoqUcdHkVRFEVRUo86PEpWDB48mMGDB7N8+XKWL1/OkCFDsNZirWXt2rWsXbuWwYMHx22moiiKomREHR5FURRFUVJPtb200hqpHSSsMZ500kkADBw4EIBBgwa51wYMGADArFmz6n2esMd4zTXXAHDbbbcBsMceewSPKzYAsHPnTl555RUArr76agCeeuqpup7aEWZmyIQJEwBYtmwZAI8++igAL730EkcddRQATz/9NACrV69m7ty5ADz88MN1PWUlGnLWRBAdY/JJUpbW2LFjAbjxxhsBOPjggwF4//3363zMpF3DLl26ZPz7hx9+yFtvvVWnYyZtjGGgWVqKoiiKojRYEqPwfPOb3wTgzTffBOC1114DoFu3bmzZsqVOx4zKk23c2CtndMghhwBQVFTEfffdB/irjq+//tq9/+OPPwbg4osvBuCvf/1rnc8d9hhvvfVWAK677rpMxxUbKr32zjvvAHDeeecB8Pzzz9fVhNBWlV26dOHvf/87AF988QUAP/rRjwDo06ePe9/5558PwD777MOnn34K+ArP6NGjAXj99dfrYgLQsFdcQXSMyScpCk+7du2cetymTRvAn2s/+OCDOh83CdewqKgIgJkzZ1JcXCx2ybkB2LhxIz//+c8BmDdvHuCpPtmQhDGGTVVjrLbwYJSce+65gH9hN27cCPgPoiQjjo44aTWx7777AnD55ZcD9XN4koDYf9ZZZ7m/tW3bFvC39Orj8IRFp06daNKkCYD72a5dOwB+/OMfu/fJttdPf/pThgwZAsAPf/hDALp27QrAqaeeWmeJOWl861vfAuDaa68FfOfvsccec9dz+/bt8RinKMCll17qHJ20II7O6tWrAe9ZKA7OmjVrAPjXv/4FQElJCZMnTwbgzDPPBHZfpMWJhD3IHHrGGWcA/pxZkbPPPhuAxx9/HIAvv/yy0rEkpKJFixYsWbIEwIUX1Abd0lIURVEUJfUkZktr0aJFAJxzzjmAv5Uisl1dCFu6++53vwv4QXOnnnoqAFu2bOGFF14A/K2P22+/vcrjHH300c6Dry1hj3G//fYD/K3G5s2bu9ceeOABAO644w7Au4ai7Ag7duwAYNiwYUyaNKlONoQlo2/fvp3NmzcDuJT6v/3tbwB89dVXmc7jVhy//e1vAbjkkksA2Lx5M9/73veA2m9vJUFi3nPPPQHv/rvzzjsBf5s5OEcceuihALVWs6Ia4xVXXAHgrsVpp51GixYtqjsngLtfBwwYwIsvvlinc0d9HQ866CAATj75ZGe/rPY7duxIt27dADjhhBN2+9y2bdtcUoV8LluSsqU1Y8YMFxIg5POW1gEHHMAzzzwD+Cqztdapy/Lzs88+Azw16OWXX3bvA7jggguYP39+jecKe4zXX3894NucxbkA+POf/wzAD37wA8BTc372s58BvtoMuMSY73znOwBs3bq10jE1aFlRFEVRlAZLImJ4Jk2a5GJ4Vq1aBcC4cePiNCkrHnvsMWD3gGSA0tJSHnzwQcBXAKpj0aJF9OzZE4BXX301t0bWEwkY/9WvfgX4ataee+5JeXk54K8SL7zwQhYvXgz4ypAEdA8dOpQZM2YA/iolLpo1awZAo0aN3Gr+L3/5S42fs9Y6xaq0tBSA//73v4CnSPbr1w+Am2++Odcmh4aoOBL4WFxc7BQqCeCWf6+ysjJ3X/bv3z9qU7NC4gREbd2xY4cbj6wEJUYC4LjjjgPgmGOOAeCUU06ps8ITFVLO4pRTTgF8ZaMqKs5PBQUFTqmrrcKjhEO/fv2csiOKx/z58921rjhnrlmzxil8EtMyb948Fw8TdVyozPPFxcVceeWVGd9TXl5OQUEBAB999BEAhx12mBvv6aefDnjxWQCjRo2iffv2lY4juwhNmzatvZ21/kQInHvuuTRq5IlNsqUQDFxKEvIlmz17trN527ZtgF9f5+mnn2blypWAJzdD5UknyOGHH+4ygbKVAaNm/PjxgL+11bZt20rbdP/85z/59a9/DVR2WAsLC+nduzfgZR/EidjRuHFj52DXlYkTJwKevDpixAjAr9uTjRMVJ23atOE3v/kNgMsGmTx5MsOHDwf8wGQJpgQ/MDKpDo843OLw9OnTx22XZ+KPf/wjAEceeSRQ+626KJGH2QUXXAD4AZ3jxo1j2rRpVX7urrvuAvxwgW3btmWdYJE0ZMzNmjVzD0oJ5JUMynyiV69eANxwww1ua0rCOSZMmFDt4lASe4YNGwbAihUr3L0atcNzwAEHAN4zoCJLly4FoG/fvu75KSEcvXr1cqEOBx54IABTpkyp9lz33nsvkH1WWhDd0lIURVEUJfXEqvBIUF2LFi2ctyreW1KZOnUq4K3oRbWRIKqFCxcC3qqyYkDVK6+84oLMRAkKIltF7777LoDb/kkaNVWHljHKamvvvfd2r1122WWA7/HLNY+aVq1aud9zpTbdeOONHH/88YAvySZV4ZEU/Ntvv92pXbfccgsAY8aMqfR+qZptjKlXvaEokLpPwoYNGzK+r1OnTgB8//vfB/zv7SOPPBKidfVDtlNldSxb6pMmTco4zgsvvBDwlR1h6tSp/Oc//wnT1NA44ogjAK82ligiy5cvB/JL4ZHt7+A2/0033QTUXuUXpePVV191qlcSkHtKgsu3bNlSqabeqlWrsrpusgU2fPhw5syZU2ebVOFRFEVRFCX1xKrwSKpZs2bN3N7f+vXr4zSpRgoLC93vUkxPVlKCFEkCX81ZuHAhv/jFL6o87l577QWQMUgrn5DUQqlALCnrACeeeCLgp/PXx1OvDxIcB9XHVtWGdevWMXv2bABGjhwJeCpCMEA2KUjBy759+7p4o0zKjhTUlHgda627dvmOrDolIFvifJJc6FTiGyXAuiZEvarIqFGjcmZT1ATLlEjsjqiT+YDE2Ei6dTBup67xm6L4WWspKSkBcKUl4kRUGfkZpEOHDoCnqEoAfcUSOVu3buWhhx4C/Dg0UY3qiio8iqIoiqKknlgUHukAG9xbltS6pCJdw4OF9aTIl+xBiprRtm1b53VLXE+2SLpzWVmZ60eVj6xbt67K1yReKS6FRzJzpBRCrnj77bcBP5NkxIgRrqBhEpCVlGTSrV69mquuuqrK98uKc5999gG8DL24SwrUxIoVKwA/Zbt79+6uoFuQikqVlFhIC+PHj3dlBUTFlH54+dwWRGLOrLVs2rQJgE8++SROk7Kmffv27vspWU2S/ZqL7FxjjIsnTSqi7Egcp7SxCfLss88CnupV2+dnTcTi8EglRqnuumLFit22PpJGq1atnKwvD7OysjJ3w8lFHDp0qPuM1K2pLdJnS+oa5DuZguiSFFiXS8SRki0tqXETN3KfyT0m9StGjx6dsceZ9CiS9GehtLSUzz//PExT641cA3m4BwPUhR49etC5c2fA38KqLnU9nxCn9tJLL3X3mTTITfIcWxOSEBBEHoxJR7ax5s6dS8uWLYHcOjpy/MLCQtdfKwnINZMSCGVlZa4uXceOHSu9X4LwxanNVEG5vuiWlqIoiqIoqSdyGaFly5Z8+9vfBvwgpbFjx0ZtRq2YM2eOWxFKCmSmapJSiHD58uXcf//9u73WoUOHKvuCde/e3QW8tm7dOldm1xpRJLINnJZU2OpSlTP1aquuf1sUSJ8sa63r2yLbPPVBgvOqSoWOC1lVSjVv6UqcSS5u0qSJKygpypDI5CLHJxkJEpfrmqlHXbdu3Vyw8rJlywC/HES+IqqdFK1r1aqVU6/yWdkBL7Bcgupljn3ttddciZCkIqEbEq5RUFBQ59TzTEjJD+ka/tlnn0VecLA6JBHnsMMOA/zvZkWk6K7YHua2uSo8iqIoiqKknsgVnkGDBrnAX0lBT3rvGmutC/z7/e9/X+l1WdFLKqCUqQ9y/fXXV0qB/uUvfwl46e1vvPEG4AezRYUEpA4ePNjFKR177LFZfVb6SAWDQh999FEg2YGREucxc+bMrFN885mzzjoL8ONVpNN7EFFzpkyZQt++fQF/pRVsSZAvyPcwiMTzBMtIJGlFXB8kCDsYdyVtQ7LpoJ1k+vTp477DMofecsstrs1NEikqKnL3mSisEydO5L777svZOaRtkRz/iSeeqFO7hVzw/vvvA97ugOx+yH0mcWVVId3to0iIiNzhkV4a4FdVluDffECybqQ5KPgPd+mlFKz2KlthktEFfj+qP/3pT4B3Q3ft2jU8o6vhiSeeAOCoo46q9Wclwl4kS/AbTuYLwZo8aUUeguJUP/zww+412UKVbSypABv8W5IfLLVBau8EAyaTWCeptrRs2bJSw9pNmzaxYMGCeAzKMRLECn6Pxffeey8uc7Ji1KhRziGRxrzyfMgVUrleFskLFiyIzeERR7S8vNzVeZLtYqkfVRXybJQg9DD7aOqWlqIoiqIoqSdyhWfYsGH84x//AKjUbTsfkIDr3r17u2CxigSDQUXhadeunVspiwIkq8tM8nvYyPZVsHJ0RZ599lknN2aiqkqu+cKOHTvcNof0dcvU7TdbROk6+uij632sXCJdtiWAVyTwPn36uPIJwZ5nskVbXQfufERKPgA89dRTQDz3Xq4ZOnSom5eEiRMnsmrVqpgsyg1yTwZVWEkMkBTmpCHqRklJiVN2MvVOrC8zZ850VZUlCUTOlxSkk0IwSSVTj0VJ5jnuuOMAP+EgDFThURRFURQl9USm8IgaEAwAzkck1a60tLRKhacqpDO3KDvSVbtly5bu30TUoajimjKliA8aNAjwqmFWp/D06NGj0ud69epV5fvFu09Cnxfwep5Jf7P99tuv3seT+K7mzZsDyUnjfvLJJwE46aSTgN271Mt+ebAYpBQGlcrRaeGMM85wv8+aNQsIN14gbA4++GDAv1/BL/KWBnVOikdKCQGA3/3ud3GZkxUSJ1ZQUMC///1vIDfBuFJcUAoWlpSU8Ic//AHw1fqkIAkQBQUFu/39kUcecQqYfGevvPJKN/dkKi6Za1ThURRFURQl9ZjqisAZY+pdIU6K2UlmyJFHHun6UkkH1LCw1tbYwyCbMY4dO9b1fwoiJdsnTpwI+FlXQWRVPWjQIN566y3A94BlJd2oUSNXRjvYXT0b6jtGaS0QzNKaNGkS4PWokbTdbNSKfffdt1pl6qKLLgIy/ztVR01jrOv39JBDDnEdl8vKygB/D762KuTxxx/vCoxJIcbTTz89q1YMufqeVoXEFkkne7lGc+fOdVkdM2bMALzv9GmnnQb4BRpzQdhjrA6JU1u5ciXgxW7J9z1TJ+e6EvUYpeP2uHHjnIog45K5JteEdS8GadKkCeApAuD3RQO/tU9Y1PcaSjfvwsJCdy0yFb/MhqKiIjd2udaimsybN48hQ4bU6bhhf08lZrBif8zi4mL3vJHMsjfffNONSTLvRLmsD1WNMbQtLTFaHJ1OnTq5/89UyybJjB8/3l2Uq6++GvBuvBNPPBHwA5OlEePIkSMr9dnq0KED3bt3B7wAZtj9oSoBXrlullYTkgoYdHiCN9JPfvITIPPDT6qeyjhq6pF1+eWXA77zFNxaiYP169c7p1WcT6k1c+utt2ZVEVq2r5YsWeKCgaWKalL6Tkm9JLkHg1SsVjt9+vScOjpJQCq5yiQ7YcKEnDo6cRFsvChbc2E5OlEi86SkdVtrE99cWpAFRFFRkUunl22omhDnRrbFSkpK3HdW5kqpcZPk2krBLdaqyLRdGUUDX93SUhRFURQl9YSm8IhiULHq8JgxYxKz8s2W7du3u6JREsjZv39/l0YngcyyIgmmI1dUQTIxZswYJ99GTbAYlkiR0i0bfA886IkLouhk2xuruLgYwFXajlvhAb9QopRKkL5unTt3dp2HpX8a+BVtTzjhBACGDx8OeIHnUkU1X3oXde7c2XUvll5u06dPj8+gkBAFVqiu91s+IMUig8GqSQ/mzYYWLVoAmQOux40bF7U5dULmgMLCQm644QYAlz5eXZXlk08+2YU/yA7A6tWr3fwiBWLzPZGgtLR0t58AO3fuBLJXwuqDKjyKoiiKoqSe0BQeSdGV1b94t/leDEsClB944AG3KpbeU8HCZpmQwoOyzy6BssE2FVEjwdKXXXaZWwlXLGIGfsrg5s2bAejatetuwYSCBDmvW7cO2D0eaODAgYAfKJ0ERG2UlGWJL+vZs6fbg3/nnXfc+0WdEuS1sWPHut5FSUeCQkeMGOHuT0lxldVWWujVq5crlSBlEWS1nK/IvBMM4F20aFFc5uQMSeYIKswAixcvdskFSUfmv+LiYtemRZTtyZMnV4p3lPuvvLzcKd4yB0WheIRBVbGcDz30kItplTkI4J577gGi+Q6HkqXVunVrXnrpJQD2339/wA/elYahURBV1oQ0thOpWZy9IGVlZbzwwgsA7mcuiDP7JSqiyAypyCWXXOImLMla+vLLL139lueeew7wHF/wMtrqStTXUL6vS5YscUHyEtgbFnF9T6+77jrXpFfqYInjnWuiGqMEd0piyNKlS1319h07dtT38NUS5r0ofRYrPiOuueYa7r777roetlbk8hrK1pRUc6+Ot99+O7I+WGF/T0ePHg14DV6zoWfPnsDuPf7qS1Vj1C0tRVEURVFSTyhbWk2bNnXKjgSBJr27bX0QGVO4//77Y7JEyRXTp09PXQDvgQceCOwe4DplypS4zImELl26uN+lW3y+Iv2HGjfefdqeMGFC6MpOFEgPt4rjy1ckwDjfA41ri8ybstPRvn37Kt87YMCAnCo7NaEKj6IoiqIoqScUV7q8vDw1XrqipIWmTZsCfpG6adOmsXTp0jhNCg0pFBpMSV+wYEFc5uSEM888E/CVOkkMkN58ipIEJAZLChBm6mwvHQUkJjIqVOFRFEVRFCX1qAyjKA0EWXlJb6k0I6mxxhgef/xxILfZkUng3nvvBfxSEYqSJOS+k+K7SSD05qFxoinbHmkfY9rHBzrGfEDHmP7xgY4xH9C0dEVRFEVRGizVKjyKoiiKoihpQBUeRVEURVFSjzo8iqIoiqKkHnV4FEVRFEVJPerwKIqiKIqSetThURRFURQl9ajDoyiKoihK6vkfm3K9t5x9dLEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize how the model is doing on the first 10 examples\n",
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "linear_model.eval()\n",
    "batch = next(iter(train_dataloader))\n",
    "predictions = linear_model(batch[0])\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(batch[0][i,0,:,:], cmap=\"gray\")\n",
    "    plt.title('%d' % predictions[i,:].argmax())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise: How can you improve the accuracy? Some things you might consider: increasing the number of epochs, changing the learning rate, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction\n",
    "\n",
    "Let's see how our model generalizes to the unseen test data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For HW: cell to change batch size\n",
    "#create dataloader for test data\n",
    "# The dataloader makes our dataset iterable\n",
    "\n",
    "batch_size_test = 256 \n",
    "test_dataloader = torch.utils.data.DataLoader(test_data, batch_size=batch_size_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.3326, test accuracy: 90.92%\n"
     ]
    }
   ],
   "source": [
    "acc_test, loss_test = evaluate(test_dataloader, linear_model, loss_fn)\n",
    "print(\"Test loss: %.4f, test accuracy: %.2f%%\" % (loss_test, acc_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now take a closer look at the results.\n",
    "\n",
    "Let's define a helper function to show the failure cases of our classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_failures(model, dataloader, maxtoshow=10):\n",
    "    model.eval()\n",
    "    batch = next(iter(dataloader))\n",
    "    predictions = model(batch[0])\n",
    "    \n",
    "    rounded = predictions.argmax(1)\n",
    "    errors = rounded!=batch[1]\n",
    "    print('Showing max', maxtoshow, 'first failures. '\n",
    "          'The predicted class is shown first and the correct class in parentheses.')\n",
    "    ii = 0\n",
    "    plt.figure(figsize=(maxtoshow, 1))\n",
    "    for i in range(batch[0].shape[0]):\n",
    "        if ii>=maxtoshow:\n",
    "            break\n",
    "        if errors[i]:\n",
    "            plt.subplot(1, maxtoshow, ii+1)\n",
    "            plt.axis('off')\n",
    "            plt.imshow(batch[0][i,0,:,:], cmap=\"gray\")\n",
    "            plt.title(\"%d (%d)\" % (rounded[i], batch[1][i]))\n",
    "            ii = ii + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the first 10 images from the test data that this small model classified to a wrong class:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Showing max 10 first failures. The predicted class is shown first and the correct class in parentheses.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeuElEQVR4nO2de5gUxfWw31oWYYEVg1zlGkExiOIdo1GQhFvIR0AIn0FFjZGggkLQBIncBQMKxkuMipcVQyBGFBUN0aggSlAUo0YURAkiQhAFFUFZlvr90XNqZmdnZmd3eqZ7h/M+zzyw09U9deZU9VSfOhdjrUVRFEVRFCWfKQi6A4qiKIqiKNlGFzyKoiiKouQ9uuBRFEVRFCXv0QWPoiiKoih5jy54FEVRFEXJe3TBoyiKoihK3hPYgscY08QY854xpiiNtscbY1bmol9+ke/yQZVlbGaMedcYUycXffMLlbFC2xonozGmkzHmNWOMSaOtzsWQonqs0LbG6TFoHfq64DHGnBdRwNfGmA+MMWelaD4OKLHW7o2cW2KM2WeM2R3zqgVgrX0L2GWM+X9+9reqZCJfzDUaGWM+Nca8JO+FSL4/G2O2GmO+NMasN8b8spJT4nU4xBiz0hizxxizLLahtfZ/wAvA8Kx0Pg2MMXWMMfcZYzYZY74yxvzbGNO3ktPiZZxljNkc+Y42GWPGS8MwyAgQN4d2G2PKjDG3pzglXsabjTHvR76j94wxw6RhWGQEMMYcZYz5xhjz50qaTgNutpGkY8aY7xljnjfGfGGM2WCMGSgNQzQXR0Z+GL41xpSkcUq8Dt+JGwP7jTFPQrh0CBnpMek4D5Eek461JOhcjJAVHVprfXkBPYFNwOl4C6mWQMskbesAO4BWMe+VADekuP75wBK/+ptr+WKOzQVeBF4Kk3yRPhwL1In8/xhgG3ByFXT4I2AIMBFYluCcM4H/BChffWAy0C6iw58AXwHtqiBjR6B+5P8tgXeAc8MiYwIZGgC7gbOrIOOUiP4LgK7ATuCMsMkIPAOsAP6cok0L4HOgbuTvQmA98GugFtAD+Bo4OuacMMzFc4EBwJ/wfgBTtU16v4kcN8BGYFjYdFhdPSY4XmGcB63HdMZaZXrUueivDv208EwBplprV1lrD1hrt1hrtyRp2xXYZa39uArXXwb8MEDzXcbyGWPOADoDDyQ4ZxnByoe19h1r7bfyZ+TVPknzCjJaa/9prX0Y+CTJOa8ARxpj2vrV56pgrf3aWjvZWvvfiA6X4P0QnJzklEQyrrPWfh3T5gDQIebvQGVMwCBgO97NKBGJZJxkrX0v8h29Ejn3+zHnBC6jMeY8YBfwXCVNewJrrLXfRP4+BjgCuMVaW2atfR54Gbgw5pxlBD8XH7XWLgY+S6N5ZffTs4HGwKKY9wLXIWSkx3gSjfNlBKvHdMZaLDoXszwXfVnwRLaeTgGaRMxSHxtj7jDJ9yGPA9YleP8KY8znxpjXjTGDYg9EFheleE/YOcUP+SLXuAMYibeQKEeQ8sVijLnTGLMHeA/YCjydpGkyHSbFWrsf2AB0yaiTPmGMaQYcjWelSURCGY0x44wxu4GP8axGf5FjYZMRuAiYZyOPSwlIqcfIGD+VmO8oaBmNMYcCU/GeDCsjnXFq8B5EgPDMxSpQmYwXAYtiF+pB6xB812OFcR5SPZYba3HoXMzyXPTLwtMMqA0MBs4CTgBOBK5P0v4wvK2EWG4DjgKaAhOAEmPMmXFtvoqcm2v8kO8q4BVr7espPico+RzW2iuAYjw5HwW+TdI0kYzpELiMAMaY2sB84EFr7XtJmiWU0Vr7e7zv6CTgIeCLuCZhkbEt0A14MEWzyvR4F/Am8I+494OUcRpwX5oW4nj51uFZAq41xtQ2xvTC+47qxZ0XCh2mSVIdGmPq4d23ShIcDlrGTPToqGScByljumNN0LmY5bno14JHHHNvt9ZutdbuAOYAP07SfifeD4bDWrvGWvuZtXa/tfZpvB+jc+POK8YzneWajOQzxhyBt+D5XSWfE5R85YiYF18CWgGXJ2lWQYdpEriMxpgCvIXKPjyLWzKSymg93sAbG1PiDgcuY4QL8XzFNqZok1RGY8xNeE9bQxJYiAKR0RhzAp6v2C1pnlJOPmttKZ5vTD88H7WxwMN41rpYwqLDdEg1F8/F85tYnuBYYDJmqsc4Uo3zwGSswlgTdC5meS76suCx1u7E62SsIlKVYX8Lbysh5WXxzFsAGGNaAodQxW0UP/BBvtPwHLbWGmO2AbcCpxljtkW2ugKVLwWFJPfhSUeH5TDGFOL5u7yZYb+qjTHGAPfhWe0GRSZdMtKRsdx3FAYZYxhGausOJJHRGDMF6Av0stZ+GXcsSBm74zmdfxSZS9cAg4wxa5K0ryCftfYta203a+3h1trewJHAq3I8pHMxFanGacItzRCM0+5kqMcYEo7zMOixsrEWh87FbM/F6ng6J3rh7eOtxtuS+g6ec9W0JG0PAT4lJsoJz+zaAG8R1gvPjNU95vhQ4Gm/+ptL+fC875vHvK7GczZrHiL5mgLnRXRQC+iN5zHfvwo6rAXUBUbgRaLVBWrHHD8DWBuUjJE+3AWsAhqk0TZejwXAryL6N3gL2a3AVWGSMaYfXwPFVZEx8t51wPux4zPBtQOREc/cHTuXbgYeAZokad8Mz/G3bsx7x0fGZj28m/RGItGJkeOBzsVIHwojfbwRzxpZFyhMV4eR91sB+4H2YdKhX3qMkSPhOA+JHlOOtcr0qHPRXx36KXxt4E4809M2PJ+chCGEkfY3Ab+N+XsFni/El3ir1fPi2j9Fkh/fHCk3I/nijl1MxbD0oOVrgmf23hXRwdvAZZWcE6/Di4lGd8mrJOb4H4lZHAQgY9tIn77BC2GV1/npyIi34FmKt0WwGy+kcjxgwiJjTD/uBh5Ks228Hi2e71bsdzQ+bDJG+jKZFKGwkTZ/A/5/nLw7I3L9HegQ1z7QuRgjV/xcmpyuDiPvXQesSNI+NDqsrh4j7yUd5yHRY8qxVpkedS76q8Mgv5wmeJFARWm0PR74V9AKVfkykrEp8C4pFolhfKmMNV9GoBOeddak0VbnYkhfqsear8egdWgiF1YURVEURclbtHiooiiKoih5jy54FEVRFEXJe3TBoyiKoihK3qMLHkVRFEVR8p7CVAeNMTXao9laayprozKGn8pkzHf5QGWsCaiM+S8fqIw1gWQyqoVHURRFUZS8Rxc8iqIoiqLkPbrgURRFURQl70npw6MoBxtdu3YFYObMmfzpT38CYMmSJQB8/fXXgfVLURRFyQy18CiKoiiKkvekLC0RhKf2d77zHQDatGlT4dimTZsAGDNmDP/5z38AWL9+PQBvvvlmhfa58kY/7LDDAPjyyy8BOHDgQKaXTJtcySjWjuHDhwPw8MMP84tf/AKAvXv3Znr5lOQyMuSvf/0rAIMGDcIY72MXLVoEwJQpU3jnnXf8+ijHwRw1EYvKGH40SktlrAlolJaiKIqiKActobDw9OvXj/79+wPQvXt3ADp06FChnVhz2rZtS506dcodq1WrVoX2uVrJPvvss0DUx+Pee+91fh+Z0LRpUwA+//xzAPbv31+hTbZlbNGiBQAvv/wyUN7y1rNnTwBeeOGF6l4+LXL5VHn11VcDMGvWLAoLC+XzAfjss8+4/vrrAXjggQeAxDqpKgfzE1csKmP4UQtPdmQsLCx095mysjK/L1+Og3mc5txpuX379lx55ZUAXHbZZQAUFRW57YNUHH300VntW3VZs2YNANdeey0Ay5cv9+W6o0ePBqB27drlrp9Ltm7dCsC2bduA8gue8ePHA/Daa68B8NVXX+W4d/5z6623At4NSL5/WfQdfvjhbmvvqKOOKtd+y5Ytue6qUkUOOeQQAK666iomTZoEeItYiD5c9OzZ0y3u27ZtC8DQoUOZOXMmkNvt6mS0a9cOiN4PL7jgAr773e8C8OGHHwJw5JFHMn/+fCC6OP/2229z3FOlMn7yk58AMG/ePDcWZ8yYAcCDDz4YivGWDuKKMnToUMaNGwdAq1atKrRbvHgx4MkW+3eu0C0tRVEURVHynpxbeFq1auW2DdLlvffeA8iKw6gfbN682fdr9uzZk1//+tdA9Mk0CAuPsHLlSgBOO+00994555wDRJ9SFixYkPuOZYnZs2fz8ccfA1ELT/v27bn88ssBnG4aNGgAwBVXXBFAL9NDtubEMnDBBRcA0b7H8uijj7J27VoAdu3alZsOZpmCAu+57pZbbgGgU6dOjBw5Eog6qd95552AZyGRp9V//vOfANStW5d7770XgE8//TR3HY/jiCOOAGDZsmUAtG7dGgBjjNsOOeOMM1z7M888E4DGjRsDcMMNN+Sqq+UQq+grr7wCQElJSSD9CCMSfHP//ffzs5/9DMCNtUsuuYRf/vKXQNSdI2wUFRUB8NhjjwFw9tlnu2Pi6vDWW28BsG7dOgYOHAjAQw89BMCFF16YUyuPWngURVEURcl7suK03LhxY2fFkf3wpUuXAnD66afz9NNPA1En3/r16/PMM88A0RWvPA288cYbLuy5qonfcuWcJZan733vewCMHTvWPU1Wl+nTp3PdddcBUR+hU045pUK7XMnYq1cvAJ544gkgajUAuO2224Co1cNvwuIoWVRU5PyWrrnmGsB7+geYOHEi06ZNq9Z1s6HDLl26ANC7d29ngZMn/sqQp0kZf348gQXlKHnooYc6HxbxQ3vooYdYtWpVuXY/+MEPAO++Iz54YkHp27evuy+lItsyytO0jEH5d9++fdxzzz0AfPTRR4DndC9+keJfdtJJJwGZWamqMxflN2b79u2AZ72Wp/5sceyxxwKeBQG8RKI7d+6s9LwwOPTKvXbhwoXOf/OYY44B/PEV9FNG+Z2X37uNGzc6y45Yw0tLS117sbb+5S9/AWDgwIGcd955QNRK5AdJZbTWJn0Btiqv+vXr2/r169s1a9bYsrIyW1ZWZvv372/79+9frl27du1su3bt3N9t2rSxBQUFtqCgoEqfV9krlWzVlTH21blzZ9u5c2e7a9cuu2vXLnvgwAF74MABO2bMmIz7/uqrr7rrJfoOcyVj/Gv58uV2+fLldv/+/e61ZcsWu2XLFtuxY0df9ZeujNn4zMpeixcvtosXL3bjfMWKFbZOnTq2Tp06gY7T4cOH2+HDh9uVK1falStX2rKyMqefuXPn2rlz57r517hxY/fq1KmT7dSpk73++uttaWmpLS0ttXv37rV79+618+bNs/PmzcuqDrOlxyFDhthVq1bZVatW2RYtWtgWLVqkbH/OOefY1atX29WrV9t+/frZfv36hU7Ghg0b2oYNG9qSkhJbUlJiBw8e7I41b97cNm/e3JaVlbn7x44dO+yOHTts69atbevWrbOqx0TnyP1R5sr06dNtvXr1bL169XzVdaNGjWyjRo3slVdeabdv3263b9/uPrOkpCRUOkznNWjQIHeP7dOnj+3Tp48v1/VTxg0bNtgNGza4sXb00UendV5RUZEtKiqyixYtsuvXr7fr16+3TZs2tU2bNs2qjLqlpSiKoihK3uOL07I41YqZqkuXLi60Thz/Yvnvf/9b7m8xw9Y0vv/97wOe2TyWTGouSX6h2rVru628F198sdrX8xtxfPz73//u3mvWrBkAjzzyCMcdd1wg/co1P//5zwHYvXs34I0FGQdBObZ269bNzTsZRyNHjuS+++4DvK2PZOzYsQOAtWvXuvDYO+64A8A5Gp588sm8/vrr2el8lujYsaObP5JiIRGyBfLEE0+49ArZCEbwgy+++AKAiy++uMKx5s2bA5RL85FOyo9scv755wPR7fBx48a5tA6///3vAW+78ZNPPkn7mm3atHEBFH379gW88Q+4EP1Y0t3ODROLFi1y28snnngiEHUNCStdu3ZNy8FaftsmTJjg1giydR7reO83auFRFEVRFCXvydjC06BBA+fcKM6RO3bs4OabbwZgz549mX5EKGnQoAFjx44t9544XYkDYXX46U9/CnhWsrlz5wLhCg8WJ/SdO3e68F2hYcOGzsohdcXyFXG0CxNPPvkk9evXB7y6XxANCa4KEhY7bNgwIJqKoEmTJn50MyfcdNNNgBcme9ZZZ1XaXtIKFBUVcckllwBk3bE2G/zoRz8CwFrrLDtigRen51zzj3/8A4haJ3r37s2gQYMAL8s+eI6tYoEUS49YKf/2t7+5a8k9Z9iwYQnTKiQj1wnuktGyZUsAhgwZAkQrC+zcudPJKffYli1bOhn9dOj1E0nOKukdZs6cydtvvw3Av//970rPX7t2rZt7d911FxAdp9mo0agWHkVRFEVR8p6MLTwDBgxwqaTFF+ess85y+8z5yi233FKh1MXUqVMzvq5UIA8rYrGbM2dOhTDsli1bun30J598Mud9yzaxYenx1r1nnnkmrbDXbFJcXOxS0WdS5kPCSGtiKQKxvMm/HTt2TOi71KhRIyCaxl/K3MyZM4dHHnkkF13NCpIaIxZ54g4qeZ3UmxNrzrRp0/jVr34FeOVaIJreIfY94Xe/+13C64rVQyx4kmARovWoZPdBrOVBUFxcDMDgwYP5wx/+AET9qsR3rrCwkIsuugiI+ppt3rzZpWeR5LthQ2pGTp48GfAsy+LfKToW361YOnfuDMCNN97IhAkTgOh3cumllwJRH0I/yXjBE+tg9MYbbwC4DLX5iBQ5layYEHXCXrduXbWv27BhQyBa0yfszJ49mx//+MdA1HkbcMU1JTOzOMCGFckhJPXdxJy+ePFit5X4zTffAJ7u43MhSVbiyy67zJdCopkwYsQINwblR646tG/fHvCclCH6nbz00ksZ9jB7SF4oqXklN9HYLXXJAdKjRw8mTpwIRGvDjRkzBoiOh5qGbGXJVlEsixYtynV3UjJhwgSeeuopILowHTZsmFtgyyKgU6dOQPlimpJT6bnnnnNbj/FbW/v373dZ4OU+FCTSlz59+rgFjwQSiJHgkEMOoXfv3kB0gdCiRQvnGiB13TZt2pS7jlcB2UJ+9913Xcb9Rx99FIhm4J80aZKr9XbYYYcB3iJY7lniCjJnzhzAC4KSwtl+oVtaiqIoiqLkPRlbeAYPHuz+36dPH8BbyT3++ONAeo5LNQFxxpUnx9hQ9HPPPReIWgKqg4RSnnDCCe49eQoII/v27XNPZGKKLCgocBYQqT8VdguPOMhJNljp9+WXX+7kis1GLtli//jHPwLRJ84wVEu/++67fbmO1NqqV68eEK0zJSH4YURM5GLpEVN7q1atXA0xCY8eMWKE2+YSi6041tZUxDoQnyIDok6wYUIyXcu/4vwK0V0Dsb7FPuVLCPOYMWNcRuJ4Jk6cGArLjoTCS92ooUOHOstWPPv27atgqfrwww9dbSrZ2pKgFvk7bCxZsoTTTz8dwFlRZd4NGDDA7QKtWLHCnSNzV1xjZAtTLLJ+ohYeRVEURVHynoxraVlrnaNkLPKehJrJSr5NmzZs2LABKF/9XBJ//etf/wL88QPys2ZIhw4dgPKOf+I0JyGGsXvNVUUsO1I3C3DJuT744IOk5wVZ++X+++8HouHLsVWbxRJ24403Zvw52aylJaGvPXr0AODUU08FPN8C8aeS/WaIWnakhowfhKF+jzBu3DimT58OwK233grAtddeC2Q2vrMto9QaevPNN4HovaVLly7Ox04CKY455hhXG0tC7v0gKD0WFxc7i5bUBINofaPf/OY3AAnv01Ul6Lp2YnVdtmxZOVkhak3u0KFDtdNi+KlD+c2TtADdunVLGkzQt29flw7if//7n3uva9euAM73RyzQJ554YrUdmXM1TkVX4kj/4IMPunuq1KmL9AeI/p7KjsnAgQPdTlFVSSZjxguem266ydeikZKldtmyZUBm+U78Umzv3r3dj4AU4NuwYYOLSBKvelGw5EKJpbS01BWCEyQjs7W2woLnsccec9uFqXQU5I+lbAeIc1rsgkccXMWZMhOH3qBusrI9IKbZ0aNHu/EpC6OwLcyrS8+ePQEvV4rcSGXLIGwFC1MhBQuHDh0KePLIIvW3v/0t4C1W5UHMz3tXUHocPXo0s2fPrvC+RC35GT0Y9IJHHJljnfIluEC2ezJxrvdThxLtJ79liaKOZNvu5ZdfdtusshUmxW4h+sAtRbb37dvn5mdVKxUEeb+RXF6Sj2jGjBnOFUZ+Y2VLa/z48S4Td1VJJqNuaSmKoiiKkvdkbOGpVauWq/MhtbQKCwudyaq6jkfSr8mTJ7v6TdW4hi8r2TvvvJMRI0aUe2/z5s0V8guIw6TkH4hcH/AsN/I9CeLMtWTJEvdkKivaBx54wOUjSEWQq3WRVxzROnXqVMEaJY5oknm7OgT9VClWveXLlzsn5+OPPx6IhqVnQpA6FP1cddVVgLclJDlp/EwvEaSMIptsC6xatcpZZyXnkB8EJeOKFSsq1B+655573D3FT4Kei/Pnzwc8y7840YtFwI+8LX7qUAI4xNIzatQol59MchKJZW737t1uN0NcPhIRa+kRi5Ckj0i3hmMYLMrC2LFjXUi7bH1J1uZPPvnEpT6pKmrhURRFURTloCXjsPSysjJee+01gHKZh3/4wx8COL8VycQovg+VIZYRWb0GSaJkXq1bt3bJ6lIhFo/i4mJXfVkc6uRp5fXXX69Qp6g6NZByjfjliLOdJIyKReqrZWLh8RtJjvjiiy9WqRJ9QUFB4JWn/UAcB2fNmuV8XSQse8CAARk5J4eN1q1bu7EnviyDBg3y1bITNLGVwMUx24+s72FCamjFJnwVa3g2MvL6gfwuzps3D4DHH3/c+TvKfVGSevbv3z+tavFi/enVq5fz91y4cCHgBc9ko/5UrpAkofI7KYEFfqIWHkVRFEVR8p6MLTzJeO6558r9LVFIp556qrMMSNK2uXPnusRT8sQZJqZOncrtt99e7r2PPvrIhQ/GJ2R7/vnn3epeePXVV10iN0mdLkmlRo0a5SxZElb7/vvv+yxF9khVUkN8Xdq2bRt4WnSJepMwXRl/iejQoYPz/ZDU8HXq1EkZMVdTEJ+CM88801UEFytdcXGxC8OXcO7KkLpU4rcnYzhIatWqBXhzMT7qLDb6pSYjEZAQtSSLj4hEjuYLMmdFr+BPmH0umDVrFuDd72UXRHxCxeJT1TD6DRs2OGuXzOdVq1a5NAs1sQ5ePNmox5m1BU88Ek43ffp05+wqzpEdOnSge/fuCc8LQ12uu+++u0Jmy23btrkMoLH1elIRHx767LPPAtHwbojmIapJxVeXLl0KwOrVqyvUmpLQ7lGjRnHNNdfkvG9CQUFBhQzCpaWlbntHnJEvvvhi96+EjMqPyZ49e5z5PKhCjNVBiheKg6RkQoXoglRuvDt27HCBBrKQl/kai5jfly5d6pxj5TuUzwuSUaNGAdC8efNQ9MdPJO1F/EMYhGvr2C9OOumkhPeO6ubayTUyj/r06eMWaYkK2lYV2YYWN5E1a9a4fEySebumLAoBmjVrBkTdYbKRIVy3tBRFURRFyXtyZuF59913AXj44YddZmJBtgwgms1Vao5I2GyQ7N+/v8IWlZ9MnTrVhczWZFP0U089VcHCI8RWVA+Cxo0bO0dBYevWrSm3qJYvXw54ldPBq+PjRxh6rhGLjqQ5kARfybae5MlZEoKJxad+/fruO5FQ4EMPPdQ5AB9++OHZ6H6VEEvxtGnTgGj19HxCLFaxQSL5zAcffOCqbEtYNoRj67QqZFJrMRWyZXvppZe6ul0jR44E4LbbbsvKZ2aDI488EoC6desC0Z0DP1ELj6IoiqIoeU/OLDwSLjd69GhXFVasAU2bNnUOkrJClTD2g4FPP/3UlSyoycyfP59JkyYlPCZ1UoLi888/d35kiaosiwOzPIUtXLgwlFWmq8MLL7wAQMeOHYGoL1lllezFxywR4ivwxRdfuPDgIH1lpJ6bJDETq1x1U9PXBCRFQj6kSkjFnj17KvhJfvvtt1kJW67JLFiwwPkkSooQCfEO+v6bDuPHjy/3dzb8dzPOtJwJF154IeCZ3KdMmQLA9u3bfbt+mDJKZoswyVi7dm3nwCpbHuIg3K1bt3KFUatC0Nlds02YdJgtsi2j1FYSh14pRpzLvCS50qNsV0hxV2OMi86SnGGZ1K5LRRBz8ZRTTqkQNLJw4UKXqd5PavpclCg2qd8lgRdt27Z1bcIkY2ymZVm4yyLt2GOPTVpstTI007KiKIqiKActOdvSSoRsX8m/Ss2mtLTUOcnVJGc5JX+QrcmanHG2MsSaJc6q7dq1czmQsmXZCRsLFiwIuguhRIJ+pFZcdWtZBoGkYunRowdAta07qag534aiKIqiKEo1CdTCoyiKkinNmzdnxowZQLTScj4jqQEk0/LVV1+dMtt5TWfjxo3Oh+e4444DwpGQNsxIwsGwJx58++232bhxIxBNTpyqWnymqIVHURRFUZS8J9AorWwTJm/0bKEy5r98oDLWBFTG7Mkn9dqaNGkCpK7flwmqQ498lVEXPCpj6NEFj8pYE1AZ818+UBlrAhqWriiKoijKQUtKC4+iKIqiKEo+oBYeRVEURVHyHl3wKIqiKIqS9+iCR1EURVGUvEcXPIqiKIqi5D264FEURVEUJe/RBY+iKIqiKHnP/wEYyzgON9x2AAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_failures(linear_model, test_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- # Exercise:\n",
    "- Try changing the loss function,\n",
    "- Try changing the optimizer -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilayer Model\n",
    "Our linear model isn't enough for high accuracy on this dataset. To improve the model, we often need to add more layers and nonlinearities.\n",
    "<img src=\"images/shallow_nn.png\"  align=\"left\"/>\n",
    "\n",
    "The output of this NN can be written as\n",
    "\\begin{equation}\\label{eq: NN1d}\n",
    "  \\hat{u}(x) = \\sigma_2(\\sigma_1(\\mathbf{x}\\mathbf{W}_1 + \\mathbf{b}_1)\\mathbf{W}_2 + \\mathbf{b}_2),\n",
    "\\end{equation}\n",
    "where $\\mathbf{x}$ is the input, $\\mathbf{W}_j$ are the weights of the neural network, $\\sigma_j$ the (nonlinear) activation functions, and $\\mathbf{b}_j$ its biases. The activation function introduces the nonlinearity and makes it possible to learn more complex tasks. Desirable properties in an activation function include being differentiable, bounded, and monotonic.\n",
    "\n",
    "\n",
    "Image source: [PragatiBaheti](https://www.v7labs.com/blog/neural-networks-activation-functions)\n",
    "<img src=\"images/activation.jpeg\"  align=\"center\"/>\n",
    "\n",
    "Adding more layers to obtain a deep neural network:\n",
    "<img src=\"images/deep_nn_annotated.jpg\"  align=\"left\"/>\n",
    "\n",
    "\n",
    "\n",
    "# Important things to know\n",
    "Deep Neural networks can be overly flexible/complicated and \"overfit\" your data, just like fitting overly complicated polynomials:\n",
    "<img src=\"images/bias_vs_variance.png\"  align=\"left\"/>\n",
    "\n",
    "\n",
    "Vizualization wrt to the accuracy and loss (Image source: [Baeldung](https://www.baeldung.com/cs/ml-underfitting-overfitting)):\n",
    "<img src=\"images/acc_under_over.webp\"  align=\"center\"/>\n",
    "\n",
    "\n",
    "To improve the generalization of our model on previously unseen data, we employ a technique known as regularization, which constrains our optimization problem in order to discourage complex models.\n",
    "\n",
    "  - Dropout is the commonly used regularization technique. The Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting.\n",
    "  - Penalizing the loss function by adding a term such as $\\lambda ||\\mathbf{W}||^2$ is alsp a commonly used regularization technique. This helps \"control\" the magnitude of the weights of the network.\n",
    "    \n",
    "Vanishing gradients: Gradients become small as they propagate backward through the layers. Squashing activation functions like sigmoid or tanh could cause this.\n",
    "\n",
    "Exploding gradients: Gradients grow exponentially usually due to \"poor\" weight initialization.\n",
    "\n",
    "<!--  <img src=\"images/test_data_rule.png\" width=\"800\" hight=\"500\" align=\"center\"/>\n",
    "  -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now implement a deep network in PyTorch. nn.Dropout() performs the Dropout operation mentioned earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For HW: cell to change activation\n",
    "class NonlinearClassifier(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.layers_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 50),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(0.2),\n",
    "            nn.Linear(50, 50),\n",
    "            nn.ReLU(),\n",
    "           # nn.Dropout(0.2),\n",
    "            nn.Linear(50, 50),\n",
    "            nn.ReLU(),\n",
    "           # nn.Dropout(0.2),\n",
    "            nn.Linear(50, 10)\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        x = self.layers_stack(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### For HW: cell to change learning rate\n",
    "nonlinear_model = NonlinearClassifier()\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(nonlinear_model.parameters(), lr=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: training loss: 0.6724944891134897, accuracy: 80.11041666666667\n",
      "Epoch 0: val. loss: 0.6639984491023612, val. accuracy: 80.04166666666667\n",
      "Epoch 1: training loss: 0.3835423557758331, accuracy: 89.10625\n",
      "Epoch 1: val. loss: 0.3745706133702968, val. accuracy: 89.125\n",
      "Epoch 2: training loss: 0.2927929045756658, accuracy: 91.61875\n",
      "Epoch 2: val. loss: 0.2848920944206258, val. accuracy: 91.71666666666667\n",
      "Epoch 3: training loss: 0.24066319421927135, accuracy: 92.97291666666668\n",
      "Epoch 3: val. loss: 0.23610803881224166, val. accuracy: 93.025\n",
      "Epoch 4: training loss: 0.20259241100152334, accuracy: 94.08333333333333\n",
      "Epoch 4: val. loss: 0.2034157008883801, val. accuracy: 93.94166666666666\n",
      "CPU times: user 36.5 s, sys: 433 ms, total: 36.9 s\n",
      "Wall time: 36.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "epochs = 5\n",
    "train_acc_all = []\n",
    "val_acc_all = []\n",
    "for j in range(epochs):\n",
    "    train_one_epoch(train_dataloader, nonlinear_model, loss_fn, optimizer)\n",
    "    \n",
    "    # checking on the training loss and accuracy once per epoch\n",
    "    acc, loss = evaluate(train_dataloader, nonlinear_model, loss_fn)\n",
    "    train_acc_all.append(acc)\n",
    "    print(f\"Epoch {j}: training loss: {loss}, accuracy: {acc}\")\n",
    "    \n",
    "    # checking on the validation loss and accuracy once per epoch\n",
    "    val_acc, val_loss = evaluate(val_dataloader, nonlinear_model, loss_fn)\n",
    "    val_acc_all.append(val_acc)\n",
    "    print(f\"Epoch {j}: val. loss: {val_loss}, val. accuracy: {val_acc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9e40644650>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAJNCAYAAADgY3uzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXjU1f328ffJThb2LCQBEvaEkAQSWWRXwBURBDe0ioqK/VWrte6t1lZbW21pbZ9al1prLYgsohYFQYSgiKxhCUtYAgRCEhIgkIVkZs7zx6S4FBUwk2+W+3VdXMDMfCf3hAp3z/nMGWOtRURERER8z8/pACIiIiLNhYqXiIiISD1R8RIRERGpJypeIiIiIvVExUtERESknqh4iYiIiNSTAKcDnIn27dvbhIQEp2OIiIiIfKe1a9cettZGnu6+RlG8EhISWLNmjdMxRERERL6TMWbvN92nrUYRERGReqLiJSIiIlJPVLxERERE6kmjmPE6nZqaGvLz86mqqnI6ipyFkJAQ4uPjCQwMdDqKiIhIvWu0xSs/P5+IiAgSEhIwxjgdR86AtZaSkhLy8/NJTEx0Oo6IiEi9a7RbjVVVVbRr106lqxExxtCuXTutUoqISLPVaIsXoNLVCOnPTEREmrNGXbycVFJSQnp6Ounp6cTExBAXF3fq99XV1Wf0HFOmTGH79u3f+pi//OUvvPHGG3URGYDCwkICAgJ4+eWX6+w5RURE5MwYa63TGb5TZmam/foBqlu3biUpKcmhRF/1xBNPEB4ezv333/+V2621WGvx82s4/fb5559n1qxZBAUFsWTJEkcyNKQ/OxERkbpmjFlrrc083X0NpxE0ETt37iQ5OZnJkyfTu3dvCgoKuP3228nMzKR37948+eSTpx47ZMgQNmzYgMvlonXr1jz00EOkpaUxaNAgioqKAHjssceYPn36qcc/9NBD9O/fn549e/Lpp58CUF5ezlVXXUVycjITJ04kMzOTDRs2nDbfjBkzmD59Ort376agoODU7f/5z3/o168faWlpjBkzBoDjx49z0003kZqaSmpqKm+//bZPvmciIiLNhYqXD2zbto17772XnJwc4uLi+M1vfsOaNWvIzs7mww8/JCcn53+uOXbsGMOHDyc7O5tBgwbx97///bTPba3l888/53e/+92pEvf8888TExNDTk4OP/vZz1i/fv1pr83Ly6O0tJSMjAwmTZrErFmzADh06BDTpk1j3rx5ZGdnM3PmTMC7khcZGcnGjRvJzs5m+PDhdfHtERERabYa7XESX/aLd7eQc7CsTp8zObYlj4/tfU7Xdu3alczML1YYZ8yYwSuvvILL5eLgwYPk5OSQnJz8lWtatGjBJZdcAkBGRgZZWVmnfe4JEyacekxeXh4AK1as4MEHHwQgLS2N3r1Pn3vmzJlcc801AFx77bXcdddd3HPPPaxcuZKRI0fSuXNnANq2bQvA4sWLT61yGWNo06bNWX8vRERE5AtNong1NGFhYad+nZubyx//+Ec+//xzWrduzQ033HDa4xSCgoJO/drf3x+Xy3Xa5w4ODv7Ox3yTGTNmcPjwYV577TUADh48yO7du8/qOUREROTcNYnida4rU/WhrKyMiIgIWrZsSUFBAQsXLuTiiy+u068xePBgZs2axdChQ9m0adNptzJzcnJwuVwcOHDg1G2PPvooM2fO5NZbb+Wee+5h7969dO7cmdLSUtq2bcvo0aP5y1/+wrPPPou1lqNHj2rVS0RE5HvQjJeP9evXj+TkZHr16sUPfvADBg8eXOdf40c/+hEHDhwgOTmZX/ziFyQnJ9OqVauvPGbGjBmMHz/+K7ddddVVzJgxg+joaP76178ybtw40tLSmDx5MgCPP/44hYWFpKSkkJ6efmr7c8qUKd84vC8iIiLfTMdJNAEulwuXy0VISAi5ubmMGTOG3NxcAgIa5oKm/uxERKQp+7bjJBrmv8xyVk6cOMGFF16Iy+XCWsvf/va3Blu6REREmjP969wEtG7dmrVr1zodQ0RERL6DZrxERERE6omKl4iIiEg9UfESERERqScqXiIiItIsWGtx+jQHnxYvY8w9xpjNxpgtxpgff+2+nxhjrDGmvS8z+MrIkSNZuHDhV26bPn0606ZN+9brwsPDAe+p8RMnTjztY0aMGMHXj8/4uunTp1NRUXHq95deeilHjx49k+hnJD09nWuvvbbOnk9ERMQJbo9ldV4pTy/YygXPLePzPaWO5vFZ8TLGpABTgf5AGnC5MaZb7X0dgTHAPl99fV+77rrrTn2Y9H/NnDmT66677oyuj42NZfbs2ef89b9evBYsWEDr1q3P+fm+bOvWrbjdbrKysigvL6+T5xQREakvldVuFm45xP1vZXPeU4uZ9MJKXv1kD/FtWuDvZxzN5ssVryRglbW2wlrrApYBE2rv+wPwANDwT2/9BhMnTuQ///kP1dXVAOTl5XHw4EGGDh166lytfv360adPH+bPn/8/1+fl5ZGSkgJAZWUl1157LUlJSYwfP57KyspTj5s2bRqZmZn07t2bxx9/HIA//elPHDx4kJEjRzJy5EgAEhISOHz4MAC///3vSUlJISUlhenTp5/6eklJSUydOpXevXszZsyYr3ydL5sxYwY33ngjY8aM+Ur2nTt3MmrUKNLS0ujXrx+7du0C4JlnnqFPnz6kpaXx0EMPfa/vq4iIyLkoPn6SN1fv47bXVpP+5CLueH0tC7ccYmj39vz5+r6s+9loXr91AJkJbZ0N+t/9zrr+gbd47QDaAaHASuB5YBzwx9rH5AHtv+u5MjIy7Nfl5OT8z2317bLLLrNvv/22tdbaX//61/YnP/mJtdbampoae+zYMWuttcXFxbZr167W4/FYa60NCwuz1lq7Z88e27t3b2uttc8995ydMmWKtdba7Oxs6+/vb1evXm2ttbakpMRaa63L5bLDhw+32dnZ1lprO3fubIuLi09l+e/v16xZY1NSUuyJEyfs8ePHbXJysl23bp3ds2eP9ff3t+vXr7fWWjtp0iT7+uuvn/Z19ejRw+7du9cuXLjQXn755adu79+/v507d6611trKykpbXl5uFyxYYAcNGmTLy8u/kvfbNIQ/OxERafxyC4/bv3680074f5/YhIfes50ffM+e/+sl9vH5m+2K3GJb7XI7kgtYY7+h0/jsAFVr7VZjzDPAIqAc2AAEA4/g3Wb8VsaY24HbATp16vTtD37/ITi06Xsm/pqYPnDJb771If/dbhw3bhwzZ87klVdeAbxl9pFHHmH58uX4+flx4MABCgsLiYmJOe3zLF++nLvvvhuA1NRUUlNTT903a9YsXnzxRVwuFwUFBeTk5Hzl/q9bsWIF48ePJywsDIAJEyaQlZXFFVdcQWJiIunp6QBkZGSQl5f3P9evWbOG9u3b06lTJ+Li4rjlllsoLS0lMDCQAwcOnPq8x5CQEAAWL17MlClTCA0NBaBtW4f/n4SIiDRZbo9l/b4jfJhTyIc5hew+7B2HSYlryT0Xdmd0cjTJHVpizLdsJ1oL33a/j/n05Hpr7SvAKwDGmKeBQuBKILv2mxIPrDPG9LfWHvratS8CL4L3sxp9mfNcjRs3jnvvvZd169ZRUVFBRkYGAG+88QbFxcWsXbuWwMBAEhISqKqqOuvn37NnD88++yyrV6+mTZs23Hzzzef0PP8VHBx86tf+/v6n3WqcMWMG27ZtIyEhAYCysjLmzJmjQXsREXFEZbWbFTsP82HOIZZsLaKkvJoAP8Ogru24eXACo5KiiW3d4puf4Pgh2JMFecu9P4+dDl1G1Ff8/+HT4mWMibLWFhljOuGd7xporf3jl+7PAzKttYe/1xf6jpUpXwkPD2fkyJHccsstXxmqP3bsGFFRUQQGBrJ06VL27t37rc8zbNgw/v3vf3PBBRewefNmNm7cCHhLT1hYGK1ataKwsJD333+fESNGABAREcHx48dp3/6rbwodOnQoN998Mw899BDWWubNm8frr79+Rq/H4/Ewa9YsNm3aRGxsLABLly7ll7/8JVOnTiU+Pp63336bK6+8kpMnT+J2uxk9ejRPPvkkkydPJjQ0lNLSUq16iYjI93L4xEk+2lrEopxCVuwspqrGQ0RwACN6RTE6OZoRPSNpGRJ4+ovLD0NeFuypLVolud7bg1tBwmAI+JaSVg98/VmNc4wx7YAa4IfW2ro776CBuO666xg/fvxX3uE4efJkxo4dS58+fcjMzKRXr17f+hzTpk1jypQpJCUlkZSUdGrlLC0tjb59+9KrVy86duzI4MGDT11z++23c/HFFxMbG8vSpUtP3d6vXz9uvvlm+vfvD8Btt91G3759T7ut+HVZWVnExcWdKl3gLYU5OTkUFBTw+uuvc8cdd/Dzn/+cwMBA3nrrLS6++GI2bNhAZmYmQUFBXHrppTz99NO88MILANx5553f/U0UEZFmb3fxiVNbiGv3HcFaiG0VwjWZHRmdHEP/xLYEBZzmPYEVpbD3k9pVrSwoyvHeHhQOnc+Hfj+AxKEQkwp+/vX7ok7DWIcPEjsTmZmZ9uvnWm3dupWkpCSHEsn3oT87ERHxeCzr9x+tLVuH2FXsnddK7tCS0cnRjE6Opnfsaea1qo7B3pXeFa285XBoM2C9K1mdBnpLVsIwiE0H/29YFfMxY8xaa23m6e7z9YqXiIiICABVNW5W5B7mw5xClmwr5PAJ77zWgC5tuXFgZ0YlRxPfJvSrF508Afs++2JGq2ADWA/4B0PH/jDyEUgYCnEZEBDkzAs7CypeIiIi4jOl5dUs2erdQszKPUxljZuI4ACG94ysndeKolWLL61M1VTC/lXekrVnORxcBx4X+AVCfCYMvd+7qhXfHwJDnHth50jFS0REROrUnsPlfJhziMU5RazZW4rHQodWIUzMiGd0cjQDu7T7Yl7LdRLyVnwxo5W/GtzVYPwhrh+cf7e3aHUcAEFhzr6wOtCoi5e19tvP6pAGpzHMFIqIyNnxeCwb8o+eGo7fWXQCgKQOLfm/kd0YnRxDSlztvJa7Bg6s/mJGa//n4KoC4+cdgB9wh3dGq/MgCI5w+JXVvUZbvEJCQigpKaFdu3YqX42EtZaSkpJTh6+KiEjjVVXj5tNd3nmtxVuLKD5+En8/w4DEtkwe0IlRSdF0bBsKbhcUZMMntTNa+z6DmtrPAY7uA5m3eGe0Op8PLermM4cbskZbvOLj48nPz6e4uNjpKHIWQkJCiI+PdzqGiIicgyPl1SzZVsTinEKW5xZTUe0mLMifET2952uN7BlFqxA/76fJbHuvtmithJNl3ieI7AXp13u3DjsPgbB2zr4gBzTa4hUYGEhiYqLTMURERJq0vSXlfJhTyKKcQtbkeee1YlqGMKFfHKOSohnUpQ3BpTtgz7vwTpZ3Xquq9tjOdt0g5araIx6GQniUsy+mAWi0xUtERETqnsdjyc4/yuLadyLuKPTOa/WKieCHI7sxOimKPsGFmLwsyM6C+SugosR7cevOkHS5d0YrcSi0jP2Wr9Q8qXiJiIg0c1U1blbuKmFRTiFLthZSVDuvdV5CG352WRKXxFYSe3Q17HkN3syCE4XeC1vGQ/cx3tWsxKHQupOzL6QRUPESERFpho5WVPPRtiI+zClk+Y5iymvntYb3jOSKBDfDAnIIPTAfVmdB2QHvReHRX5SshKHQtgvoDW5nRcVLRESkmdhXUsGHW70f0bM67whujyUqIpgbU4K4ouUuelSuJ2DvCsjd670gtD0kDIHE+7zbh+27q2h9TypeIiIiTZTHY9l04Nip87W2Fx4HoH9kDdN772OQ31baFa/C5OzyXhDS2lu0Bt4FicO870L0O80HU8s5U/ESERFpQk66vPNa3vO1CiksO0k7U8b10fn8vlsu3SvWE1S6A44DwS2952dl3uLdPoxOAT9/p19Ck6biJSIi0sgdq6jho+3eVa1l24vxry5jWNB2nm6zm4yQzbQu2wFHgfIw6DQQ+l3vXdGKSQN/VYH6pO+2iIhII7S/tOLUFuKWvAP0YysXhmzn0bBtxPrlYrBQHuL9jMPMq70zWnH9wD/wu59cfEbFS0REpBGw1rL5QBkf5hxi2Za9tCxeyyC/HH4evI2eQbvww401QZio8yBhgndFKz4TAoKdji5fouIlIiLSQFW7PKzcXcLSzXspyllBz6oNDPPbwt1+uwkIcmH9AjCxGZBwLyQOxXQcAIEtnI4t30LFS0REpAE5VlnDsq357Fy3jKD9n5Dh2czDfrkEmxpsgB/umDQCuv4QEodhOg6E4HCnI8tZUPESERFxWP7hY2xY9TEnti8l/uhqRpsdXGGq8RjDiXZJ+PecCl2GYzoPIiCkldNx5XtQ8RIREaln1u1i16ZPObj+Q1oc+ISkmi1cbqoAKArrSlnC9QT3GYVfwmBahrZ1OK3UJRUvERERX/N4qCnYxL51C6nJ/Zi4svV0o4JuQL5/R/bGj6V9yoVEp44mKqy902nFh1S8RERE6pq1ULyNyh0fc2TLEloWriLcU0ZXYK+NZn3LEQR2G0HPAZcQH9OJeKfzSr1R8RIREfm+rIWSXZC3nIrtSzF7V9CiupQWQIltz0d+GVTGnU+HvmM4L7UPnYN0OnxzpeIlIiJytqyFI3mQl4Xdk4Vr1zICKwoBKLNtWOlJJjesH+E9RzKgX18u69gGfz99uLSoeImIiJyZo/shLwv2ZGHzlmOO5QNwhFZ84k5ipedyymIGktKnH6N7xzA+Usc8yP9S8RIRETmd44dgTxbkLff+fGQPAOV+LfnEk0RWzSjWmhRiu6Uzunc09/aKJjJCp8TLt1PxEhERAThR7F3Rql3VoiQXgOqACDYFpPCBewgrXMkUtejKBb1jGJ0czSPdI2mheS05CypeIiLSPFWUwt5Pale1sqAoBwB3YBj7wtP5KHQo8450Iacqgc7tIxg9KJonk6Pp10nzWnLuVLxERKR5sBYObYJNb8HupXBoM2CxAS04FpnBmtg7mXU4gY/KYnGfCKBvx9Zc1j+G6clRdI0MxxiVLfn+VLxERKRpO3bAW7Y2vuld1fILwBU/gF1J/8cH5d15bV97SvdAcIAfQ7q151ejorkgKYqoiBCnk0sTpOIlIiJNz8njsPVdyJ4Je5YDFk9cJpvTfsaLJeks2lVDtdtDm9BALkiOZnRyNMN6tCc0SP8sim/pf2EiItI0uF3eLcSNb8LW98BViW2TQGHfu/l35UD+sc2fsl0uOrQy3HR+Z0Ynx5DRWfNaUr9UvEREpPGyFgqyvWVr02woL4KQ1lQkT2KR/wj+vLMdO1eWExxguCQliokZHRnUtZ3KljhGxUtERBqfY/mwcZa3cBVvA79A3N3GsKHNRbxQ0JUlq4/hsZDROYjfTOjCpakdaBkS6HRqERUvERFpJKrKYOs73rmtvBWAxcb358D5v+KfZf14c0s5xypr6NDqJNNGdOWqfvF00enx0sCoeImISMPldsGuj2DjTNi2AFyV0CaRE4Pu5107hFe3GnZ8dILggDIu6h3DxIx4Bndrr61EabBUvEREpGGxFgo2QPabsHk2lBdDiza4065jVcRoXt7TnmXLDuP2lNOvU2ueHt+Hy1I70KqFthKl4VPxEhGRhuHoftg0y1u4Dm8H/yBsj4vYF3cFrxV3Z+76Yo5W1BDdsozbh3VhYkY8XbWVKI2MipeIiDinqgxy5nuH5POyvLd1HMjxUb9j3snzeGPjcbavP05QwKFTW4lDtJUojZiKl4iI1C93jXduK3smbF8Aripo2wXXsIf5JPQCXt9uWLqgGLfnIOkdW/OrK1MYmxpLq1BtJUrjp+IlIiK+Zy0cXFc7tzUHKg5Di7bY9BvYHXs5r++PZP6KgxypKCYqIpipQ7swMSOOblERTicXqVMqXiIi4jtH93m3ETfOgsM7wD8Iel5CWY+rmH2sF7PWF7JtxXGC/Pczunc0EzPiGdqtPQH+fk4nF/EJFS8REalbVcdgy9vewrX3E+9tnc7HdeldLAsczIyNZXw8qwiXZydpHVvzyytTGJvagdahQc7mFqkHKl4iIvL9uWtg5+Laua33wX0S2nWDkY+xI/piZuT6MX/hQUrLc4mMCObWIYlclRFPj2htJUrzouIlIiLnxlo4sM57uOnmOVBRAqHtIOMmjnWfwJxD0cxed4Ccgn0E+fsxKjmKSRkdGdpdW4nSfKl4iYjI2Tmyt/ZzEmdCyU7wD4ael+DqczVLXam8tb6Qj1YU4fKUkhrfiifH9WZsaixtwrSVKKLiJSIi363yyBdzW/tWem/rPBjOv5vt7S5g1ubjvD37ACXlG2kfHswtQxK5ql88PWO0lSjyZSpeIiJyeq5q2Pmhd25rxwfgroZ23eGCxzjabTzz9vgz+5N8thzcSKC/YVSS912Jw3pEEqitRJHTUvESEZEvWAv5a2rntuZCZSmEtoeMKbj6XM3HZfHMXneAJQt3UOO2pMS15ImxyYxLj9NWosgZUPESEREo3VM7t/UmlO6CgBDoeSmkXsP28P7M3nCIea8d5PCJQtqHB3HToASuyognqUNLp5OLNCoqXiIizVXlEdgyz3ua/P7PvLclDIUh93Ik4RLe2XaC2Yvy2XRgJQF+hguTopiY0ZERPbWVKHKuVLxERJoTVzXkLvJuJe5Y6J3bat8TLvw5rt4TWV4Uwltr8lk853Nq3JbesS15fGwyV6TF0i482On0Io2eipeISFNnLeSv9g7Jb5nrXekKi4TMWyHtGnb4dWX2ugPM/X+5HD5xkrZhQdw4MIGJGfEkx2orUaQuqXiJiDRVpbu/NLe12zu31esySL2Wo7FDeHdTEbPn5pOdn0WAn2FkrygmZcQzomcUQQHaShTxBRUvEZGmpKLUO7e18U3YvwowkDAEhv4EV8/LydpXzezV+XyYs4xqt4deMRH87PJkxqXH0l5biSI+p+IlItLYuU5657U2vun92VMDkUkw6gnoM4ncqlbMXpfPvAVrKDru3UqcPLATEzPi6R3byun0Is2KipeISGNkrXdFK3umd4Wr6iiERUH/2yHtGo61TOKdTQXM/tdesvcfxd/PMLJnFBMz4rmgl7YSRZyi4iUi0piU7PKubG18E47kQUALSLocUq/FnTicrF1HmL00n0U5S6h2eegZHcFjlyUxLj2OyAhtJYo4TcVLRKShqyiFzXO8ZSt/NWAgcRgMfxCSxrLzmGH22nzmvbWMwrKTtA4N5Pr+/91KbIkxxulXICK1VLxERBoi10nv5yNmv+k9d8tTA1HJMOoX0GcSx4KieG/jQWa/spH1+7xbiSN6RPLE2HguSIoiOMDf6VcgIqeh4iUi0lBYC/s+8x5uumUeVB2D8GgYcAekXoM7KoVPdpUwe0E+C7ds5KTLQ4/ocB69NIlxfWOJighx+hWIyHfwafEyxtwDTAUM8JK1drox5nfAWKAa2AVMsdYe9WUOEZEGrWSXd0h+45twdC8EhkKvyyHtGkgcwa7SKuaszWfuuqUcKquiVYtArjmvI5MyOpISp61EkcbEZ8XLGJOCt3T1x1uyPjDGvAd8CDxsrXUZY54BHgYe9FUOEZEGqbzEe4p89kw4sAYw0GU4jHgYksZSZoP5z8YC3lq0inX7juJnYETPKH4+NpkLtZUo0mj5csUrCVhlra0AMMYsAyZYa3/7pcd8Bkz0YQYRkYajpso7t7Xxv3NbLojqDaOfhD6TcId34NNdh5k9L5cPNh/ipMtD96hwHr6kF+P7xhHVUluJIo2dL4vXZuApY0w7oBK4FFjztcfcArzpwwwiIs7yeGDfytq5rflw8hiEx8DAaZB6LcSksOdwOXNW5jN33VYOHquiZUgAV2d2ZGJGPKnxrbSVKNKE+Kx4WWu31m4lLgLKgQ2A+7/3G2MeBVzAG6e73hhzO3A7QKdOnXwVU0TENw7nercRN82Co/sgMAySxtbObQ3neLWHBZsKeGvep6zZewQ/A8N6RPLIZUmMSoomJFBbiSJNkbHW1s8XMuZpIN9a+/+MMTcDdwAX/ncr8ttkZmbaNWu+vlgmItLAlB/2nreVPRMOrgPjB11GeFe2ki7HExDKyt0lzF6bz/ubC6iq8dA1MoxJmR0Z3zeOaG0lijQJxpi11trM093n63c1Rllri4wxnYAJwEBjzMXAA8DwMyldIiINWk0lbH/fO7e1c7F3biu6D4z5FfSZBBEx7C0pZ/bH+cxdd4ADRyuJCAngqn7xTMyIJ71ja20lijQjvj7Ha07tjFcN8ENr7VFjzJ+BYODD2r9sPrPW3unjHCIidcfjgX2fele2cubDyTKI6AAD74K0ayG6NydOuliwsYDZa1fyeV4pfgaGdo/koUt6MTpZW4kizZVPi5e1duhpbuvmy68pIuIzxdtr57begmP7ISgckq6A1KshcRge/PhsTwmzP97A+5sPUVnjpktkGA9c3JMJfeOJaaWtRJHmTifXi4h8mxPFsHm2t3AVbPDObXW9AC58HHpdCkFh7CupYPaSXcxZm+/dSgwO4Mq+cUzKjKevthJF5EtUvEREvq6mErb9p3ZuawlYN8SkwkVPQ8pEiIim/NRW4kZW7SnFGBjSrT0PXNyTi3rHaCtRRE5LxUtEBLxzW3tXeD+UOmc+VB+HlnFw/o+8c1tRSXg8llV7Spn9fjbvby6gotpNYvswfnpRT8b3jSO2dQunX4WINHAqXiLSvBVt8x5uuvEtKMv3zm0lj4PUayBhCPj5s7+0gjmLdzBnXT77SysJDw5gXHosEzPi6depjbYSReSMqXiJSPNjLWTPgFUvQEE2GH/v3NboX0DPSyEolIpqFwvWFzB77X4+2+3dShzctT0/Ge3dSmwRpK1EETl7Kl4i0rxUlMI7P4Jt73nP27ro19BnIoRHYa3l8z2lzF6by4JNBZRXu0loF8r9Y3owvl88cdpKFJHvScVLRJqPPcth7h1QXuw94HTgD8HPj/wjFcxZnMucdfnsK60gPDiAy1NjmZgZT2ZnbSWKSN1R8RKRps9dA0ufhhV/gHZd4brFVLbvw/sbDjJ7bT6f7ioBYHC3dtw7ujsX9Y4hNEh/PYpI3dPfLCLStJXsgjm3eT87sd8P4HT7ReEAACAASURBVOLfsGJvJff/42MOlVXRqW0o943uwYR+ccS3CXU6rYg0cSpeItI0Wes99HTB/eDnD5Neo6rHWJ75YBuvfpJH18gw/n3bAAZ1baetRBGpNypeItL0VB2D9+7znjjfeTBMeJHNJyK49/kV5Bad4ObzE3jw4l56Z6KI1DsVLxFpWvatgrm3wbEDcMFjuM+/lxey8pi+eCNtw4L45y39GdYj0umUItJMqXiJSNPgdkHWc7DsGWgVD7csZF9ob+576XPW7D3CZX068NT4FFqHBjmdVESaMRUvEWn8ju6HuVNh30roczX2smd5a1MZv3h3OX7G8Idr0rgyPU6zXCLiOBUvEWnctsyDd+/xftbi+Bcp6XolD8/axKKcQgZ2actzV6fr4FMRaTBUvESkcTp5Aj54ENb/C+Iy4aqX+KgojAemZ1FWWcMjl/bitiFd8PPTKpeINBwqXiLS+Bxc7z2bq2QXDL2fivPv56kPdvLGqq30iong9Vv7k9ShpdMpRUT+h4qXiDQeHg+s/DMseRLCo+Dm91jv15v7/rKKvJJybh/WhZ+M6UFwgI6JEJGGScVLRBqH44dg3h2w+2NIGkvNZX/kzytL+PPSlcS0DOHftw1kUNd2TqcUEflWKl4i0vBtfx/m/xBqKmHsn9jdcQL3/iOb7PxjTOgbxxPjetMyJNDplCIi30nFS0QarppKWPQzWP0SxPTBXvUK/9oVwlPPryA4wJ+/XN+Py1I7OJ1SROSMqXiJSMNUuAVm3wrFW2HQ/1HU/wEeeHs7H2/fydDu7fndxDRiWoU4nVJE5KyoeIlIw2ItfP4SLHoMQlrBDXP4oCqFh59fRUW1myfGJvODQQk6JkJEGiUVLxFpOMoPe2e5dnwA3cdw4uI/8ouPinlr7VpS4loy/Zp0ukVFOJ1SROScqXiJSMOw6yOYdydUHoGLn2F19CTufTmbg0cr+b+R3bj7wu4EBfg5nVJE5HtR8RIRZ7mq4aMn4dPnIbIXNdfN5vebgnhh/md0bBPKW3cOIqNzW6dTiojUCRUvEXHO4VyYcysUZEPmreSmP8Q9s7eTU1DGted15LHLkwkP1l9TItJ06G80Eal/1sL61+H9ByEgGM81b/BqSW+eeWEtEcEBvHhjBmN6xzidUkSkzql4iUj9qjwC794DOfMhcRiFo/7Ife8X8cnOHC7sFcVvrkolMiLY6ZQiIj6h4iUi9WfvpzBnKpw4BKN+wTvhE3nspS3UuC1Pj+/Ddf07YoyOiRCRpkvFS0R8z+2CZc9A1rPQJoETkxfw6Oog5m/Ipm+n1vzh6nQS2oc5nVJExOdUvETEt47keVe58j+H9Bv4rOcD3DtrJ0XHT3Lf6B7cNaIrAf46JkJEmgcVLxHxnY1vwX/uAwzVV77EM/kpvPLaZrpEhjF32vmkdWztdEIRkXql4iUida+qDBb8FDbOhI4D2THkOf5vQQk7Cvfwg0GdefiSJFoE+TudUkSk3ql4iUjdyl8Lc26Bo/vwDH+IF+0EnvvnLlqHBvGPKecxomeU0wlFRByj4iUidcPjhk+mw9KnISKWwqvm8aMVwXyet5OLe8fw9IQ+tA0LcjqliIijVLxE5Ps7dgDm3QF5WdjeE3in40959K39wEmenZTGVf3idEyEiAgqXiLyfeW8A+/8CNw1nLjkeX66I5n3395D/4S2PHd1Gh3bhjqdUESkwVDxEpFzU10OCx+Btf+A2L6s6vdb/m9hGUcrinjokl5MHdoFfz+tcomIfJmKl4icvYKN3g+3PpxLzaB7eKp8PP+YfZAe0eG8NqU/ybEtnU4oItIgqXiJyJnzeGDVX2HxE9CiLbsu+RdTs8LZffggtw1J5P6LehISqGMiRES+iYqXiJyZ44Xw9jTYtQRPj0t4qc1P+O38YqIi3Pz7tgGc36290wlFRBo8FS8R+W47FsH8u+DkcQ4P/zW3bUllw8YixqXH8uQVKbQKDXQ6oYhIo6DiJSLfrKbKu6246q/YqGQW9H2R+z+qJtC/nD9d15cr0mKdTigi0qioeInI6RVtgzm3QeEmKvpO5d6S8SxcfJTB3drx7KQ0OrRq4XRCEZFGR8VLRL7KWlj7KnzwCASFsW7I35i6sh3HT5bx88uTufn8BPx0TISIyDlR8RKRL1SUeg9D3fYersSR/DroHl5ZXEFyhxBmXptO9+gIpxOKiDRqKl4i4rV7mfdjf8oPs7//Y9ywqR/7jlZw14iu/HhUD4IC/JxOKCLS6Kl4iTR37hpY+hSsmI5t141/Jj7DL7ICiG1tmHXHIM5LaOt0QhGRJkPFS6Q5K9nlHaA/uI5jyZO55dB41n5ezaSMeH4+NpmIEB0TISJSl1S8RJojayF7Biz4KdYvgI9Tn+XOdfGEBcMLN2RwcUqM0wlFRJokFS+R5qbyKPznPtg8h+r48/mp+y7mf+7HyJ7teGZiKlERIU4nFBFpslS8RJqTfau8W4tlB9iW/GOuyxlIldvwqyuTmDygE8bomAgREV9S8RJpDtwuyHoOlv0Gd6tO/Knzn/njulakdWzJH65Oo0tkuNMJRUSaBRUvkabu6D6YezvsW0lxl/Fcl38Ve4r8+fGobvxwZDcC/XVMhIhIfVHxEmnKNs+Fd3+MtW7md3mCe7f2IKFdGLPvTKNvpzZOpxMRaXZUvESaopMn4P0HYcO/qIzqy7TKu/g4J4zJAzrx6GVJhAbpP30RESfob1+RpubAOphzG7Z0N+sTbuOG3BGEtmjB32/uwwW9op1OJyLSrKl4iTQVHg98+if46Je4QqN4qt0zvLotnjHJ0fx6Qh/ahQc7nVBEpNlT8RJpCsoKvJ+zuGcZBzqM5uqD13G0LIzfTuzNpIx4HRMhItJAqHiJNHbbFsD8H2JdVfw76n4e3dOXzM5tmXlNOh3bhjqdTkREvkTFS6SxqqmERY/B6pc53iaZW47fwYYDUTxwcQ/uGNYVfz+tcomINDQ+LV7GmHuAqYABXrLWTjfGtAXeBBKAPOBqa+0RX+YQaXIKt8DsW6F4K59EXceUfRfTKaoN86akkxLXyul0IiLyDXx2cqIxJgVv6eoPpAGXG2O6AQ8BS6y13YEltb8XkTNhLax6EV4cSc2JwzzQ4gkm7xvL5MHdee9HQ1S6REQaOF+ueCUBq6y1FQDGmGXABGAcMKL2Ma8BHwMP+jCHSNNQfhjevgtyF5LXdghXH7oBv/Ao/nVrGkO6t3c6nYiInAFfFq/NwFPGmHZAJXApsAaIttYW1D7mEKCDhUS+y84l8PY0bOVRXomYxq8ODuHy1Fh+dWUKrUODnE4nIiJnyGfFy1q71RjzDLAIKAc2AO6vPcYaY+zprjfG3A7cDtCpUydfxRRp2FwnYcmTsPLPHAvvxk0197PreGf+eG0K49LjnE4nIiJnyaefjmutfcVam2GtHQYcAXYAhcaYDgC1Pxd9w7UvWmszrbWZkZGRvowp0jAdzoWXR8HKP/NRxDj6H36MFvGpLPzxMJUuEZFGytfvaoyy1hYZYzrhne8aCCQCNwG/qf15vi8ziDQ61sK6f8IHD1FtgnjI70HeO9KXBy7ryS2DE/HTMREiIo2Wr8/xmlM741UD/NBae9QY8xtgljHmVmAvcLWPM4g0HhWl8O49sPUddoZncP3hW2gb04l3r+1Lz5gIp9OJiMj35NPiZa0deprbSoALffl1RRqlvBUw93bs8UJeCLyJ35WMZurwbtw3ugfBAf5OpxMRkTqgk+tFnOaugWXPYJc/y9GQeG4++QSHg3szY2oaA7q0czqdiIjUIRUvESeV7oG5UyF/NUuCR3P30eu4pF83Xr8imZYhgU6nExGROqbiJeKUjbOw791HjQce9NzN0pqhPDe5D5f06eB0MhER8REVL5H6VlUGC34KG2eSG9ybKSdup1uPZBZNTCWqZYjT6URExIdUvETqU/4amHMr9sg+/mqu5i8V43hwXAo3DuyMMTomQkSkqVPxEqkPHjes+AN26dMcCYhk6smfURPXn/lXp9MtKtzpdCIiUk9UvER87Vg+zL0D9q5gsd8Q7j9xEzeNTONHF3Yn0N+nHx4hIiINjIqXiC/lvIN950dUV1fzSM2drGl1Ea/e3Jd+ndo4nUxERByg4iXiC9Xl8MHDsO41cv27M7VyGuef158FlyURFqz/7EREmiv9CyBS1wqysbNvhZKdvOgZx6v+1/KrGzMYlRztdDIREXGYipdIXfF4YNVfsYuf4IiN4IfVjxDW8wLeu6oP7cODnU4nIiINgIqXSF04XghvT4NdS/iI83jMcwf3XDmAa87rqGMiRETkFBUvke9rxyI8b0/DVXmcJ2puZVvsBGZe25fO7cKcTiYiIg2MipfIuaqpgsWPw6oX2GU686OTD3L5qJE8ObwrATomQkRETkPFS+RcFG3FM/sW/IpyeMV1CW+1voXf3TqAPvGtnE4mIiINmIqXyNmwFta8gueDRzjmCeHe6gdIGHgl8y7uRYsgf6fTiYhIA6fiJXKmykuw83+I2fE+WZ40fh10Nw/fPJzhPSKdTiYiIo2EipfImdi9DNec27Hlh/l1zY0UJd3MjPGptAkLcjqZiIg0IipeIt/GVY1d+hR88kf22g48yNNcf9VljO8bp2MiRETkrKl4iXyTkl24Zt1CQOEG/u26gA/i7mb6tQOIbxPqdDIREWmkVLxEvs5ayJ6B672fUO7y41HXfaSOuYF/DOmCn59WuURE5NypeIl8WeVRXO/eS0DOXFa7k/lLm5/y6HWjSOrQ0ulkIiLSBKh4ifzXvs+onnUr/icO8lvXNbgH3c3LY5IICdQxESIiUjdUvETcLtzLfodZ/lsKbHt+GfQ0t954NYO6tnM6mYiINDEqXtK8HTtA1cybCClYzRz3UFYnPcxz4/vTqkWg08lERKQJUvGSZu3wjDtpUbCJh8zdDJk0jd+kxjodSUREmjAVL2m27JG9tD2UxRsh1/Djux4lplWI05FERKSJ83M6gIhTDnz0ElhoP+RWlS4REakXKl7SPLldhG+dwScmnZEDM5xOIyIizYSKlzRLxze/T2vXYQ50uVrHRYiISL3RjJc0S6VZL1JpW9Nv1LVORxERkWZEK17S7Nhj+cQfXkFW+EX0iG3rdBwREWlGVLyk2dn/0Uv44yFswBSno4iISDOj4iXNi8dN+JYZrCSVEYP6O51GRESaGRUvaVbKNi+krauQfA3Vi4iIAzRcL81KSdaLVNuWpI+63ukoIiLSDGnFS5oNW1ZAp+JlfBI+hu6x+gBsERGpfype0mzs++hl/PHQQkP1IiLiEBUvaR48HsK2vMFqejNs0CCn04iISDOl4iXNQtnWxbSvKWBf4iQN1YuIiGM0XC/NQvHHL+Ky4aSNusHpKCIi0oxpxUuaPHuiiE7FS/k0bDTd4iKdjiMiIs2Yipc0eXuXvEwgLkIGaqheREScpeIlTZu1tNj8BuvoxZBBQ5xOIyIizZyKlzRpx7YuJbomn70JOqleREScp+F6adKKPv4b2FBSx9zodBQRERGteEnT5TlRQueixXwaNoqusVFOxxEREVHxkqZr79JXCMJF8IBbnI4iIiICqHhJU2UtIZv+RTbdOf/8YU6nERERAVS8pIk6un05Har3aqheREQaFA3XS5NUuPRv+NkW9B59k9NRRERETtGKlzQ5nvIjJBR+yMqwC+kaF+10HBERkVNUvKTJyVv6d4KpJui8m52OIiIi8hUqXtK0WEvwxn+xha4MGnKB02lERES+QsVLmpQjOz4lrno3eZ0naqheREQaHA3XS5NyaOnfCLLB9Bqts7tERKTh0YqXNBmeymMkHPqAz0JH0jU+xuk4IiIi/0PFS5qM3R+9RgtOEtB/itNRRERETkvFS5oGawne+BrbSGDA4FFOpxERETktFS9pEo7s/JyOJ3eyp9NEQoI0uigiIg2T/oWSJuHgRy8QYoPoNeZWp6OIiIh8I5+ueBlj7jXGbDHGbDbGzDDGhBhjLjTGrDPGbDDGrDDGdPNlBmn6PFXHSSxYwKrQ4STGxzodR0RE5Bv5rHgZY+KAu4FMa20K4A9cC/wVmGytTQf+DTzmqwzSPOxc+hqhVOF3nobqRUSkYfP1jFcA0MIYEwCEAgcBC7Ssvb9V7W0i5yxowz/ZSUcGDL3I6SgiIiLfymczXtbaA8aYZ4F9QCWwyFq7yBhzG7DAGFMJlAEDfZVBmr7SXWtIOLmdhZ3upVugRhZFRKRh8+VWYxtgHJAIxAJhxpgbgHuBS6218cCrwO+/4frbjTFrjDFriouLfRVTGrkDS17gpA2k52gN1YuISMPny63GUcAea22xtbYGmAsMBtKstatqH/MmcP7pLrbWvmitzbTWZkZGRvowpjRWnpPlJB78D5+HDiWhY0en44iIiHwnXxavfcBAY0yoMcYAFwI5QCtjTI/ax4wGtvowgzRhuUtfJ5wKTKaG6kVEpHHw5YzXKmPMbGAd4ALWAy8C+cAcY4wHOALo04zlnARseI09xHHesEudjiIiInJGfDqNbK19HHj8azfPq/0hcs5K9qyna1UOizveTaKG6kVEpJHQRwZJo5S/+AVO2gC6jZ7qdBQREZEzpuIljY7nZAWJB95jbegQEjp1cjqOiIjIGVPxkkZnx8dv0JITkHGT01FERETOioqXNDp+6//JPmLIGD7W6SgiIiJnRcVLGpWSvE30qNrIzvgJBAcGOh1HRETkrKh4SaOyb/EL1Fh/uo65w+koIiIiZ03FSxoNT3UVifnzWdfifDp3SnA6joiIyFlT8ZJGY9vHM2jNcWw/DdWLiEjjpOIljYbf+tc4QBT9Ro53OoqIiMg5UfGSRqFk71Z6Va5nZ/x4gnRSvYiINFIqXtIo5C3+Ky7rR+IoDdWLiEjjpeIlDZ6n5iSJ+99mQ4sBdEro6nQcERGRc6biJQ1ezrI3acsxPBqqFxGRRk7FSxo8s/Y1DtGe9BETnY4iIiLyvah4SYNWvH87SRVryY27kqAgnVQvIiKNm4qXNGh5i17AAgkaqhcRkSZAxUsaLLerhsT989jY4jw6JvZwOo6IiMj3puIlDVbOsrdozxFc6Tc7HUVERKROqHhJw7XmHxTRlrQLJjmdREREpE6cUfEyxnQ1xgTX/nqEMeZuY0xr30aT5qw4fyfJFZ+TGzuOoKAgp+OIiIjUiTNd8ZoDuI0x3YAXgY7Av32WSpq93YtewACdR93pdBQREZE6c6bFy2OtdQHjgeettT8FOvguljRnbpeLhH1z2RySQXyXXk7HERERqTNnWrxqjDHXATcB79XepkOVxCe2LJ9DNCXUpN/odBQREZE6dabFawowCHjKWrvHGJMIvO67WNKcudf8g8O0ps8F1zkdRUREpE4FnMmDrLU5wN0Axpg2QIS19hlfBpPmqejAHvqUf8bq2BsYFBzsdBwREZE6dabvavzYGNPSGNMWWAe8ZIz5vW+jSXO0a9ELBBgPnXRSvYiINEFnutXYylpbBkwA/mmtHQCM8l0saY7cbjcJe+ewObgvcV1TnI4jIiJS5860eAUYYzoAV/PFcL1Indq8/G06UExNmobqRUSkaTrT4vUksBDYZa1dbYzpAuT6LpY0R641r3KElqRceL3TUURERHziTIfr3wLe+tLvdwNX+SqUND9FB/eReuJT1sdeQ//gFk7HERER8YkzHa6PN8bMM8YU1f6YY4yJ93U4aT52LPobgcZN/IXTnI4iIiLiM2e61fgq8A4QW/vj3drbRL43t9tNQt5stgX3IbZbqtNxREREfOZMi1ektfZVa62r9sc/gEgf5pJmZOOKd4nnECdTf+B0FBEREZ860+JVYoy5wRjjX/vjBqDEl8Gk+aj5/FWOEU7yqBucjiIiIuJTZ1q8bsF7lMQhoACYCNzso0zSjBQW7Cf9RBa5MZcRGBzqdBwRERGfOqPiZa3da629wlobaa2NstZeid7VKHVg+8KXCTJu4i7QUL2IiDR9Z7ridTr31VkKaZbcbg+d977FjqBkOvTo63QcERERn/s+xcvUWQpplrI/WUBne4CqVJ1ULyIizcP3KV62zlJIs1S96u8cJ5SkUXo3o4iINA/fenK9MeY4py9YBtDx4nLODhUepO+J5WyJGUe/kHCn44iIiNSLby1e1tqI+goizcu2D15ihKkh9oI7nY4iIiJSb77PVqPIOXG53HTMe4tdQb2I6Xme03FERETqjYqX1LsNKxfR1e6nss9kp6OIiIjUKxUvqXcnP/s75YTQc9TNTkcRERGpVypeUq8KDh2i34mPyY2+hMAWLZ2OIyIiUq9UvKRebV30Mi1MNTEj73A6ioiISL1T8ZJ643K56bhnFnsCuxPTa5DTcUREROqdipfUm3Url9Dd7qUiRUP1IiLSPKl4Sb2pWvV3Kgmmx+gpTkcRERFxhIqX1IuComIyjn/EzsgxBIa2djqOiIiII1S8pF5sWfgKYeYkUSN1Ur2IiDRfKl7icy63h7jds9gfmEh00mCn44iIiDhGxUt8bs1nH5Nkd3Ei5QYwxuk4IiIijlHxEp+r/OwVqgii26hbnY4iIiLiKBUv8amDRYfJLFvCrshRBIa1cTqOiIiIo1S8xKc2LfoHEaaSyBE6qV5ERETFS3zG5fYQt2smBwI7EZU83Ok4IiIijlPxEp9ZvSqLFJvL8eTJGqoXERFBxUt8qHzlK1QTQLdRtzkdRUREpEFQ8RKfOFhcwnllH7Kr/YUERLR3Oo6IiEiDoOIlPpG98DVamQraD7/d6SgiIiINhk+LlzHmXmPMFmPMZmPMDGNMiPF6yhizwxiz1Rhzty8zSP1zuT3E7nqTQwFxRKZc6HQcERGRBiPAV09sjIkD7gaSrbWVxphZwLWAAToCvay1HmNMlK8yiDNWff4pg+02tic/QIyG6kVERE7xWfH60vO3MMbUAKHAQeBXwPXWWg+AtbbIxxmknp349BVqCKDrqKlORxEREWlQfLbVaK09ADwL7AMKgGPW2kVAV+AaY8waY8z7xpjuvsog9S+/uJT+ZYvY3W4EAS21mCkiIvJlPitexpg2wDggEYgFwowxNwDBQJW1NhN4Cfj7N1x/e205W1NcXOyrmFLHshe9ThtzgnYaqhcREfkfvhyuHwXssdYWW2trgLnA+UB+7a8B5gGpp7vYWvuitTbTWpsZGRnpw5hSV1xuD9E736QooAPtU0Y7HUdERKTB8WXx2gcMNMaEGmMMcCGwFXgbGFn7mOHADh9mkHq0cvUqMu0WjiVdB346qUREROTrfDZcb61dZYyZDawDXPz/9u48SMr7vvP4+9s9wy1OoRNJIAlZt7gESLKs0yvbcoSQbVn3ETteZyux167aJLt/rCrZ/WezriTlZLdS2thZZwNYjiwLhCVZIJC4wQIxA+gyOrglBgHDIY6Z6d/+MW0bY44Bpvvp4/2qmlJP9zPqz4+H6vnwPN9+Gl4HngR6A1Mi4jvAHsDLmteI3Yt+SDt5RtzuaUZJko6kpO9qTCk9ATxx2N0HgDtL+bwqvw1bdzCh9QXeH3IjIwecnXUcSZIqkueD1C1Wzp7CkNjNoM94tEuSpKOxeOmUtXUUOONXT7EtfyanX/25rONIklSxLF46ZYtfe40JqZnWS++DXD7rOJIkVSyLl05Z68If0EGOCxyqlyTpmCxeOiUbWlqZ2PoCHwy6gYZBw7KOI0lSRbN46ZQsnzWNodHKwBv9XEZJko7H4qWT1jlU/2O2509nyDVeIUSSpOOxeOmkLXztdSYWVrLzU1+FfEkvCSdJUk2weOmk7Vz4Awg4/zaH6iVJ6gqLl07Khm27mNj6POsGXUfDkOFZx5EkqSpYvHRSfjnrKc6KHQz4tEP1kiR1lcVLJ6yto8DQd6axMz+YwaP+IOs4kiRVDYuXTtiC5U1cX1jBjkvuhXxj1nEkSaoaFi+dsO0Lf0g+Euff/s2so0iSVFUsXjohG7btZuLO51k3cAL5ISOyjiNJUlWxeOmELJ39b5wb2+h/w9eyjiJJUtWxeKnL2joKDHn7x+zKDWTQ6MlZx5EkqepYvNRl85ev4sbCL9lxyZehoUfWcSRJqjoWL3XZtoX/TEMUGHabQ/WSJJ0Mi5e6ZP22PUzcMZMNA8aRHzoy6ziSJFUli5e6ZPHsn3J+roV+1ztUL0nSybJ46bjaOgoMfnsau3P9GTTmnqzjSJJUtSxeOq55K97gpsIydo78EjT2yjqOJElVy+Kl49q64J/pER2cc6tD9ZIknQqLl45p/ba9TNgxk039R5E/89Ks40iSVNUsXjqmBS8/y4W5D+l73dezjiJJUtWzeOmoDrYXGPzWVPbm+jFw3JezjiNJUtWzeOmoXl35FrcUlrDjosnQ2DvrOJIkVT2Ll47qo/n/l57RztkO1UuS1C0sXjqiddv2MHHHc2w57SryZ1+ZdRxJkmqCxUtHNH/OTC7ObabPdV6pXpKk7mLx0u852F5g0JtT+CT6MmDcvVnHkSSpZli89HvmrnyH2wqL2XHRJOjRN+s4kiTVDIuXfs+HC35Er2jjrFv+fdZRJEmqKRYv/Y4PWvYwfvtzfNTvcvLnjso6jiRJNcXipd8xb+4LXJbbQO+Jj2cdRZKkmmPx0m8cbC8w8M0p7I9e9L/2/qzjSJJUcyxe+o25TWu5vbCI7RdOgp6nZR1HkqSa05B1AFWOTfN/RJ84QM+bHaqXJKkUPOIloHOofsL252jpewn5YWOyjiNJUk2yeAmAua+8xBW5dfSc8IcQkXUcSZJqksVLHGwv0P+NqRyMnvQf/0DWcSRJqlkWL/Fy87vcUZjPx8PvhF4Dso4jSVLNcrhebJj3r/SL/fS5+ZtZR5EkqaZ5xKvOvb9tL+O3z+DjPheRO3981nEkSappFq86N2fubEbl3qPH+McdqpckqcQsXnXsQHsHp70xhYPRg9PGP5h1HEmSap7Fq4693PQ+nyvMY8cFUen11AAAGGRJREFUn4c+g7OOI0lSzXO4vo6tnz+V/rGPfjd5pXpJksrBI1516v1te7l2+wy29xlObvj1WceRJKkuWLzq1OxX5jI29ysar3WoXpKkcrF41aED7R30XTOFNho5bfzDWceRJKluWLzq0KzmdXyh8Co7LrgD+g7JOo4kSXXD4lWH1s2bxsDYy+mf+aOso0iSVFcsXnXmvZY9jNs+g529zyM34jNZx5Ekqa5YvOrMrHnzmZB7i4Zxj0HO3S9JUjn5m7eOHGjvoM/qKXSQp9+ER7KOI0lS3bF41ZFZzRu4szCX7ed/FvqdkXUcSZLqjsWrjrw7fxqDYw9DbvxG1lEkSapLFq868V7LHsZte45dvc4hd9EtWceRJKkuWbzqxIvzFnJDfg25sY84VC9JUkb8DVwH9rd10HvVFDrI0W/CY1nHkSSpbpW0eEXEdyJiTUSsjohpEdHrkMe+HxF7Svn86vTSqvV8Mc1lx7Bbof/ZWceRJKlulax4RcS5wLeAcSmlK4E8cF/xsXHAoFI9t37X2nn/xtDYxWCH6iVJylSpTzU2AL0jogHoA2yOiDzwP4E/K/FzC3i3ZQ/jPp7B7p5nkht5e9ZxJEmqayUrXimlTcD3gPXAFqA1pfQS8CfAjJTSllI9t37rhXlL+Ex+Fbkxj0Aun3UcSZLqWilPNQ4CJgEjgHOAvhHxCPAV4O+78PPfiIjXIuK1lpaWUsWsafvbOui5agoFcvSd+FjWcSRJqnulPNV4O/B+SqklpdQGPAP8JXAxsDYiPgD6RMTaI/1wSunJlNK4lNK4oUOHljBm7Xpp1UbuSnPYec5NMGBY1nEkSap7pSxe64GJEdEnIgK4DfiblNJZKaXhKaXhwCcppYtLmKGuvT3/p5wZOxl44x9lHUWSJFHaGa+lwNPACmBV8bmeLNXz6Xet3bqHsdums6fHUHKX3JF1HEmSRInf1ZhSeiKldGlK6cqU0sMppQOHPd6vlM9fz56fv4ybck3kxjwM+Yas40iSJLxyfU3a39ZBj1VTiYA+XqlekqSKYfGqQS+u2sRd6WVaz/40DLog6ziSJKnI4lWD3pz/DOfEdgZ82qF6SZIqicWrxqzduptx22bwSY8h5C79QtZxJEnSISxeNWbmguXcknsdRj0I+cas40iSpEP4drcasr+tg4ZV02iIAg0TH886jiRJOoxHvGrIi6s2c3dhNjvPuh4GX5h1HEmSdBiLVw1ZM/9ZhsU2+t/w9ayjSJKkI7B41Yi1W3czetsM9jUOJHfZF7OOI0mSjsDiVSOmL1jJZ3PLSdfcDw09s44jSZKOwOH6GrC/rYOG5qk0RgeNE7+WdRxJknQUHvGqAS+s2sSkwsu0njEBTh+ZdRxJknQUFq8a0Dz/OYbnPqL/DR7tkiSpklm8qtyvPtrNmG0z2N/Qn7h8UtZxJEnSMVi8qtyzC5u4I/dL0tX3QWOvrONIkqRjcLi+iu1v6yDX/BQ9ogMm/mHWcSRJ0nF4xKuKPd+8mbsLs9g1dAyccVnWcSRJ0nFYvKrYyoXPc1FuC6dd75XqJUmqBhavKvXOR7sZ3TKdAw39iCsmZx1HkiR1gcWrSv1s0Wq+kFtGuupe6NEn6ziSJKkLHK6vQvvbOqDpx/SMNpjgUL0kSdXCI15V6PnmzUwuzGL3kGvgrKuyjiNJkrrI4lWFli/4BZfkNtHveq9UL0lSNbF4VZl3PtrN6G3TOZjvQ1z5pazjSJKkE2DxqjLPLFzDnbklFK78CvTsl3UcSZJ0AhyuryL72zroaH6K3nEQJjyedRxJknSCPOJVRX7etJl7CrPZM/hKOGd01nEkSdIJsnhVkWWLZnNZbj19r/MSEpIkVSOLV5V4+8PdjN76LG25XsRVX8k6jiRJOgkWryrxzOI3+YP8YgpX3AO9+mcdR5IknQSH66vAvoMdtDX9hL5xACZ47S5JkqqVR7yqwM9XbeHuwmz2DroUzh2bdRxJknSSLF5VYMnCOVyde58+E/8QIrKOI0mSTpLFq8K9/eFuRm19lvZcT+Lqr2YdR5IknQKLV4X76eK3uDu/iI7LJkHvgVnHkSRJp8Dh+gq272AH+5qepl/sc6hekqQa4BGvCjazufNK9fsGXAznTcg6jiRJOkUWrwq2aNGrjM6tpZdD9ZIk1QSLV4V668NdXLN1Oh3RSFxzf9ZxJElSN7B4VainF7/DPfkFtF96F/QZnHUcSZLUDRyur0D7DnbwSdMz9I9PYIIfiC1JUq3wiFcFmtm8mcmFWezrPwIuuCHrOJIkqZtYvCrQvEULuDb3Dr0mOFQvSVItsXhVmDe37GLU1ul0RAMx6oGs40iSpG5k8aow/7bkV9yTn0/HJXdC39OzjiNJkrqRw/UVZN/BDvaufJZBsQfGP551HEmS1M084lVBnmvezOQ0i/39zocRN2UdR5IkdTOLVwV5ZeEiJubepOf4xyDnrpEkqdb4271CvLF5F9e0TKcQDcToh7KOI0mSSsDiVSF+smQtX8nPo33kHXDamVnHkSRJJeBwfQX45GA7e5qmMzh2w7VeqV6SpFrlEa8KMLNpC5MKsznQ91y46Jas40iSpBKxeFWAOYuXcmN+NT2ufRRy+azjSJKkErF4ZeyNzbu4aut0CuSIMQ9nHUeSJJWQxStjTy19j3vzr9Jx0Weh/zlZx5EkSSXkcH2GPjnYTuvKGQyNVhjvUL0kSbXOI14Zmtm0hbsLsznY5yy4+Pas40iSpBKzeGVo1uJf8pl8M43jHoG8Bx8lSap1Fq+MrNncypVbZxBAjHkk6ziSJKkMLF4ZeWrpe9yXf4X2C2+DgedlHUeSJJWB57cysPdAOx+vfJ4zYwdc+3jWcSRJUpmU9IhXRHwnItZExOqImBYRvSJiSkS8XbzvhxHRWMoMlWhm82YmF2bR1nsoXHJH1nEkSVKZlKx4RcS5wLeAcSmlK4E8cB8wBbgUuAroDXy9VBkq1S8WreCWfBMNYx+BfN31TkmS6lapTzU2AL0jog3oA2xOKb306wcjYhkwrMQZKsrqTa1cufU58o0F8Er1kiTVlZId8UopbQK+B6wHtgCth5WuRuBh4MVSZahEP176Pvc1zKVt+M0weETWcSRJUhmV8lTjIGASMAI4B+gbEQ8dssn/BuallOYf5ee/ERGvRcRrLS0tpYpZVnsPtLOt6UXOiY9pvPaxrONIkqQyK+Vw/e3A+ymllpRSG/AMcD1ARDwBDAW+e7QfTik9mVIal1IaN3To0BLGLJ/nmopD9b2GwKfuzDqOJEkqs1IWr/XAxIjoExEB3Aa8GRFfB+4A7k8pFUr4/BXnxSUruT2/goYxD0FDj6zjSJKkMivZcH1KaWlEPA2sANqB14Engb3AOmBxZx/jmZTSX5UqR6VYvamVyz8qDtWPfTTrOJIkKQMlfVdjSukJ4IlyPmelmrb0A77Z8Art53+ahiEXZR1HkiRlwI8MKoO9B9r5aOUvOC+20uCV6iVJqlsWrzKY0bSZyWkW7T0HwaVfzDqOJEnKiMWrDF5Y0sQd+eXkR98Pjb2yjiNJkjJi8Sqx1Ztaueyjn9NABzH2sazjSJKkDFm8Smzq0nXc3zCX9mETYeinso4jSZIyZPEqoT0H2tmy8iWGx4cO1UuSJItXKT3XtJm708u09+gPl0/KOo4kScqYxauEZi5exefzy8iPug8ae2cdR5IkZcziVSKrNrZy6dbn6UG7Q/WSJAmweJXM1KXreKBhDu3njIMzr8g6jiRJqgAWrxLYc6CdjU0vc1FsdqhekiT9hsWrBGas3MzdaTYdjafBFZOzjiNJkiqExasEpi9Zwxfzy8hdcy/06Jt1HEmSVCEsXt2sc6j+BXpykBj7aNZxJElSBbF4dbOpSz/gwYY5dJw1Cs6+Jus4kiSpgli8utGeA+283/Qql8QG8g7VS5Kkw1i8utH0lZu4pzCbjoY+cOWXso4jSZIqjMWrG01f8iZ3NSwhd/VXoOdpWceRJEkVxuLVTZo37uSSrS/SiwMO1UuSpCOyeHWTaUvX8UDDXDrOuArOGZN1HEmSVIEsXt1g9/421jYt4PL4gPy4RyEi60iSJKkCWby6wfSVm5lcmEUh3wuuvjfrOJIkqUJZvE5RSomfLXmLuxuWEFfdA70GZB1JkiRVKIvXKWre2MrIlpfowz5irNfukiRJR2fxOkXTlq3vHKofehkMuzbrOJIkqYJZvE7B7v1tvNO0iKvjXfJjH3OoXpIkHZPF6xRMX7mZuwuzKeR7OlQvSZKOy+J1klJK/HTJO3ypYSFxxSToMzjrSJIkqcJZvE5S88ZWLm55ib584lC9JEnqEovXSZq6dD0PNLxCx5CRcP51WceRJElVwOJ1Enbvb+PNpiWMjnccqpckSV1m8ToJz67czOQ0m0KuB1xzf9ZxJElSlbB4naCUEk8v+RVfaVxAXP4H0HdI1pEkSVKVsHidoKaNrYzY+jL90l5i7GNZx5EkSVXE4nWCpi5dx0ONcygMGgHDb8w6jiRJqiIWrxOwa38bq5teY1y8Rc6hekmSdIIsXidg+uubmJxmk6IBRj2QdRxJklRlLF5dlFLiJ0ve5d7GBcRld0K/M7KOJEmSqozFq4tWbtjJ8JY5DEi7YMyjWceRJElVyOLVRdOWrefBxrkUBpwPF96SdRxJklSFLF5dsGt/G81NK5gYa8iNfQRy/rFJkqQTZ4Pogumvb+Lu9DIp8jDqoazjSJKkKmXxOo6UEk8teZevNs4nPvV56H921pEkSVKVsngdx8oNOzmv5VUGpZ0O1UuSpFNi8TqOqUvX81DjXAr9z4WLb8s6jiRJqmIWr2No3dfGiuaV3BDN5MY8Arl81pEkSVIVs3gdw/SVm5icXiZFDkY7VC9Jkk6NxesoOofq3+P+xvnExZ+FAcOyjiRJkqqcxesoXt+wk3Nb5jEkbYexj2UdR5Ik1QCL11FMXbqeBxtfodDvLBj577KOI0mSaoDF6wha97WxvLmZG2MluTEPQ74h60iSJKkGWLyO4NnXNzEpzSEARj+cdRxJklQjLF6HSSnx1NL3ebDHPOKiW2HQBVlHkiRJNcLidZgV63dyZstChha2OVQvSZK6lcXrMNOWrefhxrkU+p4Bn/p81nEkSVINsXgdonVfG8uaV3NzrCA3+kHIN2YdSZIk1RCL1yF+tmIjdxXmkqMAYx7JOo4kSaoxFq+ilBI/XrqOh3u+CiNugsEXZh1JkiTVGItX0Yr1Ozhj2yLOLGyFsY9mHUeSJNUgi1fR1KUbeLhxLqn3ELj0i1nHkSRJNcjiBbR+0saS5je4NZYTox+Ahp5ZR5IkSTXI4gX87PWN3JVeIU8HjPE0oyRJKo2SFq+I+E5ErImI1RExLSJ6RcSIiFgaEWsj4qmI6FHKDF1x0yWn883+C+GCT8PpI7OOI0mSalTJildEnAt8CxiXUroSyAP3Af8D+NuU0sXADuBrpcrQVSN2L2fAvg0O1UuSpJIq9anGBqB3RDQAfYAtwK3A08XHfwTcXeIMx9e6CQYNh8vuyjqJJEmqYSUrXimlTcD3gPV0Fq5WYDmwM6XUXtxsI3BuqTJ02egH4U9fh8ZeWSeRJEk1rJSnGgcBk4ARwDlAX+BzJ/Dz34iI1yLitZaWlhKlPETO9xlIkqTSKmXbuB14P6XUklJqA54BbgAGFk89AgwDNh3ph1NKT6aUxqWUxg0dOrSEMSVJksqjlMVrPTAxIvpERAC3AW8Ac4EvF7d5FJhewgySJEkVo5QzXkvpHKJfAawqPteTwJ8D342ItcAQ4AelyiBJklRJGo6/yclLKT0BPHHY3e8B40v5vJIkSZXIiXJJkqQysXhJkiSVicVLkiSpTCxekiRJZWLxkiRJKhOLlyRJUplYvCRJksrE4iVJklQmFi9JkqQysXhJkiSVicVLkiSpTCxekiRJZWLxkiRJKhOLlyRJUplYvCRJksrE4iVJklQmFi9JkqQysXhJkiSVSaSUss5wXBHRAqwr8dOcDmwr8XNUsnpefz2vHep7/a69ftXz+ut57VCe9V+QUhp6pAeqoniVQ0S8llIal3WOrNTz+ut57VDf63ft9bl2qO/11/PaIfv1e6pRkiSpTCxekiRJZWLx+q0nsw6QsXpefz2vHep7/a69ftXz+ut57ZDx+p3xkiRJKhOPeEmSJJVJ3RWviPhcRLwdEWsj4i+O8HjPiHiq+PjSiBhe/pSl0YW1PxYRLRGxsvj19SxylkJE/DAitkbE6qM8HhHx/eKfTXNEjCl3xlLqwvpvjojWQ/b9fy13xlKJiPMiYm5EvBERayLi20fYpib3fxfXXsv7vldELIuIpuL6//II29Tka34X116zr/kAEZGPiNcjYuYRHstuv6eU6uYLyAPvAhcCPYAm4PLDtvkPwD8Wb98HPJV17jKu/THgH7LOWqL1fwYYA6w+yuNfAF4AApgILM06c5nXfzMwM+ucJVr72cCY4u3TgHeO8He/Jvd/F9dey/s+gH7F243AUmDiYdvU6mt+V9Zes6/5xfV9F5h6pL/fWe73ejviNR5Ym1J6L6V0EPgxMOmwbSYBPyrefhq4LSKijBlLpStrr1kppXnA9mNsMgn4l9RpCTAwIs4uT7rS68L6a1ZKaUtKaUXx9m7gTeDcwzaryf3fxbXXrOL+3FP8trH4dfhgc02+5ndx7TUrIoYBdwL/dJRNMtvv9Va8zgU2HPL9Rn7/Reg326SU2oFWYEhZ0pVWV9YO8KXiqZanI+K88kSrCF3986ll1xVPS7wQEVdkHaYUiqcTRtP5r/9D1fz+P8baoYb3ffF000pgKzArpXTUfV9jr/ldWTvU7mv+3wF/BhSO8nhm+73eipeO7TlgeErpamAWv/3XgGrfCjo/4uIa4O+BZzPO0+0ioh/wU+A/ppR2ZZ2nnI6z9pre9ymljpTSKGAYMD4irsw6U7l0Ye01+ZofEV8EtqaUlmed5UjqrXhtAg5t9MOK9x1xm4hoAAYAH5clXWkdd+0ppY9TSgeK3/4TMLZM2SpBV/5u1KyU0q5fn5ZIKT0PNEbE6RnH6jYR0Uhn8ZiSUnrmCJvU7P4/3tprfd//WkppJzAX+NxhD9Xqa/5vHG3tNfyafwNwV0R8QOdYza0R8a+HbZPZfq+34vVLYGREjIiIHnQO1M04bJsZwKPF218G5qTi9F2VO+7aD5tpuYvOeZB6MQN4pPjutolAa0ppS9ahyiUizvr1fENEjKfztaEmfvkU1/UD4M2U0t8cZbOa3P9dWXuN7/uhETGweLs38FngrcM2q8nX/K6svVZf81NK/zmlNCylNJzO33VzUkoPHbZZZvu9oRxPUilSSu0R8SfAL+h8l98PU0prIuKvgNdSSjPofJH6fxGxls5h5PuyS9x9urj2b0XEXUA7nWt/LLPA3SwiptH57q3TI2Ij8ASdw6aklP4ReJ7Od7atBT4BHs8maWl0Yf1fBv44ItqBfcB9tfDLp+gG4GFgVXHeBeC/AOdDze//rqy9lvf92cCPIiJPZ6H8SUppZj285tO1tdfsa/6RVMp+98r1kiRJZVJvpxolSZIyY/GSJEkqE4uXJElSmVi8JEmSysTiJUmSVCYWL0lVIyI6ImLlIV9/0Y3/7+ERsfoEtu8bEbOLtxcUL8IoScfkC4WkarKv+BEoleA6YHFEDAL2Fj/vTZKOySNekqpeRHwQEX8dEasiYllEXFy8f3hEzCl+CPDLEXF+8f4zI+JnxQ+GboqI64v/q3xE/J+IWBMRLxWv+H34c11UvBjpvwIPAMuBa4pH4M4o05IlVSmLl6Rq0vuwU41fPeSx1pTSVcA/AH9XvO/vgR8VPwR4CvD94v3fB14tfjD0GGBN8f6RwP9KKV0B7AS+dHiAlNK7xaNuy4HxdH6w8NdSSqNSSlu7dbWSao5XrpdUNSJiT0qp3xHu/wC4NaX0XvFDoT9MKQ2JiG3A2SmltuL9W1JKp0dECzDskA8IJiKGA7NSSiOL3/850JhS+u9HyfLLlNK1EfFT4NsppY3dvFxJNcgjXpJqRTrK7RNx4JDbHRxhDjYi/rE4hD+yeMrxc8DMiPjOST6npDpi8ZJUK756yH8XF28v4rcffvsgML94+2XgjwEiIh8RA7r6JCmlbwJ/Cfw34G7g58XTjH97avEl1QPf1SipmvQuHmX6tRdTSr++pMSgiGim86jV/cX7/hT454j4T0AL8Hjx/m8DT0bE1+g8svXHwJYTyHET8C/AjcCrJ7USSXXJGS9JVa844zUupbQt6yySdCyeapQkSSoTj3hJkiSViUe8JEmSysTiJUmSVCYWL0mSpDKxeEmSJJWJxUuSJKlMLF6SJEll8v8BAh4A7MgZsEEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, 10 * pltsize))\n",
    "plt.plot(range(epochs), train_acc_all,label = 'Training Acc.' )\n",
    "plt.plot(range(epochs), val_acc_all, label = 'Validation Acc.' )\n",
    "plt.xlabel('Epoch #')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Showing max 10 first failures. The predicted class is shown first and the correct class in parentheses.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeyklEQVR4nO2deZhUxbXAfyWDqCxR2R0dDaKgoggqCshiBIGnRoKKxAWDBCUgm1vICyIyihtKogYVFMG8xGgiJKhkQiLigiIqqCEIOKCIRhAUFEVW6/1x+1Tf6enu6Zm+3X27Ob/vm4+ZvkvXoerWrTqrsdaiKIqiKIpSyOyX6wYoiqIoiqJkGl3wKIqiKIpS8OiCR1EURVGUgkcXPIqiKIqiFDy64FEURVEUpeDRBY+iKIqiKAVPzhY8xpjGxpiVxpgDUzj3JGPMa9loV1AUunwAxpjjjTFvGWNMCufmq4zV6cemxpj3jTF1stG2oCj0fjTG1DHGrDDGNE/x3JXGmMbZaFtQFHofwj4jY0GP1Vz3YaALHmPMgMiE/60xZo0xpkuS08cCM62130WunWmM2WWM+cb3UwvAWvsesNUYc36Q7a0u6cjnu8ehxphNxphX5bOwyCcYY44xxuwwxvxfFaeWApNtJJmTMeY4Y8wCY8xXxphyY8xP5MSwyBgzvr4xxuw1xjyQ5JLYcTrZGPOBMWZbZLIZKCdaazcCLwJXZ1aK1CjUfjTGHGWMmWeM2WKM2WCMedAYU5TkkquBl621n0Wuv9EYszzShx8aY26UE621O4EZeP2ec9Low4TjPAx96KcmMkZe9o8ZY9ZF+vEdY0wfOTEsMgYwVg82xswyxnwe+ZkgJ4ZprObNXGOtDeQH6AmsA87AW0gVA8UJzq0DbAYO9302E7gtyf0vA54Lqr3Zls93bDrwMvBqmOSLact84BXg/5Kc0xz4Ejgg8ncRsBq4DqgF/Aj4Fjg2jDJG2lMP+Abommo/ArcCrSNj4HRgC9DJd7wzsDzXshVyPwLzIvPFAUAz4N/AyCTn/wfo7Pv7JqB9RNZWked6gO/44ZF+r5OPfRjneKVxnus+DGCc1gUmAEdFnsXzgG3AUWGSMYCx+jjwZ+CgiKxrgEFhG6v5MtcEqeG5FZhorV1srf3eWvuptfbTBOeeDmy11n5SjfsvBM42uTMXpC2fMaYT0AZvEMeykNzKB3haLGAr8EIVp/YEllprd0T+bg0cBkyx1u611i4AFgFX+K5ZSAhk9HEh8DnegxqPSv1orb3FWrsyMgbeiFzb0XfNG0ALY8yRmWp0KhR4P/4QeNpau8NauwEoA06Id6IxpgRogdcvAFhr77bWLrXW7rHWrgL+hrdQleOf4C1kz8igDFWSRh/GEm+cLyQEz2JNZbTWfmutnWCt/SjyLD4HfAic4rtmIbmXMa2xCpwP3G2t3W6t/Qh4DLhKDoZhrObTXBPIgidiejoVaBxRS30SUd0l8ns4EVgV5/NhxpgvjTFvG2Mu9B+ILC524+3IskoQ8kXu8SBwLVCpnkcu5ROMMQ2AiXgr7qpI1IcVbom3wAPCIWMMVwJP2MhWIg5JZYz0/2l4uzIArLV7gHKgbYDtrBb7QD/+BhhgjDnIGFMM9MF7kcTjRGBtpF8qYYwxQBd8fRjhfQqnDyuN8xD0YaAyGmOaAsdS8VnMuYwEM1ZNzO9tYo7nbKzm21wTlIanKVAbuAhv8jgZaAeMS3D+wXjqRz/3A8cATYCbgZnGmM4x52yLXJttgpBvJPCGtfbtJN+TK/mEUuCxFDVvsTKuwttF3miMqW2MOQfohqeK9ZNrGQGIaGC6AbOSnBavH/08DLwL/CPm81zLWOj9+DLeLvlr4BPgLeCvCc6tqg8n4M2DsVrXfO5DRxXjvFBkrA38AZhlrV0ZczjXMqY7VsuAscaY+saYlnjanTA9i3k11wS14BHH3AestZ9ZazcD9wH/k+D8LUB9/wcRFfMXETXzPLwB3C/muvp4qrNsk5Z8xpjD8BY8v67ie3IlH8aYk4EewJQUL6kgo7V2N9AXOBfYAFwPPI33kPvJmYwxXIHnR/VhknMqjVPBGHMP3k6kfxwNkfZjhjDG7If3EpiN58fRCDgEuCvBJcn68FpgIHCu9RxA/eRtH8aQbJznvYyR8fB7YBee9jyWXMoYxFgdiff++QDP9Pok4XkW82+uCdBpaT0w0Pd3P2BZgnPPBD6o4n4PAff5/i4GdpAj56x05It06o5Ip24AvsJ7QDcAtUIi32g8hzFp4zd4D9rSBOdfDvyzinu+BlwTlj6Madtq4Koqzok7TvH8uZYDDeMcKwK2A0dqP2ZEvkZ4JuEf+D7rSwJHcTynzu1AUcznV+FNrC0SXPcB0C3f+zDROM/1sxiEjHjmj8fxIiMPDKGMgYzVmHMmAU/GfJaTsZqPc02Qwk8E3sQzSR2C5yBXmuDc/YFN+KKc8MxF9fC0TufgqbG6+45fCszLdqcGIR9etE8z388oPMe0ZiGS76CYNk4G/gI0TnB+U+ALfJEhwEl40QgHATfgORHW8R3PqYy+dnSKPKj1qzgv3jj9VWSCaZbk3iu0HzMq41q8UNwiPFX3HOCPSc5/j4qRdJdFJujjEpxfHPk/ydWLMu0+9I3FuOM8BH0YxDh9GFgM1EtwTc7nmwDG6tFAQ7wopj54EVknhGGs5uNcE2SUVinegmA1nhPVMuD2eCdaa3fhhepd7vt4FPApnurqHmCItXah7/hleAM8V9RYPmvtTmvtBvnB0/Dsjvwu5FQ+60UB+Nv4DbDDWrspwfkbgQXABb6PrwA+w7PLng30tBVNBbnuQ+FKYLa1NplvR6JxOgkoAcp9OU7+13dc+zHz9AN64y1Gy/GcGsckOf8RKkZ+3Ib3EnnT14d+eS7F8weJNXNlhYD6EJKP87wepxHfpGvw/Ck3+PrxMt9luR6nkP5YPQUvlH0bcAdwmbXW72Cfs7Gal3NNtleFvpVbY2AlcVSRcc49CXg9V21V+RK2+3i8RaApYBmr049N8BbDcfOhhPWn0PsRT8O6Amie4rkrgSa5brf24T4pY0GP1Vz3oYncWFEURVEUpWDR4qGKoiiKohQ8uuBRFEVRFKXg0QWPoiiKoigFjy54FEVRFEUpeJKVqccYk9cezdZaU9U5KmP4qUrGQpcPVMZ8QGUsfPlAZcwHEsmoGh5FURRFUQoeXfAoiqIoilLw6IJHURRFUZSCRxc8iqIoiqIUPLrgURRFURSl4EkapZULDjnkEABKSkoqHVu3bh0AY8aMYfny5QCsXr0agHfffTdLLazMwQcfDMDXX38NwPfff5+ztiiKoiiKUhnV8CiKoiiKUvCEQsNz7rnn8uMf/xiA7t27A9CyZctK54k258gjj6ROnToVjtWqVSuzjUzCn//8ZwC+/fZbAB599FGee+65tO/bpEkTAL788ksA9uzZk/Y9FWVfpV69egA89NBDXHbZZQC8/vrrAJx11lkA7Nq1KzeNUxJSVFQk1bPZu3dvjlsTDD/96U8BOPXUUwEYPXp0pXP2228/XnvtNQD3Ppk2bRoAX3zxRTaamTEaN24MwC9/+Uuuu+46AFauXAnAjBkzAJg8eXLg35v1Bc/RRx/N8OHDARgyZAgABx54IMZUmQuJY489NqNtqylLly4F4MYbbwTgpZdeCuS+8hDUrl27wv1zwX333VehTUuXLuWjjz4CoLi4GIBFixaxbNkyABYuXAjAZ599BqiZL+zsv//+AIwcOZJbbrkFiE6qsvDu2bMnixYtArxNB8Cll17KXXfdBYSrj//zn/8AUFZW5uaNbt26Ad7zKebxjh07AvDCCy8AMG/ePO64445sN1eJw3nnnQfAE0884cbipEmTAJg1a1aoxlsqlJaWMmLECMB750F0oy4LOj/ff/89p59+OoD7t02bNgBuwZ5vHH300QB88MEHgCe3yC5zSjxlR1CoSUtRFEVRlIIn6xqeww8/nFGjRlXrGlF1ya4tbKxfvz7we/bs2dOp+mT3nQsNT48ePQDo27cvABdeeCEAO3bscJ/Vr18fgEGDBjFo0KAKny1evBjwdiSZ+H9Kl6Ii7xE46qijALj88suBqPnDz+zZs1mxYgUAW7duzU4DM8x++3l7nilTpgBw/PHHc+211wLw1FNPATB16lQA1q5d64IK/vWvfwFwwAEH8OijjwKwadOm7DU8hqZNmwLwt7/9DYDjjjsO8OQRZCfZqVMn97v0Y/v27QHo0KEDrVu3BuDKK6/MQss9ExvAG2+8AcDMmTOz8r1hRwJTZsyYwcUXXwzgxtqgQYP4+c9/DkRdHcLK7bffDsD111/v5hvhq6++Arxx++yzzwJRs+rcuXMr3Uu0H40aNWLz5s0Za3MmGD16NEOHDq3w2datWyktLQWiZrvy8vKMtUE1PIqiKIqiFDwZ0fA0atTIaXHE5l9WVgbAzp073apWnHzr1q3L/PnzgeiqXnY7y5Yt47vvvqtwftj4xS9+Efg9u3fv7jQ74iOUC6666ioAPv30UwDmzJnjjv39739PeN31118PeFoBiDpeh4G2bdsC0KtXL+cn0Llz5yqvGzNmjNtN/upXvwLgr3/9a4ZamXkaNGjA448/DsCGDRsATy7RygniRLh582bnn3bAAQcA0KdPn5xqdsB7Vv74xz8C0KxZswrHrLUuZcWdd94JwCuvvOI0PKLhEifKF198kdNOOy0r7RZk19uvXz/Ae97fe++9jH7nCSecAMAVV1wBwF133cWWLVsy+p3VRXwEb7jhBm644QYAzjnnHAD+9Kc/8fbbbwM4jZzMUWGhRYsWAFx99dUAfP75526cynO3c+dOICorRDVCEJ0/pW9OOeUUwNNI54uGRzQ4N910k9NwiSZ2xIgRfPLJJ1lrS6ALnrp16wIwf/5891L5yU9+UuGcxYsXO/WxdHJJSYkTOp8c0cSBTJx2g6Rnz57u94kTJwZ+/1SRflyyZEm1rrv33nsz0Zy0kInnZz/7GeA5AsqLXl7qMtl888037jpx2u3Xr59z6H3yySeBaITewIEDM9z64Ondu7cbu2LGEidzP6+++irgRTKJA728pGWDkgtEvT9r1qxKCx0xC9x5553OCVleLvEQE9gPfvAD3nzzTSAaLLB79+5gGx6DbAAbNWoEwCWXXOLU+tu3bw/sew499FDAixCScdywYUPAWyjKcxFmZGM8ZMgQZ3I98cQTgfAteOSlLmbgl156ibFjx1Z5nZg4x44dy7///W8A5+z8yiuvADBs2DC3GQ0rt956K+BFYoHnoC0mSZlvMv1sxaImLUVRFEVRCp5ANDxiehF1Xdu2bV34oDg3+vGr7wA+/vjjIJqRdSSktUGDBhU+T8f0JvmFateu7Ux5L7/8co3vlw516tRx7cnlTj4IunXr5sakyHTttdfy2GOPAcnzr4jqeMWKFS489sEHHwSiGsxTTjnFqdjzhVatWrmxFU+zI4j5Y+7cuWzbtg3IjKN+qogZSvruiCOOqHSO5Dnxm2CT4c/jJSat5s2bA5mfnyTEWJxUx44dyzHHHANEzXAbNmzgv//9b8r3LCkpoUOHDoBndoRoWP4Pf/jDSuenYtINE88884wzL7dr1w6Iuk2EgcMOO8yFWVcXsXbcdtttTiMkmhEhE1aFoOjVqxcQdfUQM9bw4cOd9ioZ99xzDwD9+/d3muRk7hPVQTU8iqIoiqIUPGlreOrVq+ccOMUBdPPmzS5LYpA26DBRr14955gryG5SsmHWhAsuuADwtGTTp08HchcCXVxczGGHHQZ4vg35zLPPPut8zMS2nMpuIxbZaYnPjuyixek1H5AdVNeuXenSpUuV5w8bNgzwkqVJ2oFMO9Um4+STTwao0HbR0A0YMACIH9KbjP79+7vfRduVLc3zP/7xDyCqoejVq5dL/3DuuecCnq+DyCiaHtFUih8ZRP1FBg4cGDe1QiLC4HwvWgvpC8m6v2XLFiejBMEUFxc7+VLV4mWTevXqVUqm+4c//KFa97jllluc35WErAuZDN1Oh3POOYe7774biPqH3XzzzUDUUTsR4kA/cuRIwLNyiH+WangURVEURVFSJG0NT9++fZ2dUXZEXbp0cZEHhcqUKVMqlboIIpoqTJ73a9eudTtlCQt94IEHgMRJ5mR1/89//rPCv7mmfv36LgJQ/FBqgkQVJIv4CSui/ZB/W7VqFdd3SXaV4vMkJWDuu+8+/vKXv2SjqdVGNADV1VRIwkK/D4tE/2QLqZEn2pzS0lKuueYaILpLljQA/s+EX//613HvK5oP0YRJFBhEa1KJdl60ydlGEpRedNFF/OY3vwFwmhHxnSsqKnJJIEX7tn79epe6RBLThonVq1e7UHIpIyFJS6uD+AqKT9rBBx8cUAuDRRK3zpo1yz1TUq5FyhLt2LEj7rWtWrUCopYRf3LGoNcRaS94OnXq5H6XOkrZjKvPNlLkVDJ/QtQJe9WqVTW+r5iMJAQ6LEjNE5lwRX5xGI1F8kTIpBWWBc/QoUNd/0ioZ02QWjAip5gXJHQ7jMgEIjWvRMXsNzeLI/CPfvQjxo8fD3iOr+DlHwK4//77s9PgKhBzmiwQzjvvvLjFF1NBwmNl87Ju3TqXWTpX3HzzzTz//PNAdHE6cOBAt8iWhYBkkfYX1BSzwQsvvODMj7GmrT179rhiqVKcMldIO3r37u0WPDK3yAZ6//33d46wsgFr3rw5X3/9NRCtwST10cKCPG+//e1vAW/uFHNyqnz44YcALoBFFjxFRUXuuQ5DUWnZFDVp0sSZHS+55BIg2vZ4HHvsscyePRuIBj8J3333nXsOgkJNWoqiKIqiFDxpa3guuugi93vv3r0Bz9lKki6988476X5FKJDQc9kd+0PRJUNqIpVdKkioqDhkQmItSjaJzfLsV63HcvHFF7usp7FhlLnmkUceCeQ+UmvroIMOAqJ1pvyJCsOGJMiUHaHUrDn88MOdKlpCo4cOHerMXKLNE6fasCA7WnFkrIlDo2ggxelcmD9/fs4zR0O0Bp3869dgiVZdNHD+LOaSBmTMmDEuK3Es48ePz7lmR0yIv//97wG49NJLE+7md+3aVUlLtXbtWrp27QpEs/JLwIf8nWuk70QTNWjQINdXqczt3bt3d0n7JEWC0K1bN2eqfPHFFwNrc03xm4TFvJwsu76Y6KZNm+bm0lhmz54duLVINTyKoiiKohQ8RmrKxD1oTOKDEay1cctByGcPP/wwEF3tlpSUuJA6f/VzSW72+uuvA8H4AVlrTVXnpCIjRNPY+yvzilOghFH6benVRTQ7fo2KJB9bs2ZNwuuClDEe4tshNZREI9C6detKCRanTZvmKhjLLjS2LlNNqErGdOSrDmPHjnWlJ8QuLxXs0+n7TPehaN2kppQ8d23btnX+Z+Ic2Lp1a5dkMlb7kQ6ZljEZEqotfnIdO3Z0vlixgQbXXHNNjR14cymj7/4ALFy4kDPPPLPCMUma2bJlS6d1qC5BPYsyL4jfRrdu3RIGE/Tp08dpjDdu3Og+O/300wGc749oQdq1a1djR+ZM9KFfiyXI+03ej/GYNGmSe48+8cQTAM4vq0ePHk7zKikMUnXYz4SM8l5s2bIlp556KhC/BqRodiRJ8d69e11dO3/yT/BKAdXUUpBIxrQXPPfccw/XXXddjRoVD1EnL1y4EIg67dWEoDq2V69e7kUndcDKy8td5lKJHJDJRvK9+Nm9e7erzSPIgsFaW2nBM2fOHGcuTNZH2ZpkJTeCTC6lpaVMmDABiJq5XnvtNSeHZKEuhAWP1DUrKytzE6mYC4Ko35OtPpTMpzLxlpWV8bvf/Q6I1rsZNWqUm4SDfK5ztRg45JBDnPlAFgD+aCVBXkAnnHBCjaP4wrDgEUdmv2O+5PESk086DvZBPYsS7SfzvEQj+RGT3aJFi5yZVUwnUgMPoptRqbO1a9cu93xWN5dSJvpQ5v327du7TXIqwSlLly5lypQpQHRzLW4TRUVFLp+YbEol4q4qMiGjRGKNGjXKBRPIXCmO6Z07d3a5dsQB//nnn3fPpeR8k36UDOE1IZGMatJSFEVRFKXgSVvDU6tWLVfLRNRURUVFrr6NrD6ri7RrwoQJ3HbbbTW9RyAr2alTp7qaHsL69esrZXMVp1DJoRG5P+Ct1uX/SRBH0eeee87tvqV2zuOPP87gwYOralrWd5Vi2urSpYtTqcoOZtu2bW4XWQgmLckvJdqtd99914VfBulMl0vNQKzmbvHixU5zGWQl42zLKJqrIUOGVMjrkQjJGSLavJoQBg2PZPMdMGCAc6SXPD3xtCjVJahnUcweoukZMWKEyyYsGoJ7770X8AICRNOfLMOwX9MjGiFJH5FqfcNM96FkZJf3ib+umWg9SktLAU/uZKZHMQeKk/B7773ngmqSkQkZ27ZtC3gBBJJ3KLbGJEQ1j/KO7Nixo+tnQd4d6Tifq4ZHURRFUZR9lrQ1PIk4++yzgejuX/w9pBJxqsydO9dVpK4uQa1kN27cmHatpPLyclf7Rlbt4qj99ttvu/uL3bpDhw689dZbVd4327tKcQAdPHiwS/glO5Jx48a5RG75quER2/rdd9/tfF1Ek9W3b9+0nJMTkSvNwBFHHOEc4sVvpU2bNkkrp9eUbMkoAQRPPvmk3DOl6z7//HPAm7f8wRTVIQyO2eLUW6tWLcaNGwfAHXfcEdj3BP0sitP4uHHjXAI6qckoST3PPPPMalWKb9mypfOFFI10//79kybAE8KgpasuzzzzDOBpzVKp0J5JGdu0acPTTz8NRH1ZxYduzZo17rmUd+GyZctccI68KyTcPl4wVKqohkdRFEVRlH2WtBMPJkJs4oJE75x22mkucZikQZ8+fbpLrOUP3QsLEydOdDWkhI8//tjtpmKTzi1YsKCSdmbJkiUuwZKkhpfEWSNGjHC2ZgkdlpIOYUPqw0yePLnSMWl7PiM+BZ07d3YlDCQ0sn79+i61u4RzV4XUpRKftjD8H0n454IFCypFnfmjX/IJ8RsQP0LR7Kxbty7urlc0APJcyy50+vTpFcrl5As33XQTUDG0N50dcraQ2ntdu3Z15T3Ev0U0PtUNoS8vL3elf+R5Xrx4sUuzkI918JIhWto+ffq4yN5c1bxbvny5e9fLWIynWZNnUrQ7EE1Jk8lxm7EFTywSanb77bc7J0JxAG3ZsiXdu3ePe10Y6nI98sgjlRyoNmzY4DJJ+msSJUMWC4LUmZKMthDt9EIvvhoWpHihOM6dccYZ7thJJ50ERCfezZs3Oyd8WeTGc4gV9XtZWZlzRpcXsnxfLhkxYgQAzZo1C0V7giQ2SCLeYmf9+vUud4nUX3rooYcAz4lUwtZlYxJ22rdv74r7+qlprp1sIs9R79693YsuXkHb6iJmaHGhWLp0qcswLnW58mFBmApSs+uss85ymeBzWeQ3lf6TedYY496jErCTSdSkpSiKoihKwZM1Dc/7778PwNNPP+0cCwVJTATRjLVSV0VCg3PJnj17UnIgrikTJ050YcGZcBjNFj169Mh1E6qN7DQkBYAkmExkepJds9SNE41C3bp1nYOkhAE3aNDAhXY3bNgwE82vFqJFFSdz2RnuKzz11FOAF4qfqF5W06ZNndOkJHsLO2vWrGHt2rVANDQbwmE+TZV06hAmQ0y2gwcPdhmPJbDi/vvvz8h31oTatWu77N/CpEmTXHoW0diI068fcQdp164dM2bMyHBLg6FFixaAl35GNG/Z0KiqhkdRFEVRlIInaxoecVwaPXq0q3wryaeaNGninEBlFS5h7PsCmzZtCkWF5nT56KOPXHjvO++8k+PWpIZUGm7VqhUQ9bOSukOJEP+reMiO5auvvnJ26Vz6ygwcOBDwysBAtOZONmzm2UICIcTRXPyvdu3a5fyzJIFpPCfKsrIywEs9IInv8kXDs3379kp+hDt37nQ10RRPMyIpJ6QMwvr164Fw9PPw4cMrBYIYY5yGRzTPxcXF7rhopcVvZ+/evSn7k+Ya8X3cunVrVq04WVvwCBs3buT8888HcHU1zjjjDFcXRF6YSv5RXl7uTDdt2rQByKgpMAjkRZksg2t18TucSySf/JsLpMCpZJu9+uqrgajshYCYDiVTsswnM2fOZMmSJVVeLzXRdu/eXSnCNOy0bdvWLfCEOXPm5IXTcjaRTNMSySRuBGFY8Kxbt87lwoq3OUo2hmUBP3XqVGbNmpWZBgaMFBvdunVrViND1aSlKIqiKErBk3UNjx8xX8m/Sv4jO+3Y3ERK7pG8V6lknM1XxDQ8bNiwal0nofoLFiwoCM1IPOfWfR0JiJFacTWt85gJ5syZw7x584CoE7IxxtXGkszEfiRli+TQEq1JPiB5sEpKSlxF+DFjxmT8e8PT44qiKIqiKBkiY7W0wkA+1kWpLmGScfz48a4Krt+5Ll1yVS09W2SyD5s1a+bSPkhYdi4SroVpnGaKXMnYsGFDV2n8xBNPBLx6RJkIHNBnUWXMB7SWlqIoiqIo+yyq4VEZQ4/uKlXGfCCXMkrNtsaNGwOwatWqTHyNPouojPlAIhl1waMyhh6dZFXGfEBlLHz5QGXMB9SkpSiKoijKPktSDY+iKIqiKEohoBoeRVEURVEKHl3wKIqiKIpS8OiCR1EURVGUgkcXPIqiKIqiFDy64FEURVEUpeDRBY+iKIqiKAXP/wPUo8Q5rkyaAQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_failures(nonlinear_model, test_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recap\n",
    "To train and validate a neural network model, you need:\n",
    "- Data split into training/validation/test sets,\n",
    "- A model with parameters to learn,\n",
    "- An appropriate loss function,\n",
    "- An optimizer (with tunable parameters such as learning rate, weight decay etc.) used to learn the parameters of the model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Compare the quality of your model when using different:\n",
    " - batch sizes, \n",
    " - learning rates,\n",
    " - activation functions.\n",
    "\n",
    "3. Bonus: What is a learning rate scheduler?\n",
    "\n",
    "If you have time, experiment with how to improve the model. Note: training and validation data can be used to compare models, but test data should be saved until the end as a final check of generalization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework solution\n",
    "Make the following changes to the cells with the comment \"#For HW\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#####################To modify the batch size##########################\n",
    "batch_size = 32 # 64, 128, 256, 512\n",
    "\n",
    "# The dataloader makes our dataset iterable \n",
    "train_dataloader = torch.utils.data.DataLoader(training_data, batch_size=batch_size)\n",
    "val_dataloader = torch.utils.data.DataLoader(validation_data, batch_size=batch_size)\n",
    "##############################################################################\n",
    "\n",
    "\n",
    "##########################To change the learning rate##########################\n",
    "optimizer = torch.optim.SGD(nonlinear_model.parameters(), lr=0.01) #modify the value of lr\n",
    "##############################################################################\n",
    "\n",
    "\n",
    "##########################To change activation##########################\n",
    "###### Go to https://pytorch.org/docs/main/nn.html#non-linear-activations-weighted-sum-nonlinearity for more activations ######\n",
    "class NonlinearClassifier(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.layers_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 50),\n",
    "            nn.Sigmoid(), #nn.ReLU(),\n",
    "            nn.Dropout(0.2),\n",
    "            nn.Linear(50, 50),\n",
    "            nn.Tanh(), #nn.ReLU(),\n",
    "           # nn.Dropout(0.2),\n",
    "            nn.Linear(50, 50),\n",
    "            nn.ReLU(),\n",
    "           # nn.Dropout(0.2),\n",
    "            nn.Linear(50, 10)\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        x = self.layers_stack(x)\n",
    "\n",
    "        return x\n",
    "##############################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bonus question: A learning rate scheduler is an essential deep learning technique used to dynamically adjust the learning rate during training. This strategic can significantly impact the convergence speed and overall performance of a neural network.See below on how to incorporate it to your training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "nonlinear_model = NonlinearClassifier()\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(nonlinear_model.parameters(), lr=0.1)\n",
    "\n",
    "# Step learning rate scheduler: reduce by a factor of 0.1 every 2 epochs (only for illustrative purposes)\n",
    "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/6, Learning Rate: 0.1\n",
      "Epoch 0: training loss: 0.3729931376775106, accuracy: 89.26875\n",
      "Epoch 0: val. loss: 0.36384286838769914, val. accuracy: 89.29166666666667\n",
      "Epoch 2/6, Learning Rate: 0.010000000000000002\n",
      "Epoch 1: training loss: 0.24737287873774766, accuracy: 92.67916666666667\n",
      "Epoch 1: val. loss: 0.24119560087223849, val. accuracy: 92.73333333333333\n",
      "Epoch 3/6, Learning Rate: 0.010000000000000002\n",
      "Epoch 2: training loss: 0.23363943438356122, accuracy: 93.12708333333333\n",
      "Epoch 2: val. loss: 0.22890273841718833, val. accuracy: 93.10833333333333\n",
      "Epoch 4/6, Learning Rate: 0.0010000000000000002\n",
      "Epoch 3: training loss: 0.22633988959466417, accuracy: 93.37083333333334\n",
      "Epoch 3: val. loss: 0.22283465002973873, val. accuracy: 93.28333333333333\n",
      "Epoch 5/6, Learning Rate: 0.0010000000000000002\n",
      "Epoch 4: training loss: 0.2252282265871763, accuracy: 93.37291666666667\n",
      "Epoch 4: val. loss: 0.221913401906689, val. accuracy: 93.30833333333334\n",
      "Epoch 6/6, Learning Rate: 0.00010000000000000003\n",
      "Epoch 5: training loss: 0.22455617488796512, accuracy: 93.38125000000001\n",
      "Epoch 5: val. loss: 0.22132232775290808, val. accuracy: 93.34166666666667\n",
      "CPU times: user 52 s, sys: 820 ms, total: 52.8 s\n",
      "Wall time: 53.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "epochs = 6\n",
    "train_acc_all = []\n",
    "val_acc_all = []\n",
    "for j in range(epochs):\n",
    "    train_one_epoch(train_dataloader, nonlinear_model, loss_fn, optimizer)\n",
    "    #step the scheduler\n",
    "    scheduler.step()\n",
    "\n",
    "    # Print the current learning rate\n",
    "    current_lr = optimizer.param_groups[0]['lr']\n",
    "    print(f\"Epoch {j+1}/{epochs}, Learning Rate: {current_lr}\")\n",
    "        \n",
    "    # checking on the training loss and accuracy once per epoch\n",
    "    acc, loss = evaluate(train_dataloader, nonlinear_model, loss_fn)\n",
    "    train_acc_all.append(acc)\n",
    "    print(f\"Epoch {j}: training loss: {loss}, accuracy: {acc}\")\n",
    "    \n",
    "    # checking on the validation loss and accuracy once per epoch\n",
    "    val_acc, val_loss = evaluate(val_dataloader, nonlinear_model, loss_fn)\n",
    "    val_acc_all.append(val_acc)\n",
    "    print(f\"Epoch {j}: val. loss: {val_loss}, val. accuracy: {val_acc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}