0.997AccKaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d4c08f48-fe23-4ddb-ac46-d97f05397514",
    "_uuid": "f2156d1dd26a1243e18512002e10872c5bd7271e"
   },
   "source": [
    "# Introduction to CNN Keras Swish activation- Acc 0.996 (top 7%) \n",
    "## [Shahariar Rabby](https://about.me/shahariarrabby)\n",
    "#### **Inspired by Yassine Ghouzam, PhD**\n",
    "#### 25/OCT/2017\n",
    "\n",
    "* **1. Introduction**\n",
    "* **2. Data preparation**\n",
    "    * 2.1 Load data\n",
    "    * 2.2 Check for null and missing values\n",
    "    * 2.3 Normalization\n",
    "    * 2.4 Reshape\n",
    "    * 2.5 Label encoding\n",
    "![](http://)    * 2.6 Split training and valdiation set\n",
    "* **3. CNN**\n",
    "    * 3.1 Make custom activation function\n",
    "    * 3.2 Define the model\n",
    "    * 3.3 Set the optimizer and annealer\n",
    "    * 3.4 Data augmentation\n",
    "* **4. Evaluate the model**\n",
    "    * 4.1 Training and validation curves\n",
    "    * 4.2 Confusion matrix\n",
    "* **5. Prediction and submition**\n",
    "    * 5.1 Predict and Submit results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "eb88b372-a6e5-40c8-a1c6-c03799165490",
    "_uuid": "e9aff3cf1bb8daa73bec67b970d12195677679f3"
   },
   "source": [
    "# 1. Introduction\n",
    "\n",
    "This is a 5 layers Sequential Convolutional Neural Network for digits recognition trained on MNIST dataset. I choosed to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, I will prepare the data (handwritten digits images) then i will focus on the CNN modeling and evaluation.\n",
    "\n",
    "I achieved 99.52%(in validation set) and 99.6%(on kaggle submisson) of accuracy with this CNN trained in 30 min on a GEFORCE 940MX, you can use tensorflow-gpu with keras. Computation will be much much faster !!!\n",
    "\n",
    "**For computational reasons, i set the number of steps (epochs) to 2, if you want to achieve 99+% of accuracy set it to 30.**\n",
    "\n",
    "This Notebook follows three main parts:\n",
    "\n",
    "* The data preparation\n",
    "* The CNN modeling and evaluation\n",
    "* The results prediction and submission\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"http://img1.imagilive.com/0717/mnist-sample.png\" ></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "f67b9393-8ea1-4e23-b856-2ce149cfe421",
    "_execution_state": "idle",
    "_uuid": "72334cb006d02a4bcfc2a2fe622524eba824c6f8"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "np.random.seed(2)\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import itertools\n",
    "\n",
    "from keras.utils.np_utils import to_categorical # convert to one-hot-encoding\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, Activation\n",
    "from keras.optimizers import RMSprop, adam\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.callbacks import ReduceLROnPlateau\n",
    "\n",
    "# For adding new activation function\n",
    "from keras import backend as K\n",
    "from keras.utils.generic_utils import get_custom_objects\n",
    "\n",
    "sns.set(style='white', context='notebook', palette='deep')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "6d2fb3e6-ab71-4974-b5a2-4af1ebdb99f4",
    "_execution_state": "idle",
    "_uuid": "86061d98eccaa02efe0dab0fa3884e71fcf4c310"
   },
   "source": [
    "# 2. Data preparation\n",
    "## 2.1 Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "5e51d00e-62fd-4141-bf73-50ac4f2da7d0",
    "_execution_state": "idle",
    "_uuid": "84bbd5ab8d7895bd430d5ecfe2f7ddf77baa7b74",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the data\n",
    "train = pd.read_csv(\"train.csv\")\n",
    "test = pd.read_csv(\"test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_cell_guid": "86570a36-5c20-460a-9dfd-2070548532a7",
    "_execution_state": "idle",
    "_uuid": "1213b979d5ed3e0d13824d17d694c79d2ece92fa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    4684\n",
       "7    4401\n",
       "3    4351\n",
       "9    4188\n",
       "2    4177\n",
       "6    4137\n",
       "0    4132\n",
       "4    4072\n",
       "8    4063\n",
       "5    3795\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAFXCAYAAABOYlxEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGbtJREFUeJzt3XtQVPfB//HPcpXLImVGTWcoiRiZmFqrSDVOI2pjRdvH\nRNBQd/1h+zNxWiaNwWkZvBuLkTCp2JgJlabmSQcEShO1zq+tNlgKLRpimUrUkTShxomYWLzkkUW5\nn98fGfcpseLGuJz1m/frLzkc5YODvj3LetZhWZYlAABglCC7BwAAgNuPwAMAYCACDwCAgQg8AAAG\nIvAAABgoxO4Bt0tnZ6eOHz+uESNGKDg42O45AAD4VV9fn9ra2jR+/HgNGzbsuvcbE/jjx49ryZIl\nds8AAGBI7dq1SykpKdcdNybwI0aMkPTxJ3rXXXfZvAYAAP/68MMPtWTJEm//PsmYwF97WP6uu+5S\nfHy8zWsAABgaN/q2NE+yAwDAQAQeAAADEXgAAAxE4AEAMBCBBwDAQAQeAAADEXgAAAxE4AEAMBCB\nBwDAQAQeAAADEXgAAAxkzL3oTdH84iN2Txjgvid+a/cEAMAt4AoeAAADEXgAAAxE4AEAMBCBBwDA\nQAQeAAADEXgAAAxE4AEAMBCBBwDAQAQeAAADEXgAAAxE4AEAMBCBBwDAQAQeAAADEXgAAAxE4AEA\nMBCBBwDAQCF2DwAA3JoX95yze4LXE+mj7J6AT+AKHgAAAxF4AAAMxEP0+Nx5uirN7gkDPJ15wO4J\nAAzEFTwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAbiTnYAANzA\nuZ812j1hgFE5k30+l8DjM3n1v+faPWGARf93v90TACAg8BA9AAAG4goeACSlv/ZXuycMsGfhg3ZP\nwB3O6MC3/bzM7gkDjMj+P3ZPAAB8TvAQPQAABiLwAAAYyOiH6AFTfGvPZrsneP0+fZ3dEwD4gMAD\nAIbE33/5L7snDDDp8ZF2T/Arvz5Ef+HCBc2YMUMtLS06ffq0XC6X3G63Nm7cqP7+fklSVVWVMjIy\nlJmZqZqaGklSZ2ennnzySbndbi1fvlwXL17050wAAIzjt8D39PRow4YNGjZsmCSpoKBAOTk5Ki8v\nl2VZOnjwoNra2lRaWqrKykrt3LlTRUVF6u7uVkVFhZKSklReXq4FCxaouLjYXzMBADCS3wJfWFio\nxYsXa+TIjx8COXHihKZMmSJJSk1N1aFDh/TWW29p0qRJCgsLk9PpVEJCgpqbm9XY2Kjp06d7zz18\n+LC/ZgIAYCS/BH737t2Ki4vzRlqSLMuSw+GQJEVFRam9vV0ej0dOp9N7TlRUlDwez4Dj184FAAC+\n88uT7F577TU5HA4dPnxYJ0+eVF5e3oDvo3d0dCgmJkbR0dHq6OgYcNzpdA44fu1cAADgO79cwe/a\ntUtlZWUqLS3VuHHjVFhYqNTUVDU0NEiS6urqlJKSogkTJqixsVFdXV1qb29XS0uLkpKSlJycrNra\nWu+5kyf7/uo5AABgCP+bXF5entavX6+ioiIlJiYqLS1NwcHBysrKktvtlmVZWrlypcLDw+VyuZSX\nlyeXy6XQ0FBt3bp1qGYCAGAEvwe+tLTU++OysuvvDZ+ZmanMzMwBxyIiIrR9+3Z/TwMAwFjcqhYA\nAAMReAAADETgAQAwEIEHAMBABB4AAAMReAAADETgAQAwEK8HD+C2+69Xd9k9YYD/t2iJ3ROAIccV\nPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAg\nAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAY\niMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAA\nBiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8A\ngIFC/PUL9/X1ad26dTp16pQcDoc2bdqk8PBwrVq1Sg6HQ2PHjtXGjRsVFBSkqqoqVVZWKiQkRNnZ\n2Zo1a5Y6OzuVm5urCxcuKCoqSoWFhYqLi/PXXAAAjOK3K/iamhpJUmVlpXJycrRt2zYVFBQoJydH\n5eXlsixLBw8eVFtbm0pLS1VZWamdO3eqqKhI3d3dqqioUFJSksrLy7VgwQIVFxf7ayoAAMbx2xX8\n7NmzNXPmTEnS2bNnFRMTo0OHDmnKlCmSpNTUVNXX1ysoKEiTJk1SWFiYwsLClJCQoObmZjU2Nurx\nxx/3nkvgAQDwnV+/Bx8SEqK8vDzl5+dr/vz5sixLDodDkhQVFaX29nZ5PB45nU7vz4mKipLH4xlw\n/Nq5AADAN35/kl1hYaEOHDig9evXq6ury3u8o6NDMTExio6OVkdHx4DjTqdzwPFr5wIAAN/4LfB7\n9+5VSUmJJCkiIkIOh0Pjx49XQ0ODJKmurk4pKSmaMGGCGhsb1dXVpfb2drW0tCgpKUnJycmqra31\nnjt58mR/TQUAwDh++x78nDlztHr1ai1ZskS9vb1as2aNxowZo/Xr16uoqEiJiYlKS0tTcHCwsrKy\n5Ha7ZVmWVq5cqfDwcLlcLuXl5cnlcik0NFRbt27111QAAIzjt8BHRkbq+eefv+54WVnZdccyMzOV\nmZk54FhERIS2b9/ur3kAABiNG90AAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLw\nAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgnwKfn59/\n3bG8vLzbPgYAANweIYO9c+3atXr//fd1/PhxvfPOO97jvb29am9v9/s4AABwawYNfHZ2tlpbW/XM\nM8/ohz/8ofd4cHCwxowZ4/dxAADg1gwa+Pj4eMXHx2vfvn3yeDxqb2+XZVmSpCtXrig2NnZIRgIA\ngE9n0MBfU1JSopKSkgFBdzgcOnjwoN+GAQCAW+dT4H/zm9+ourpacXFx/t4DAABuA5+eRf/FL35R\nw4cP9/cWAABwm/h0BX/PPffI7XZr6tSpCgsL8x7/9yfeAQCAwOFT4EeNGqVRo0b5ewsAALhNfAo8\nV+oAANxZfAr8fffdJ4fDMeDYyJEjVVtb65dRAADgs/Ep8M3Nzd4f9/T0qLq6WkePHvXbKAAA8Nl8\n6hebCQ0N1bx58/TGG2/4Yw8AALgNfLqC37t3r/fHlmXpnXfeUWhoqN9GAQCAz8anwDc0NAx4+wtf\n+IK2bdvml0EAAOCz8ynwBQUF6unp0alTp9TX16exY8cqJMSnnwoAAGzgU6WPHz+uFStWKDY2Vv39\n/Tp//rxefPFFffWrX/X3PgAAcAt8CvzmzZu1bds2b9CPHj2q/Px8vfrqq34dBwAAbo1Pz6K/cuXK\ngKv1iRMnqqury2+jAADAZ+NT4IcPH67q6mrv29XV1bwWPAAAAcynh+jz8/P1/e9/X2vXrvUeq6ys\n9NsoAADw2fh0BV9XV6eIiAjV1NToV7/6leLi4vTmm2/6exsAALhFPgW+qqpKFRUVioyM1H333afd\nu3errKzM39sAAMAt8inwPT09A+5cx13sAAAIbD59D3727Nn67ne/q3nz5kmS/vjHP+qhhx7y6zAA\nAHDrfAp8bm6u9u/fryNHjigkJERLly7V7Nmz/b0NAADcIp/vNzt37lzNnTvXn1sAAMBt8qlfLhYA\nAAQ+Ag8AgIEIPAAABiLwAAAYiMADAGAgAg8AgIEIPAAABiLwAAAYiMADAGAgn+9k92n09PRozZo1\nam1tVXd3t7Kzs3Xvvfdq1apVcjgcGjt2rDZu3KigoCBVVVWpsrJSISEhys7O1qxZs9TZ2anc3Fxd\nuHBBUVFRKiwsVFxcnD+mAgBgJL9cwe/bt0+xsbEqLy/XL3/5S+Xn56ugoEA5OTkqLy+XZVk6ePCg\n2traVFpaqsrKSu3cuVNFRUXq7u5WRUWFkpKSVF5ergULFqi4uNgfMwEAMJZfruDnzp2rtLQ0SZJl\nWQoODtaJEyc0ZcoUSVJqaqrq6+sVFBSkSZMmKSwsTGFhYUpISFBzc7MaGxv1+OOPe88l8AAAfDp+\nuYKPiopSdHS0PB6PVqxYoZycHFmWJYfD4X1/e3u7PB6PnE7ngJ/n8XgGHL92LgAA8J3fnmT3wQcf\naOnSpXrkkUc0f/58BQX974fq6OhQTEyMoqOj1dHRMeC40+kccPzauQAAwHd+Cfz58+e1bNky5ebm\natGiRZKk+++/Xw0NDZKkuro6paSkaMKECWpsbFRXV5fa29vV0tKipKQkJScnq7a21nvu5MmT/TET\nAABj+eV78Dt27NDly5dVXFzs/f752rVrtXnzZhUVFSkxMVFpaWkKDg5WVlaW3G63LMvSypUrFR4e\nLpfLpby8PLlcLoWGhmrr1q3+mAkAgLH8Evh169Zp3bp11x0vKyu77lhmZqYyMzMHHIuIiND27dv9\nMQ0AgM8FbnQDAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwA\nAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIP\nAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjA\nAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi\n8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICB\n/Br4pqYmZWVlSZJOnz4tl8slt9utjRs3qr+/X5JUVVWljIwMZWZmqqamRpLU2dmpJ598Um63W8uX\nL9fFixf9ORMAAOP4LfAvvfSS1q1bp66uLklSQUGBcnJyVF5eLsuydPDgQbW1tam0tFSVlZXauXOn\nioqK1N3drYqKCiUlJam8vFwLFixQcXGxv2YCAGAkvwU+ISFBL7zwgvftEydOaMqUKZKk1NRUHTp0\nSG+99ZYmTZqksLAwOZ1OJSQkqLm5WY2NjZo+fbr33MOHD/trJgAARvJb4NPS0hQSEuJ927IsORwO\nSVJUVJTa29vl8XjkdDq950RFRcnj8Qw4fu1cAADguyF7kl1Q0P9+qI6ODsXExCg6OlodHR0Djjud\nzgHHr50LAAB8N2SBv//++9XQ0CBJqqurU0pKiiZMmKDGxkZ1dXWpvb1dLS0tSkpKUnJysmpra73n\nTp48eahmAgBghJCbn3J75OXlaf369SoqKlJiYqLS0tIUHBysrKwsud1uWZallStXKjw8XC6XS3l5\neXK5XAoNDdXWrVuHaiYAAEbwa+Dj4+NVVVUlSRo9erTKysquOyczM1OZmZkDjkVERGj79u3+nAYA\ngNG40Q0AAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjA\nAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi\n8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICB\nCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBg\nIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYKsXvA\njfT39+vpp5/W22+/rbCwMG3evFl333233bMAALgjBOwVfHV1tbq7u/XrX/9aP/rRj/Tss8/aPQkA\ngDtGwF7BNzY2avr06ZKkiRMn6vjx44Oe39fXJ0n68MMPvccu/s9H/ht4C7rOnLnpOecudw/BEt9F\n32TzpY96hmiJb8748HvsuXTnbe656BmCJb7xae+lwPqz58vm7kvnh2CJ73zZfPli4Gw+c+bmf67+\n9T8XhmCJ786cufnft+cvtw3BEt/1/NvXxbXeXevfJzksy7KGZNWntHbtWs2ZM0czZsyQJM2cOVPV\n1dUKCfnP/yb529/+piVLlgzlRAAAbLdr1y6lpKRcdzxgr+Cjo6PV0dHhfbu/v/+GcZek8ePHa9eu\nXRoxYoSCg4OHYiIAALbp6+tTW1ubxo8f/x/fH7CBT05OVk1Njb71rW/p6NGjSkpKGvT8YcOG/cd/\nwQAAYKrBnnwesA/RX3sW/T/+8Q9ZlqUtW7ZozJgxds8CAOCOELCBBwAAty5g/5scAAC4dQQeAAAD\nBeyT7ALBnXo3vaamJv30pz9VaWmp3VNuqqenR2vWrFFra6u6u7uVnZ2thx56yO5Zg+rr69O6det0\n6tQpORwObdq06aZPAg0EFy5cUEZGhl5++eU74vks6enpio6OliTFx8eroKDA5kWDKykp0Z/+9Cf1\n9PTI5XLp0UcftXvSoHbv3q09e/ZIkrq6unTy5EnV19crJibG5mU31tPTo1WrVqm1tVVBQUHKz88P\n6K/l7u5urV69Wu+//76io6O1YcMG3XPPPUP28Qn8IP79bnpHjx7Vs88+q5///Od2zxrUSy+9pH37\n9ikiIsLuKT7Zt2+fYmNj9dxzz+mjjz7SggULAj7wNTU1kqTKyko1NDRo27ZtAf910dPTow0bNmjY\nsGF2T/FJV1eXLMu6I/6RKkkNDQ36+9//roqKCl29elUvv/yy3ZNuKiMjQxkZGZKkTZs2aeHChQEd\nd0mqra1Vb2+vKisrVV9fr5/97Gd64YUX7J51Q1VVVYqMjFRVVZX++c9/Kj8/Xzt37hyyj89D9IP4\ntHfTCwQJCQkB/QX/SXPnztVTTz0lSbIs6464h8Hs2bOVn58vSTp79mzA/6UoSYWFhVq8eLFGjhxp\n9xSfNDc36+rVq1q2bJmWLl2qo0eP2j1pUH/961+VlJSkJ554Qj/4wQ80c+ZMuyf57NixY3r33Xf1\nne98x+4pNzV69Gj19fWpv79fHo9n0HujBIJ3331XqampkqTExES1tLQM6ccP7N8dm3k8Hu9DhJIU\nHBys3t7egP6iSktL8+kWl4EiKipK0se/1ytWrFBOTo7Ni3wTEhKivLw8vf7669q+fbvdcwa1e/du\nxcXFafr06frFL35h9xyfDBs2TI899pgeffRRvffee1q+fLn2798fsH/2Ll26pLNnz2rHjh06c+aM\nsrOztX//fjkcDrun3VRJSYmeeOIJu2f4JDIyUq2trZo3b54uXbqkHTt22D1pUOPGjVNNTY1mz56t\npqYmnTt3Tn19fUN2IcMV/CA+7d30cGs++OADLV26VI888ojmz59v9xyfFRYW6sCBA1q/fr2uXLli\n95wbeu2113To0CFlZWXp5MmTysvLU1tbYN1f+5NGjx6thx9+WA6HQ6NHj1ZsbGxAb46NjdWDDz6o\nsLAwJSYmKjw8XBcvXrR71k1dvnxZp06d0gMPPGD3FJ+88sorevDBB3XgwAH99re/1apVq9TV1WX3\nrBtauHChoqOj5Xa79frrr+vLX/7ykD5KSeAHkZycrLq6Okny6W56+PTOnz+vZcuWKTc3V4sWLbJ7\njk/27t2rkpISSVJERIQcDoeCggL3j9KuXbtUVlam0tJSjRs3ToWFhRoxYoTdswb16quvel9B8ty5\nc/J4PAG9efLkyfrLX/4iy7J07tw5Xb16VbGxsXbPuqkjR45o2rRpds/wWUxMjJxOpyRp+PDh6u3t\nveELrQSCY8eOadq0aaqoqNDcuXP1pS99aUg/Ppejg/jmN7+p+vp6LV682Hs3PdxeO3bs0OXLl1Vc\nXKzi4mJJHz9RMJCfDDZnzhytXr1aS5YsUW9vr9asWRPQe+9EixYt0urVq+VyueRwOLRly5aAfvRs\n1qxZOnLkiBYtWiTLsrRhw4Y74vkkp06dUnx8vN0zfPa9731Pa9askdvtVk9Pj1auXKnIyEi7Z93Q\n3Xffreeff147duyQ0+nUM888M6QfnzvZAQBgoMB9XBEAANwyAg8AgIEIPAAABiLwAAAYiMADAGAg\nAg/gP2poaFBWVtYN379q1Srt3r37tv16AG4vAg8AgIEIPIBBvfnmm3K5XEpPT9c3vvEN/eEPf/C+\n789//rMyMjI0f/58/f73v5f08cvpFhQUKD09XQ8//LBeeeUVm5YDn2+Be2soAAGhrKxMmzdv1pgx\nY3T48GFt2bJF8+bNkyRdvXpVVVVVunDhghYuXKivfe1rqq6uliTt2bNH3d3deuyxxzR+/Hg7PwXg\nc4nAAxjUc889p5qaGu3fv19NTU0DXoApPT1dISEhGjVqlCZOnKimpiYdPnxYJ0+e1BtvvCFJunLl\nit5++23de++9dn0KwOcSgQcwKLfbralTp2rq1KmaNm2afvzjH3vf9+/3W7csS6Ghoerr61Nubq7m\nzJkjSbp48aIiIyPV1NQ05NuBzzO+Bw/ghj766CO99957euqppzRjxgzV19cPePWu3/3ud7IsS62t\nrTp27Ji+8pWv6IEHHlBVVZV6enrU0dEht9tN3AEbcAUP4IZiY2P19a9/Xd/+9rcVHR2tiRMnqrOz\nU1euXJEkRUZGKiMjQ729vfrJT36iuLg4LV68WKdPn1Z6erp6e3uVkZGhqVOnqqGhwebPBvh84dXk\nAAAwEA/RAwBgIAIPAICBCDwAAAYi8AAAGIjAAwBgIAIPAICBCDwAAAYi8AAAGOj/Az9I9QxmzKcq\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2806b1f8710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Y_train = train[\"label\"]\n",
    "\n",
    "# Drop 'label' column\n",
    "X_train = train.drop(labels = [\"label\"],axis = 1) \n",
    "\n",
    "# free some space\n",
    "del train \n",
    "\n",
    "g = sns.countplot(Y_train)\n",
    "\n",
    "Y_train.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5aea4062-1790-4987-b739-c4bebd79030f",
    "_uuid": "b7b1b1d36243c885e57374c8b60c5a7e10abe922"
   },
   "source": [
    "We have similar counts for the 10 digits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5b7d4b66-a140-4fcc-a889-bcef007c880a",
    "_uuid": "5d77934302869925c19128c77e247b3c8ca84d71"
   },
   "source": [
    "## 2.2 Check for null and missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "ececaa00-2ae3-4d13-b631-438df085b030",
    "_execution_state": "idle",
    "_uuid": "cdf27c27e2a5b15e6d7bfc70de7a18c08f3feb7a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count       784\n",
       "unique        1\n",
       "top       False\n",
       "freq        784\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the data\n",
    "X_train.isnull().any().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_cell_guid": "92572e80-8543-4e72-8767-5c9be8381b04",
    "_execution_state": "idle",
    "_uuid": "a0089bb7ec9aec76373db475399aea24699ae989"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count       784\n",
       "unique        1\n",
       "top       False\n",
       "freq        784\n",
       "dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.isnull().any().describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "09d04cae-4245-4659-85dd-ef48531da295",
    "_uuid": "c0bee59691c2df0b275c78e38e7f9907d02ac038"
   },
   "source": [
    "I check for corrupted images (missing values inside).\n",
    "\n",
    "There is no missing values in the train and test dataset. So we can safely go ahead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "6812040d-80ad-43d2-a571-275f4f20067b",
    "_uuid": "2954681f25f0dcbe986e6914396cdbce61db591f"
   },
   "source": [
    "## 2.3 Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "159d5854-437a-4d0f-bc1e-fc3f7e43d178",
    "_uuid": "0ecf4b52510ab7957d0d4eb646c0aa1ba5986273"
   },
   "source": [
    "We perform a grayscale normalization to reduce the effect of illumination's differences. \n",
    "\n",
    "Moreover the CNN converg faster on [0..1] data than on [0..255]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_cell_guid": "cdc4340b-6e24-4e12-be99-ac806098ff17",
    "_execution_state": "idle",
    "_uuid": "b5d4f8fcf2a967e2c7d57daedf95aa8c5ab7f8cb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Normalize the data\n",
    "X_train = X_train / 255.0\n",
    "test = test / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7413df94-bcb9-4f75-b174-c127d4445766",
    "_uuid": "a66741bf1ac597094f3a3166877008feef27c519"
   },
   "source": [
    "## 2.3 Reshape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_cell_guid": "34b6a5f7-8fd2-4387-8ef4-c9dc19584fed",
    "_execution_state": "idle",
    "_uuid": "f0a6ad80dab8e0f2c2e46165ccd9cd82dd162bc3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\n",
    "X_train = X_train.values.reshape(-1,28,28,1)\n",
    "test = test.values.reshape(-1,28,28,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "8decd1ce-7b7e-431d-8458-eaca18e0e1f7",
    "_uuid": "f4fb5553e188d9956f5d8b3a5d275ab00ea667ce"
   },
   "source": [
    "Train and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices. \n",
    "\n",
    "Keras requires an extra dimension in the end which correspond to channels. MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "bdb422e2-bdec-444f-97a5-283a1e54bf2c",
    "_uuid": "39b7a31e843bac6b705461bcce89da216b91799e"
   },
   "source": [
    "## 2.5 Label encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_cell_guid": "4b7f3e78-44dc-4561-b1f0-9429ee024cf4",
    "_execution_state": "idle",
    "_uuid": "cabefd1478d5c1bdfe57fd6a34395340916a854c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\n",
    "Y_train = to_categorical(Y_train, num_classes = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "ae068bd8-b12e-4768-8a7e-0fc865dd7562",
    "_uuid": "dcfb688587dfc6feafd27442a3505e35dc01b82d"
   },
   "source": [
    "Labels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "32152fc3-a570-4d64-8a7d-6c689a4acd33",
    "_uuid": "d8abbbf31483b94e1b29d07c4c8253d1311648a7"
   },
   "source": [
    "## 2.6 Split training and valdiation set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_cell_guid": "3a698301-9759-4279-ae48-fd980f89ea53",
    "_execution_state": "idle",
    "_uuid": "6e51c925c6e0f1b936679c9649fef345c853555f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the random seed\n",
    "random_seed = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_cell_guid": "dcd25ebb-d845-4d32-9867-082e352b1396",
    "_execution_state": "idle",
    "_uuid": "b779ac76d8317647db92d5a88b4098d212d72884",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Split the train and the validation set for the fitting\n",
    "X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=random_seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "3c99964f-4500-4f1c-947d-c67e644e34db",
    "_uuid": "72ed54a305eebf80e8b8b0f7eabde1332a5a85a3"
   },
   "source": [
    "I choosed to split the train set in two parts : a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model.\n",
    "\n",
    "Since we have 42 000 training images of balanced labels (see 2.1 Load data), a random split of the train set doesn't cause some labels to be over represented in the validation set. Be carefull with some unbalanced dataset a simple random split could cause inaccurate evaluation during the validation. \n",
    "\n",
    "To avoid that, you could use stratify = True option in train_test_split function (**Only for >=0.17 sklearn versions**)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "adbeacf0-0dc0-4675-b2df-9c9663750f32",
    "_uuid": "60eed15ec5bc0d354385301789ecb8538fc02267"
   },
   "source": [
    "We can get a better sense for one of these examples by visualising the image and looking at the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_cell_guid": "5f76131b-4ba0-45f1-a98c-bd4e7d561793",
    "_execution_state": "idle",
    "_uuid": "e0dae8943d3d35f075dba3d7ba31bde1d4bf2ff4"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAFJCAYAAAASfw+VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEWdJREFUeJzt3V9I1fcfx/HX8Vg4PJgXCQWWf/rDSAkJaeDIbhJdW7PA\n5kwUUlZKULJlmbPVpljRdiWmLLoqq0lXwVYbC0LI1kXM4ig1yBasVRis2ZFKy+/v4sfc+mdvT+d7\nzjGfj6u0z77f97cznn0953w6HsdxHAEAxhUT6QEAYDIglgBgQCwBwIBYAoABsQQAg1i3T/Dw4UP5\n/X4lJSXJ6/W6fToACMqTJ080MDCgzMxMxcXFPff7rsfS7/ertLTU7dMAQEh0dHQoOzv7ue+7Hsuk\npKSxAWbNmuX26QAgKLdv31ZpaelYs54VVCxHR0e1e/duXb16VdOnT1dTU5NSUlJeuPafH71nzZql\n5OTkYE4HAGHzsqcLg3qB5+eff9bw8LC+++47ffbZZ9q7d+9rDQcA0S6oWF68eFHLli2TJGVlZcnv\n94d0KACINkHFMhAIyOfzjX3t9Xr1+PHjkA0FANEmqFj6fD4NDQ2NfT06OqrYWNdfKwKAiAkqlkuW\nLFFXV5ckqaenRwsXLgzpUAAQbYK6HczLy9O5c+f08ccfy3EcNTc3h3ouAIgqQcUyJiZGX331Vahn\nAYCoxd5wADAglgBgQCwBwIBYAoABsQQAA2IJAAbEEgAMiCUAGBBLADAglgBgQCwBwIBYAoABsQQA\nA2IJAAbEEgAMiCUAGBBLADAglgBgQCwBwIBYAoABsQQAA2IJAAbEEgAMiCUAGBBLADAglgBgQCwB\nwIBYAoABsQQAA2IJAAbEEgAMiCUAGBBLADAglgBgQCwBwIBYAoABsQQAA2IJAAbEEgAMiCUAGBBL\nADAglgBgQCwBwCA20gMAwejv7zev7ezsNK/dsWOHaV1BQYH5mKmpqea1tbW15rXp6enmtXh93FkC\ngEHQd5Zr1qyRz+eTJCUnJ2vPnj0hGwoAok1QsXz06JEcx9Hhw4dDPQ8ARKWgfgy/cuWKHjx4oIqK\nCpWXl6unpyfUcwFAVAnqzjIuLk6VlZVau3atfv/9d33yySc6ffq0YmN5vQjAmymouqWlpSklJUUe\nj0dpaWlKTEzUwMCAZs+eHer5ACAqBPVj+IkTJ7R3715J0p07dxQIBJSUlBTSwQAgmgR1Z1lUVKQd\nO3aopKREHo9Hzc3N/AgO4I0WVOGmT5+ub775JtSzAEDU4k3pAGDAz86IKseOHTOtu3HjhvmYH330\nkXltXV2dea1Vd3e3ee2KFSvMazds2GBa58Y1TUXcWQKAAbEEAANiCQAGxBIADIglABgQSwAwIJYA\nYEAsAcCAWAKAAbEEAAO2OyIog4OD5rXbt283r83NzTWtm0xb+HJycsxrJ/KpA9Y/1+rqavMx29ra\nzGunGu4sAcCAWAKAAbEEAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAHTwISnFxsXnt8uXLzWtL\nSkqCGeeNkZCQYF67b98+07qsrCzzMSeyM2sis74JuLMEAANiCQAGxBIADIglABgQSwAwIJYAYEAs\nAcCAWAKAAbEEAANiCQAGbHfEU/r7+03rrl69aj7mqVOngh0H4/D7/aZ1169fNx9zqm1hnAjuLAHA\ngFgCgAGxBAADYgkABsQSAAyIJQAYEEsAMCCWAGBALAHAgFgCgAHbHfGUefPmmdZdu3bN5UmmJut2\nU0l69913Q35+j8djXltVVWVa19bWFuw4UYU7SwAwMMXy0qVLKisrkyTduHFDJSUlWrdunXbt2qXR\n0VFXBwSAaPDKWB48eFANDQ169OiRJGnPnj2qqanR0aNH5TiOzpw54/qQABBpr4zl3Llz1dLSMvZ1\nb2+vli5dKknKzc1Vd3e3e9MBQJR4ZSzz8/MVG/vv60CO44w9CRwfH6/79++7Nx0ARIkJv8ATE/Pv\nfzI0NMQ/FgpgSphwLBctWqQLFy5Ikrq6upSdnR3yoQAg2kw4ltu3b1dLS4uKi4s1MjKi/Px8N+YC\ngKhielN6cnKyOjs7JUlpaWk6cuSIq0MBQLRhB88UMJFdIVbp6ekhP+abaiJ//tYdVG6x7sqRpH37\n9rk4SfRhBw8AGBBLADAglgBgQCwBwIBYAoABsQQAA2IJAAbEEgAMiCUAGBBLADBgu+MUcPv27UiP\nMGlYtyZu2rTJfMzTp08HO864CgoKTOtaW1vNx2Qb68txZwkABsQSAAyIJQAYEEsAMCCWAGBALAHA\ngFgCgAGxBAADYgkABsQSAAzY7jgFzJo1K+THnMgnFrqxhW5wcNC8tri42LzWujUxLS3NfMyjR4+a\n177//vvmtQkJCea1eH3cWQKAAbEEAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAWAKAATt4poCJ\n7KCpqqoyrZs3b575mH///bd57ffff29at27dOvMxJ+LcuXOmdTk5Oa6cH9GLO0sAMCCWAGBALAHA\ngFgCgAGxBAADYgkABsQSAAyIJQAYEEsAMCCWAGDAdkc8pa2tzbSuvb3dfMwZM2YEO85L8SFgCDfu\nLAHAwBTLS5cuqaysTJLU19enZcuWqaysTGVlZfrhhx9cHRAAosErfww/ePCgTp48qbfeekuS1Nvb\nq/Xr16uiosL14QAgWrzyznLu3LlqaWkZ+9rv9+vs2bMqLS1VfX29AoGAqwMCQDR4ZSzz8/MVG/vv\nDejixYu1bds2dXR0aM6cOWptbXV1QACIBhN+gScvL0+ZmZljv+7r6wv5UAAQbSYcy8rKSl2+fFmS\ndP78eWVkZIR8KACINhN+n+Xu3bvV2NioadOmaebMmWpsbHRjLgCIKqZYJicnq7OzU5KUkZGh48eP\nuzoUAEQb3pQOAAZsd8RTjh07FvJjFhQUmNeePn3atI4tjAg37iwBwIBYAoABsQQAA2IJAAbEEgAM\niCUAGBBLADAglgBgQCwBwIBYAoAB2x2ngPfee8+8NjU11bTu2rVr5mOmp6eb1/b395vWZWVlmY+5\nYcMG89q6ujrzWkwt3FkCgAGxBAADYgkABsQSAAyIJQAYEEsAMCCWAGBALAHAgFgCgAE7eKLM4OCg\naZ3f7zcf07orR5La2trMa91g3e3T09NjPuaMGTPMa9nBg5fhzhIADIglABgQSwAwIJYAYEAsAcCA\nWAKAAbEEAANiCQAGxBIADIglABiw3THKWD+I68iRI+ZjRnoLoxsSEhIiPQKmGO4sAcCAWAKAAbEE\nAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAWAKAAdsdw+DYsWMhP2ZOTk7IjzmZWD8FEwgV7iwB\nwGDcO8uRkRHV19fr5s2bGh4eVnV1tebPn6+6ujp5PB4tWLBAu3btUkwMzQXwZhs3lidPnlRiYqL2\n79+ve/fuafXq1Xr77bdVU1Ojd955R1988YXOnDmjvLy8cM0LABEx7i1hQUGBtmzZIklyHEder1e9\nvb1aunSpJCk3N1fd3d3uTwkAETZuLOPj4+Xz+RQIBLR582bV1NTIcRx5PJ6x379//35YBgWASHrl\nk423bt1SeXm5CgsLtWrVqqeenxwaGuIfYQUwJYwby7t376qiokK1tbUqKiqSJC1atEgXLlyQJHV1\ndSk7O9v9KQEgwsaNZXt7uwYHB3XgwAGVlZWprKxMNTU1amlpUXFxsUZGRpSfnx+uWQEgYsZ9Nbyh\noUENDQ3PfX8in/8CAG8CdvBEman+F5F1Z471g90kKS0tLdhxgDG8mxwADIglABgQSwAwIJYAYEAs\nAcCAWAKAAbEEAANiCQAGxBIADIglABiw3TEM1q1bZ1577do1FyeJjP7+fvPaFStWhPz8PT09IT8m\nph7uLAHAgFgCgAGxBAADYgkABsQSAAyIJQAYEEsAMCCWAGBALAHAgFgCgAHbHcPg3Llz5rXW7X5u\nfQrk4cOHTeva29tdOf/Ro0dN60pKSlw5P/Ay3FkCgAGxBAADYgkABsQSAAyIJQAYEEsAMCCWAGBA\nLAHAgFgCgAE7eMIgJyfHvHbDhg2mddadNtLEdttUVVWZ1k1kV1JmZqZ5bUJCgnktEE7cWQKAAbEE\nAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAWAKAAbEEAAO2O0aZurq6kB+zra0t5McEphruLAHA\nYNw7y5GREdXX1+vmzZsaHh5WdXW1Zs+erY0bNyo1NVXS/z+SdOXKleGYFQAiZtxYnjx5UomJidq/\nf7/u3bun1atXa9OmTVq/fr0qKirCNSMARNy4sSwoKFB+fr4kyXEceb1e+f1+Xb9+XWfOnFFKSorq\n6+vl8/nCMiwARMq4z1nGx8fL5/MpEAho8+bNqqmp0eLFi7Vt2zZ1dHRozpw5am1tDdesABAxr3yB\n59atWyovL1dhYaFWrVqlvLy8sX/MNS8vT319fa4PCQCRNm4s7969q4qKCtXW1qqoqEiSVFlZqcuX\nL0uSzp8/r4yMDPenBIAIG/c5y/b2dg0ODurAgQM6cOCApP+/D7C5uVnTpk3TzJkz1djYGJZBASCS\nxo1lQ0ODGhoanvv+8ePHXRsIAKIRb0oHAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAWAKAAbEE\nAANiCQAGxBIADIglABgQSwAwIJYAYEAsAcCAWAKAAbEEAANiCQAGxBIADIglABgQSwAwGPfTHUPh\nyZMnkqTbt2+7fSoACNo/jfqnWc9yPZYDAwOSpNLSUrdPBQCvbWBgQCkpKc993+M4juPmiR8+fCi/\n36+kpCR5vV43TwUAQXvy5IkGBgaUmZmpuLi4537f9VgCwJuAF3gAwIBYAoABsQQAA2IJAAauv3Xo\nWaOjo9q9e7euXr2q6dOnq6mp6YUv009Ga9askc/nkyQlJydrz549EZ4oeJcuXdLXX3+tw4cP68aN\nG6qrq5PH49GCBQu0a9cuxcRMvr9n/3tNfX192rhxo1JTUyVJJSUlWrlyZWQHnKCRkRHV19fr5s2b\nGh4eVnV1tebPnz+pH6sXXdPs2bOj47FywuzHH390tm/f7jiO4/z6669OVVVVuEdwxcOHD53CwsJI\njxES3377rfPBBx84a9eudRzHcTZu3Oj88ssvjuM4zs6dO52ffvopkuMF5dlr6uzsdA4dOhThqV7P\niRMnnKamJsdxHOevv/5yli9fPukfqxddU7Q8VmH/K+fixYtatmyZJCkrK0t+vz/cI7jiypUrevDg\ngSoqKlReXq6enp5IjxS0uXPnqqWlZezr3t5eLV26VJKUm5ur7u7uSI0WtGevye/36+zZsyotLVV9\nfb0CgUAEpwtOQUGBtmzZIklyHEder3fSP1YvuqZoeazCHstAIDD2o6okeb1ePX78ONxjhFxcXJwq\nKyt16NAhffnll9q6deukva78/HzFxv77DI3jOPJ4PJKk+Ph43b9/P1KjBe3Za1q8eLG2bdumjo4O\nzZkzR62trRGcLjjx8fHy+XwKBALavHmzampqJv1j9aJripbHKuyx9Pl8GhoaGvt6dHT0qf+JJ6u0\ntDR9+OGH8ng8SktLU2Ji4thWz8nuv895DQ0NKSEhIYLThEZeXp4yMzPHft3X1xfhiYJz69YtlZeX\nq7CwUKtWrXojHqtnrylaHquwx3LJkiXq6uqSJPX09GjhwoXhHsEVJ06c0N69eyVJd+7cUSAQUFJS\nUoSnCo1FixbpwoULkqSuri5lZ2dHeKLXV1lZqcuXL0uSzp8/r4yMjAhPNHF3795VRUWFamtrVVRU\nJGnyP1YvuqZoeazCvt3xn1fDf/vtNzmOo+bmZs2bNy+cI7hieHhYO3bs0J9//imPx6OtW7dqyZIl\nkR4raH/88Yc+/fRTdXZ26vr169q5c6dGRkaUnp6upqamSbnP/7/X1Nvbq8bGRk2bNk0zZ85UY2Pj\nU08PTQZNTU06deqU0tPTx773+eefq6mpadI+Vi+6ppqaGu3fvz/ijxV7wwHAYPK8AQsAIohYAoAB\nsQQAA2IJAAbEEgAMiCUAGBBLADAglgBg8D/EnQHhLaqzSwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2806a17ec18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Some examples\n",
    "g = plt.imshow(X_train[0][:,:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. CNN\n",
    "## 3.1 Make custom activation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we know  SWISH activation function recently published by a team at Google. \n",
    "If you are not familiar with the Swish activation (mathematically, f(x)=x*sigmoid(x))\n",
    "> Read [Swish: a Self-Gated Activation Function](https://arxiv.org/abs/1710.05941)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def swish(x):\n",
    "    return (K.sigmoid(x) * x)\n",
    "\n",
    "get_custom_objects().update({'swish': swish})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d5265777-aeb3-449d-b171-d88cad74c0a4",
    "_uuid": "5fa18b37a9acd9e098bac1d12264b0dd4310fdd3"
   },
   "source": [
    "## 3.1 Define the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "504fa00e-148c-4364-9b68-218b3aaedfdb",
    "_uuid": "7697570491420f957f6e4d3569d51410b5277250"
   },
   "source": [
    "I used the Keras Sequential API, where you have just to add one layer at a time, starting from the input.\n",
    "\n",
    "The first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 32 filters for the two firsts conv2D layers and 64 filters for the two last ones. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.\n",
    "\n",
    "The CNN can isolate features that are useful everywhere from these transformed images (feature maps).\n",
    "\n",
    "The second important layer in CNN is the pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important. \n",
    "\n",
    "Combining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.\n",
    "\n",
    "Dropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting. \n",
    "\n",
    "'Swish' is the ( Swish activation (mathematically, f(x)=x*sigmoid(x))).\n",
    "\n",
    "The Flatten layer is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.\n",
    "\n",
    "In the end i used the features in two fully-connected (Dense) layers which is just artificial an neural networks (ANN) classifier. In the last layer(Dense(10,activation=\"sigmoid\")) the net outputs distribution of probability of each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_cell_guid": "1e0f3f88-2ad7-459e-8e02-aecc5f3511ae",
    "_execution_state": "idle",
    "_uuid": "f7991ef6871a26f9fa57acdcd460a69bab53e804",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the CNN model \n",
    "# my CNN architechture is In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> Flatten -> Dense -> Dropout -> Out\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n",
    "                 activation ='swish', input_shape = (28,28,1)))\n",
    "model.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n",
    "                 activation ='swish'))\n",
    "model.add(MaxPool2D(pool_size=(2,2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "\n",
    "model.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n",
    "                 activation ='swish'))\n",
    "model.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n",
    "                 activation ='swish'))\n",
    "model.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(256, activation = \"swish\"))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(10, activation = \"sigmoid\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "0e26b6a3-ef9d-4ae5-9d79-d5e4b073251a",
    "_uuid": "7065512cf892ba49f0c06e239e3a20ff13667ef2"
   },
   "source": [
    "## 3.2 Set the optimizer and annealer\n",
    "\n",
    "Once our layers are added to the model, we need to set up a score function, a loss function and an optimisation algorithm.\n",
    "\n",
    "We define the loss function to measure how poorly our model performs on images with known labels. It is the error rate between the oberved labels and the predicted ones. We use a specific form for categorical classifications (>2 classes) called the \"categorical_crossentropy\".\n",
    "\n",
    "The most important function is the optimizer. This function will iteratively improve parameters (filters kernel values, weights and bias of neurons ...) in order to minimise the loss. \n",
    "\n",
    "I choosed adam (with default values), This is used to perform optimization and is one of the best optimizer at present. The author claims that it inherits from RMSProp and AdaGrad (Well it inherits from them).\n",
    "\n",
    "The metric function \"accuracy\" is used is to evaluate the performance our model.\n",
    "This metric function is similar to the loss function, except that the results from the metric evaluation are not used when training the model (only for evaluation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_cell_guid": "a4c55409-6a65-400a-b5e8-a1dc535429c0",
    "_execution_state": "idle",
    "_uuid": "420c704367b397b8255fefe9d882b35ac8929b95",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the optimizer\n",
    "optimizer = adam(lr=0.001, epsilon=1e-08, decay=0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "_cell_guid": "0bd89b79-1a74-40da-af9a-7a76a8b96ff2",
    "_execution_state": "idle",
    "_uuid": "b4c0288622227a3cd05479aa765e324dbb852f34",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "6bc49be3-11e5-48fb-b70f-601aec0f0c86",
    "_uuid": "f4d89b29d55b328e8c446880505c54e674eeec38"
   },
   "source": [
    "<img src=\"http://img1.imagilive.com/0717/learningrates.jpg\"> </img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "35b826b6-c516-47d0-8469-f8d04a102118",
    "_uuid": "bb90d48f4426d1baeb0404f2431e04e3cade59df"
   },
   "source": [
    "In order to make the optimizer converge faster and closest to the global minimum of the loss function, i used an annealing method of the learning rate (LR).\n",
    "\n",
    "The LR is the step by which the optimizer walks through the 'loss landscape'. The higher LR, the bigger are the steps and the quicker is the convergence. However the sampling is very poor with an high LR and the optimizer could probably fall into a local minima.\n",
    "\n",
    "Its better to have a decreasing learning rate during the training to reach efficiently the global minimum of the loss function. \n",
    "\n",
    "To keep the advantage of the fast computation time with a high LR, i decreased the LR dynamically every X steps (epochs) depending if it is necessary (when accuracy is not improved).\n",
    "\n",
    "With the ReduceLROnPlateau function from Keras.callbacks, i choose to reduce the LR by half if the accuracy is not improved after 3 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "_cell_guid": "b5987a18-6bbe-42a2-9d31-333ebc4f7af1",
    "_execution_state": "idle",
    "_uuid": "c4a5b4e462ec5362c47eef4fcc7956fd4e203307",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set a learning rate annealer\n",
    "learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n",
    "                                            patience=3, \n",
    "                                            verbose=1, \n",
    "                                            factor=0.5, \n",
    "                                            min_lr=0.00001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_cell_guid": "970db455-b393-4b25-806d-92c6766c12c0",
    "_execution_state": "idle",
    "_uuid": "26b0647c46efdb6b1096cf7335a7bf2a3417543a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epochs = 30 # Turn epochs to 30 to get 0.9967 accuracy\n",
    "batch_size = 86"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "241a0f4f-13f5-4b13-be1e-4e3e4a714c06",
    "_uuid": "f24df64b223e0177c94025b6767ab19b722c5386"
   },
   "source": [
    "## 3.3 Data augmentation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e2d41e30-0724-40fb-a901-750e514ba9f9",
    "_uuid": "9e498b91419439f0fa791e595f202d9a0d56ad6b"
   },
   "source": [
    "In order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n",
    "\n",
    "For example, the number is not centered \n",
    "The scale is not the same (some who write with big/small numbers)\n",
    "The image is rotated...\n",
    "\n",
    "Approaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more. \n",
    "\n",
    "By applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.\n",
    "\n",
    "The improvement is important : \n",
    "   - Without data augmentation i obtained an accuracy of 98.114%\n",
    "   - With data augmentation i achieved 99.67% of accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_cell_guid": "ade93d2d-90c6-4401-af95-f7d65f8c0a20",
    "_execution_state": "idle",
    "_uuid": "d4e9e1ade3c04f9ca4d8cd44e799f9e09524d5a1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Without data augmentation i obtained an accuracy of 0.98114\n",
    "#history = model.fit(X_train, Y_train, batch_size = batch_size, epochs = epochs, \n",
    "#          validation_data = (X_val, Y_val), verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "_cell_guid": "b342befe-1a6f-44bf-8dab-28033a729122",
    "_execution_state": "idle",
    "_uuid": "21d6192c87d92d497c797656474bccd9cefc5647",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# With data augmentation to prevent overfitting (accuracy 0.99286)\n",
    "\n",
    "datagen = ImageDataGenerator(\n",
    "        featurewise_center=False,  # set input mean to 0 over the dataset\n",
    "        samplewise_center=False,  # set each sample mean to 0\n",
    "        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n",
    "        samplewise_std_normalization=False,  # divide each input by its std\n",
    "        zca_whitening=False,  # apply ZCA whitening\n",
    "        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n",
    "        zoom_range = 0.1, # Randomly zoom image \n",
    "        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n",
    "        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n",
    "        horizontal_flip=False,  # randomly flip images\n",
    "        vertical_flip=False)  # randomly flip images\n",
    "\n",
    "\n",
    "datagen.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "22f80b8a-d4f6-4a34-b33d-ff7334f45d94",
    "_uuid": "51f16d0a5b9d9373438474e7defa7348359d7c18"
   },
   "source": [
    "For the data augmentation, i choosed to :\n",
    "   - Randomly rotate some training images by 10 degrees\n",
    "   - Randomly  Zoom by 10% some training images\n",
    "   - Randomly shift images horizontally by 10% of the width\n",
    "   - Randomly shift images vertically by 10% of the height\n",
    "   \n",
    "I did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.\n",
    "\n",
    "Once our model is ready, we fit the training dataset ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_cell_guid": "b453af8d-9736-43e3-b486-7a1cd7dd8909",
    "_execution_state": "idle",
    "_uuid": "cf36b3d029f95b553be02d612e097a9769ee8252"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "60s - loss: 0.5097 - acc: 0.8410 - val_loss: 0.0592 - val_acc: 0.9819\n",
      "Epoch 2/30\n",
      "57s - loss: 0.1557 - acc: 0.9547 - val_loss: 0.0387 - val_acc: 0.9890\n",
      "Epoch 3/30\n",
      "57s - loss: 0.1156 - acc: 0.9668 - val_loss: 0.0449 - val_acc: 0.9857\n",
      "Epoch 4/30\n",
      "57s - loss: 0.0947 - acc: 0.9716 - val_loss: 0.0417 - val_acc: 0.9898\n",
      "Epoch 5/30\n",
      "57s - loss: 0.0864 - acc: 0.9755 - val_loss: 0.0318 - val_acc: 0.9898\n",
      "Epoch 6/30\n",
      "57s - loss: 0.0778 - acc: 0.9776 - val_loss: 0.0277 - val_acc: 0.9926\n",
      "Epoch 7/30\n",
      "57s - loss: 0.0726 - acc: 0.9782 - val_loss: 0.0272 - val_acc: 0.9919\n",
      "Epoch 8/30\n",
      "57s - loss: 0.0671 - acc: 0.9800 - val_loss: 0.0247 - val_acc: 0.9924\n",
      "Epoch 9/30\n",
      "57s - loss: 0.0659 - acc: 0.9804 - val_loss: 0.0250 - val_acc: 0.9919\n",
      "Epoch 10/30\n",
      "\n",
      "Epoch 00009: reducing learning rate to 0.0005000000237487257.\n",
      "57s - loss: 0.0636 - acc: 0.9815 - val_loss: 0.0228 - val_acc: 0.9919\n",
      "Epoch 11/30\n",
      "57s - loss: 0.0503 - acc: 0.9860 - val_loss: 0.0210 - val_acc: 0.9933\n",
      "Epoch 12/30\n",
      "57s - loss: 0.0417 - acc: 0.9874 - val_loss: 0.0173 - val_acc: 0.9938\n",
      "Epoch 13/30\n",
      "57s - loss: 0.0428 - acc: 0.9877 - val_loss: 0.0202 - val_acc: 0.9936\n",
      "Epoch 14/30\n",
      "57s - loss: 0.0390 - acc: 0.9883 - val_loss: 0.0199 - val_acc: 0.9950\n",
      "Epoch 15/30\n",
      "57s - loss: 0.0404 - acc: 0.9878 - val_loss: 0.0200 - val_acc: 0.9940\n",
      "Epoch 16/30\n",
      "57s - loss: 0.0385 - acc: 0.9885 - val_loss: 0.0160 - val_acc: 0.9945\n",
      "Epoch 17/30\n",
      "57s - loss: 0.0383 - acc: 0.9887 - val_loss: 0.0167 - val_acc: 0.9938\n",
      "Epoch 18/30\n",
      "\n",
      "Epoch 00017: reducing learning rate to 0.0002500000118743628.\n",
      "57s - loss: 0.0396 - acc: 0.9883 - val_loss: 0.0191 - val_acc: 0.9936\n",
      "Epoch 19/30\n",
      "58s - loss: 0.0308 - acc: 0.9911 - val_loss: 0.0155 - val_acc: 0.9943\n",
      "Epoch 20/30\n",
      "57s - loss: 0.0299 - acc: 0.9914 - val_loss: 0.0191 - val_acc: 0.9945\n",
      "Epoch 21/30\n",
      "\n",
      "Epoch 00020: reducing learning rate to 0.0001250000059371814.\n",
      "57s - loss: 0.0270 - acc: 0.9910 - val_loss: 0.0148 - val_acc: 0.9948\n",
      "Epoch 22/30\n",
      "57s - loss: 0.0273 - acc: 0.9920 - val_loss: 0.0142 - val_acc: 0.9955\n",
      "Epoch 23/30\n",
      "57s - loss: 0.0248 - acc: 0.9924 - val_loss: 0.0135 - val_acc: 0.9950\n",
      "Epoch 24/30\n",
      "57s - loss: 0.0254 - acc: 0.9919 - val_loss: 0.0152 - val_acc: 0.9957\n",
      "Epoch 25/30\n",
      "57s - loss: 0.0225 - acc: 0.9932 - val_loss: 0.0143 - val_acc: 0.9948\n",
      "Epoch 26/30\n",
      "57s - loss: 0.0246 - acc: 0.9924 - val_loss: 0.0153 - val_acc: 0.9948\n",
      "Epoch 27/30\n",
      "57s - loss: 0.0257 - acc: 0.9926 - val_loss: 0.0156 - val_acc: 0.9952\n",
      "Epoch 28/30\n",
      "\n",
      "Epoch 00027: reducing learning rate to 6.25000029685907e-05.\n",
      "57s - loss: 0.0237 - acc: 0.9926 - val_loss: 0.0167 - val_acc: 0.9940\n",
      "Epoch 29/30\n",
      "57s - loss: 0.0221 - acc: 0.9931 - val_loss: 0.0135 - val_acc: 0.9957\n",
      "Epoch 30/30\n",
      "57s - loss: 0.0217 - acc: 0.9934 - val_loss: 0.0128 - val_acc: 0.9964\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "history = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),\n",
    "                              epochs = epochs, validation_data = (X_val,Y_val),\n",
    "                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n",
    "                              , callbacks=[learning_rate_reduction])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e758621d-b27b-40ff-a93f-bebd2e0e5243",
    "_uuid": "0a1834f2a9f2db15dcaba4a84004b9627d714469"
   },
   "source": [
    "# 4. Evaluate the model\n",
    "## 4.1 Training and validation curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_cell_guid": "eb4b1b73-cbd4-40e8-9790-066fcef4c4c0",
    "_execution_state": "idle",
    "_uuid": "3a831860dd5bb65c8ead1ddcf4ae18ae20dd7f3e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAFJCAYAAAChG+XKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8FNXB//HPZjf3TUiAcFEkQDSAIo2p1VrFKrfyYEUp\nQoIIfYQXWltvVRRFoUEoRKNSRcTKQytQFIQHLfh4qSCWX6NSRaOEq1xLuCXchM1ts9n9/THsZjfZ\nXEgCu8l+36/Xec3szO7mZDPZ75wzZ2ZMLpfLhYiIiASNsEBXQERERHwpnEVERIKMwllERCTIKJxF\nRESCjMJZREQkyFgCXQGAsrIy8vPzSUpKwmw2B7o6IiIi51VlZSVFRUX06dOHqKioGuuDIpzz8/MZ\nM2ZMoKshIiJyQS1dupSrr766xvKgCOekpCTAqGSnTp0CXBsREZHz68iRI4wZM8aTf9UFRTi7u7I7\ndepEly5dAlwbERGRC6O2Q7kaECYiIhJkFM4iIiJBplHd2k6nk6ysLHbs2EFERAQzZ84kOTnZs/6N\nN95gxYoVtG3bFoDp06fTo0eP5qlxA7zwAmzaBG++ecF+pIiISLNpVDivXbsWu93O8uXLycvLIzs7\nm/nz53vW5+fn8+yzz9KnT59mq+i52LABVq+Gl16CWo61i4iIBK1GdWtv2rSJfv36AZCWlkZ+fr7P\n+i1btvD6668zevRo/vznPze9lueoVy9jum3bBf/RIiIiTdaocLbZbFitVs9js9mMw+HwPL7lllvI\nyspi0aJFbNq0ifXr1ze9puegd29jun37Bf2xIiIizaJR4Wy1WikuLvY8djqdWCxGD7nL5eLXv/41\nbdu2JSIigp///Ods3bq1eWrbQGo5i4gEXnZ2NmPHjmXIkCHcdNNNjB07lgcffLBBr922bRuvvPJK\nres3bNjA8uXLG123goICRo0a1ejXn2+NOuacnp7O+vXrGTp0KHl5eaSmpnrW2Ww2fvnLX/L+++8T\nExPDxo0bGTFiRLNVuCHcLWeFs4hI4DzxxBMArFq1ij179jBp0qQGv7Z37970dn+Z+3HjjTc2uX7B\nrFHhPGjQIHJzc8nMzMTlcjFr1izWrFlDSUkJGRkZ/P73v2fcuHFERERw3XXX8fOf/7y5612nNm2g\nc2d1a4uIuD32GKxY0bzvOXIk5OSc++s2btzI888/T3h4OKNGjSIqKoqlS5ficDgwmUy88sorfP/9\n9yxbtow5c+YwePBg0tPT2bt3L+3atWPu3Ln8/e9/Z8+ePWRmZvLoo4/SqVMnDhw4wJVXXsn06dM5\nceIEkyZNwm630717d7744gs+/vhjv/XJzc3lT3/6E5GRkSQkJDBr1iwcDgcPP/wwLpeL8vJyz1lH\nDz30EDabjdLSUn7/+99zww03NPFT9K9R4RwWFsYzzzzjsywlJcUzf/vtt3P77bc3rWZN1KsXrF8P\nxcUQGxvQqoiISDXl5eWsOLu38Nprr/H6668THR3NtGnT+Ne//kXHjh09zz1w4ACLFi2ic+fOZGZm\nsnnzZp/32rdvHwsXLiQ6OpqBAwdSVFTEggULGDBgAGPGjCE3N5fc3Fy/9XC5XEydOpW33nqLjh07\nsmjRIubPn8+1115LQkICzz33HLt27aKkpIT//Oc/nDp1iv/5n//h+PHj7Nu377x9PkFx+c7zoXdv\nI5x37ID09EDXRkQksHJyGtfKPV+6d+/umW/Xrh2TJ08mNjaWPXv2kJaW5vPcxMREOnfuDEDnzp0p\nLy/3Wd+1a1fPIOWkpCTKy8vZvXs3w4cPB/B7Ywm3kydPYrVaPTsDP/nJT3jxxRd57LHH2LdvH7/9\n7W+xWCzcd999XHbZZWRkZPDII4/gcDgYO3Zs0z+IWrTqcAbjuLPCWUQkuISFGeORz5w5w8svv8yn\nn34KwN13343L5fJ5rslkqvO9/K1PTU3lm2++oXfv3uTl5dX62sTERGw2G4WFhXTo0IF///vfdOvW\njY0bN9KhQwf+8pe/8M033/Diiy/y9NNPU1xczOuvv05hYSGZmZncfPPN5/ibN0yrDWf3iG0ddxYR\nCV5Wq5X09HQyMjKwWCzEx8dTWFjY5JsgTZw4kccff5wPPviADh06eM4oqs5kMjFz5kweeOABTCYT\nbdq0Yfbs2ZhMJh555BHeeustHA4Hv/vd7+jWrRvz5s3jgw8+wOl0NnjkeWOYXNV3UQKgoKCAAQMG\nsG7duma7K9XBg9ClC4wYAStXNstbiohIC/HPf/6TxMRE+vbty2effcZrr73G4sWLA10tj/pyr9W2\nnC+6COLidDqViEgo6tKlC1OmTMFsNuN0OnnqqacCXaVz0mrD2WQyjjt/8w04HFBLj4aIiLRCKSkp\nTbpISaC16ltG9uoFFRWwZ0+gayIiItJwrTqcdaUwERFpiRTOIiIiQaZVh7NOpxIRkZaoVYdzSgqE\nh6vlLCIS7MaOHcvu3btZtWoV69atq7H++uuvr/P1H3/8MUePHqWoqIisrKwm1aV///41rkJ2obXq\ncLZY4LLLjHAO/NncIiJSn1/96lcMGDDgnF+3ePFibDYbSUlJTQ7nYNDqTzDq3Ru2boXDh41zn0VE\nQlIAbkt1//33M27cOK655ho2b97Mq6++Sk5ODk899RRnzpyhsLCQO++8kzvvvNPzmrlz59K+fXtG\njRrF1KlT2bVrF5dccgl2ux2AnTt3kp2dTWVlJSdPniQrK4vTp0+zbds2Jk+eTE5ODpMnT+btt9/2\ne7epbdu2sWDBAsLDwykoKGDo0KHcd999futfUFDAlClTqKysxGQy8fTTT9OrVy+efPJJ9u/fT1lZ\nGePGjeP2229nzpw5bNy4EYfDweDBg7nnnnua9NG2+nB2H3fetk3hLCJyIY0cOZJ33nmHa665hlWr\nVjFq1Cj279/PLbfcwuDBgzl69Chjx471CWe3jz/+mPLyct5++20OHTrERx99BMCuXbuYPHkyPXv2\nZM2aNaxatYqZM2fSu3dvsrKyCA8PB2q/29RNN93EoUOHWL16NXa7nX79+tUazs899xzjxo1j4MCB\nbNu2jSlTprB48WK+/PJL3n77bQDP3a7WrFnD4sWL6dChA6tWrWryZ9fqw9l7xHYjekpERFqHANyW\nql+/fuTk5HDq1Cm++uornn76aY4dO8aiRYv4xz/+gdVqxeFw+H3tvn376Nu3LwAXXXSR565UHTp0\n4NVXXyUqKori4mLP3aiqq+1uUzfddBOpqalYLBYsFgtRUVG11n/37t385Cc/AaB3794cOXIEq9XK\nlClTmDp1KjabjWHDhgGQk5PDCy+8wLFjx+jXr1/jPjAvrfqYM1SFs0Zsi4hcWGFhYQwZMoSsrCwG\nDhyI2WzmL3/5C2lpaTz//PMMGTKkxh2o3C699FLP3aSOHj3K0aNHAfjjH//Igw8+yLPPPktqaqrn\n9SaTyee9vO82BXjuNuV+bkOkpKTw1VdfAbBt2zbat29PYWEhW7ZsYd68ebz++uvk5ORgt9v58MMP\nefHFF1m8eDHvvPMOBw8ePPcPzEurbzn37GlMNWJbROTCGzFiBAMHDvR0S998883MnDmT999/n7i4\nOMxms+d4srcBAwaQm5vLyJEjueiii0hMTARg2LBhPPTQQ8THx9OpUydOnjwJwFVXXcXjjz/OjBkz\ngNrvNvX99983uO6PP/44U6dO5S9/+QsOh4M//vGPJCUlUVRURGZmJmFhYYwfP56IiAjatGnDqFGj\niIqK4vrrr+eiJh5HbdRdqZxOJ1lZWezYsYOIiAhmzpxJcnJyjedNnTqVNm3aMGnSpDrf73zclcpb\ncrJxGc9Dh5r9rUVERM5ZfbnXqG7ttWvXYrfbWb58OY8++ijZ2dk1nrNs2TJ27tzZmLdvdr17G6O1\nf/gh0DURERGpX6PCedOmTZ4D3mlpaeTn5/us//rrr/n222/JyMhoeg2bgY47i4hIS9KocLbZbD4j\n5Mxms2fEXWFhIfPmzWPatGnNU8Nm4H06lYiISLBr1IAwq9VKcXGx57HT6cRy9obJH374ISdPnuSe\ne+6hqKiIsrIyevTowa9+9avmqXEj6AYYIiLSkjQqnNPT01m/fj1Dhw4lLy+P1NRUz7px48Yxbtw4\nAFatWsWePXsCGsygbm0REWlZGhXOgwYNIjc3l8zMTFwuF7NmzWLNmjWUlJQEzXFmb+3bQ9u2ajmL\niEjL0KhwDgsL45lnnvFZlpKSUuN5gW4xu5lMRuv588+hvBwiIwNdIxERkdq1+iuEufXuDU4nnMP5\n5yIiIgERMuHsHrGt484iIhLsQiacNWJbRERaCoWziIhIkAmZcE5OhqgodWuLiEjwC5lwDgsz7lC1\nfbsxMExERCRYhUw4g9G1XVoK//lPoGsiIiJSu5ALZ1DXtoiIBLeQCmfdAENERFqCkApnjdgWEZGW\nIKTCOTXVGBimcBYRkWAWUuEcGQk9euiYs4iIBLeQCmcwjjsfO2YUERGRYBRy4azjziIiEuxCNpzV\ntS0iIsEq5MJZp1OJiEiwC7lwVre2iIgEu0aFs9PpZNq0aWRkZDB27Fj279/vs/6jjz5ixIgR3HHH\nHSxatKhZKtpcEhKgUyeFs4iIBK9GhfPatWux2+0sX76cRx99lOzsbM+6yspKXnjhBd544w2WL1/O\nm2++yYkTJ5qtws2hd2/Yvx9KSgJdExERkZoaFc6bNm2iX79+AKSlpZGfn+9ZZzabef/994mLi+PU\nqVM4nU4iIiKap7bNxH3ceceOwNZDRETEn0aFs81mw2q1eh6bzWYcDofnscVi4R//+Ae33XYb11xz\nDdHR0U2vaTPScWcREQlmjQpnq9VKcXGx57HT6cRisfg8Z/DgwWzYsIGKigrefffdptWymel0KhER\nCWaNCuf09HQ2bNgAQF5eHqmpqZ51NpuNu+66C7vdTlhYGNHR0YSFBdegcJ1OJSIiwcxS/1NqGjRo\nELm5uWRmZuJyuZg1axZr1qyhpKSEjIwMbr31VsaMGYPFYqFnz54MGzasuevdJBdfDHFxCmcREQlO\njQrnsLAwnnnmGZ9lKSkpnvmMjAwyMjKaVrPzyGQyWs/ffgsOB1ga9SmIiIicH8HV33wB9eoFdjvs\n3RvomoiIiPgK2XDWiG0REQlWCmeFs4iIBJmQD2edTiUiIsEmZMO5Rw9jIJhaziIiEmxCNpzDw+Gy\ny4xwdrkCXRsREZEqIRvOYHRtnz4NR44EuiYiIiJVQjqcdaUwEREJRiEdzhqxLSIiwUjhjMJZRESC\nS0iHc8+exlSnU4mISDAJ6XC2WuGSS9RyFhGR4BLS4QxG1/ahQ/DDD4GuiYiIiEHhfPa4844dga2H\niIiIW8iHs06nEhGRYBPy4awR2yIiEmwUzroBhoiIBBlLY17kdDrJyspix44dREREMHPmTJKTkz3r\n33vvPRYtWoTZbCY1NZWsrCzCwoJzPyApCdq2VctZRESCR6MSc+3atdjtdpYvX86jjz5Kdna2Z11Z\nWRl/+tOfWLx4McuWLcNms7F+/fpmq3BzM5mM4867d4PdHujaiIiINDKcN23aRL9+/QBIS0sjPz/f\nsy4iIoJly5YRHR0NgMPhIDIyshmqev707g2VlfD994GuiYiISCPD2WazYbVaPY/NZjMOh8N4w7Aw\n2rdvD8CSJUsoKSnh+uuvb4aqnj867iwiIsGkUcecrVYrxcXFnsdOpxOLxeLzOCcnh7179zJ37lxM\nJlPTa3oe6XQqEREJJo1qOaenp7NhwwYA8vLySE1N9Vk/bdo0ysvLefXVVz3d28FMp1OJiEgwaVTL\nedCgQeTm5pKZmYnL5WLWrFmsWbOGkpIS+vTpw8qVK7n66qv59a9/DcC4ceMYNGhQs1a8OSUnQ1SU\nurVFRCQ4NCqcw8LCeOaZZ3yWpaSkeOa3t7CUM5shNdUIZ6cTgvSsLxERCRGKobN694aSEjhwINA1\nERGRUKdwPkvHnUVEJFgonM/S6VQiIhIsFM5n6XQqEREJFgrns1JTjYFgCmcREQk0hfNZUVHQvTt8\n9x3s2RPo2oiISChTOHsZMwZ++AHS0uBvfwt0bUREJFQpnL1kZcHixcb82LFVYS0iInIhKZy9mExG\nKOflwU9/Cm++CT/6EeTmBrpmIiISShTOfvToARs2wNSpxkVJbrwR/vAHOHvjLRERkfNK4VyL8HB4\n5hn45z+hSxdjvl8/DRYTEZHzT+FcjxtugG+/hdGj4YsvjMFiS5aAyxXomomISGulcG6AhARYutQI\nZYBx44zBYqdOBbZeIiLSOimcG8hkgrvuqhos9tZbRiv6X/8KdM1ERKS1UTifox494P/9P5g2zRgs\n9vOfG/MaLCYiIs1F4dwIFgtMn26M6L7kEpgxA9LT4ZVX4NixQNdORERaOoVzE1x/vTFY7Ne/hi1b\n4IEH4KKLYPhweOcdsNsDXUMREWmJGhXOTqeTadOmkZGRwdixY9m/f3+N55SWlpKZmcnu3bubXMlg\n1qYNvPEGHDwIL7xg3Hry3XfhV7+Czp3h/vvh3//W6G4REWm4RoXz2rVrsdvtLF++nEcffZTs7Gyf\n9Zs3b2bMmDEcOHCgWSrZEnTqBI88YrSk8/KM+fBwmDcPrr0WLr8cZs82jlOLiIjUpVHhvGnTJvr1\n6wdAWloa+fn5Puvtdjvz5s2jR48eTa9hC/SjHxmt6IIC+L//g4wM2LsXpkyB5GQYMAAWLQKbLdA1\nFRGRYGRpzItsNhtWq9Xz2Gw243A4sFiMt/vxj3/cPLVr4SwWGDrUKKdOwYoVxo01PvnEKL/9LQwe\nbHSFp6YapWdPaNcu0DUXEZFAalQ4W61WiouLPY+dTqcnmMW/hASYONEou3cbFzRZvNg4Pv3uu77P\nbdvWN6zd85deCjExgam/iIhcOI1K1PT0dNavX8/QoUPJy8sjNTW1uevVqqWkGLen/MMf4MgR2LnT\nKDt2VM1/9ZVxudDqunatCu2ePaFXL6N06WJcKEVERFq+RoXzoEGDyM3NJTMzE5fLxaxZs1izZg0l\nJSVkZGQ0dx1bLZPJGNHdubNxMRNvDodxnNod1t7hvXatUbzFxvqGda9exuPLLoPo6Av3O4mISNOZ\nXK7An+RTUFDAgAEDWLduHV26dAl0dYKezQbffw/bt/uWnTuhrMz3uSYTdOtWFdg9ekDHjkbp0MGY\nJiSo1S0iciHVl3s6UNwCWa1w1VVG8VZZCf/5j9HCrh7cH3xgFH/Cw6uC2ju0vec7dTJKu3YQpkvX\niIicVwrnVsRshu7djTJkiO+6kyeN0N6/HwoL4ehRo3jPb9sGX39d988IDzfC2t0dX1vp2NEYrS4i\nIudOX58hIjHRuJvWT39a+3NcLigu9h/cR47A4cNGOXLEuNjKl1/W/l4mE7Rvb4w8j4gwSnh4w+ej\nooyR6bGxDZ9GRqp7XkRaB4WzeJhMRpe51WqMKK+Ly2W0xt2BXVs5cQIqKozrjFdUGOV8CQuDiy82\nrnl+ww3Qrx9ccYXRoyAi0pK0znB+8UVYvRpGjjQuz9W+faBr1OqYTEaruG1bIwAbyuWqCml3YNvt\nNefLyqCkxCjFxf6n1ZcVFxuD4pYtMwoY1z7/2c+MoL7hBvjJT4xWuYhIMGud4exwGDdd/uc/4eGH\njUt0jR0Lv/ylvpkDzGSq6sKOjW3+93e5YNcu+Ne/jE3gX//yHQwXEWEE9A03GOX6640ufxGRYNJ6\nT6U6fBjeesu4FFdenrEsIQFGjTKC+vrrdYAyRBw9aoS0O7C/+Qaczqr1ffrANddUXdilZ0+jWz88\nPHB1FpHWrb7ca73h7G3zZiOkly6FQ4eMZT16wF13GUF96aWNe9+SEuPuFgUFxnx8vNGP6j3VkOWg\nc+YMbNxY1bL+4gvjz+fNbDY2Ee/AdpcOHbRfJyJNo3D2Vllp3HFiyRJYtco4SAlw3XVGSGdkGAdR\nwTd4DxzwPz1xov6fGRPjG9jV5zt1giuvNMrFF+tbPwAqKoyu8B07apbjx2s+PyGh6hKqHToYR1Hc\nx9ErKup+7J4PDze60xMTjU3O37z344iI2uvvchnva7dDeblRqs+DsSlGR/sW9Q6IBIbCuTY2G7zz\njhHUa9ca33Dh4ca3rnuYcW2sVrjkEqN06WIUqxVOn4YffjBKbfPub0p/EhKMkO7btyqw+/QxwlwC\n4vjxqou6eIf27t2NG3luNhubmd3u27Ven5iYqtPS/IVwY/+LzeaqoK4tvN37i977jfXNO52+I/S9\n56s/9p6Pj/e9BG3v3sY0OVmj7qV10RXCamO1Gq3lsWPh4EF4803429+MC1p36QJXX10VvO4Qdk/b\ntGn8zy0vrwrqH34wLum1eXNVyc01+lu9detWFdbuctlldTenpFm0a2eM9v7Zz3yXu699fuqUEWDh\n4cYRDPe8v8dmc9XV1ZxOo3v95EmjnDjRsPnycuN8bqvVmEZGGptBffMuF5SWGh1CpaU1i3v58eNV\nyyorm+czdO+QuM9hd89HRUFcnO9ndPw4fPaZcbjBW1SUsd9cPbhTU3WnNmmdQrflHKxKSmDrVt/A\n3rzZGNVUndVqpIf7nCb3vL9l7nmr1UgId1KEhakrXfzyPi/d+1uiIfPuUfnh4ed+udfycuMww7Zt\nNS9D63WnWo/kZOOqdN4h72/nyN/66Oiqc/vrKxER+leR5qOWc0sTE2O02q++2nd5YaFvWO/bZzSl\nTpwwTu719611LrzD2l28l7nPfXJ/U7nn61rmvmyX02kUl8v/1N8yi8X4LOoq0dG60Pd55A6xCy0y\n0jh3vvr58y6XMdTDO6y3bTPKpk3n9wI3YGyS3kHt3ly9N11/j73nw8ONf4uGFve/kcVi9GT4K05n\n7esiIoxDBXFxRnHPe0+jos5tp8PhqL0Hprzc9/BFXcX7cIb78EpUVNUhldrmvR9HRbXeKwMqnFuK\nDh1gwACj+FNebvR9Hj9uBLZ76j1//LgR4u5AdBf3f3htj51O4/2Li6GoyDhefy4HTM8n93U+vQM7\nKsr3v9ffsurzVqsx8iohwXfaXOnkdBqf35kzRrHbq3ot/E1rW+f+FnI3Uxs6jYysf2RZkDOZqoZ6\nDBpUc73LZWy61UOg+gA971JWZmzO51Lcfz73n8O9H1v9z+TvcUWFsRkcPGhM6xqCcqGYzTUD3GTy\nH76lpcbnGWzch2/c/871zVss/v9Gdf39wsLgttvgppsuzO+kcG4tIiOrbh11vrlcRljbbMY3jPe0\n+rLy8prfXA0JIYej6mBoXcX7OWfOGN3/ZWXGz20OsbFVQe0vvM3mqsCtXk6frpovLm78qK3mZLVW\nHeZwF/ewcH8lKurcm2uVlVU7cw0p7m2luNj4PBMSqkr1z7y2ecB06hSWsyX61CljQEB9JSzMONzT\nrp1xJUH3/KXt/S9v5p0bh6Puj8NdKiurdgL8ldrW2e2+m6F7vvrUPX/4sDHg0eXybakmJsJFF9Uc\nMFh9IGFkZB2HEiwuwi0uIixOIsJdhJurpk67A3txBeW2CsqLjfmKEt9SWVaBo9QoznLjcYkjgpPO\nNpysjOe4ow3HK+I5WWHlzJkwioqMzbCsrPn+9fbsUThLMDOZqnZFg/XSqO7WflmZUUpLfafV592j\ns06d8p265w8ehC1bGv5fHh1d1RTp2LFq3l0iImr2ezZ0Wn34dEOmpaVVo8pOnDCGm7svzhNo7juX\nxMYav+PevUZSnE+xscbATqfTOETU0D7xuDgjpGNja0/H6knpLzktFjCbsVgstDGbaXP2sfc6n6l7\nh8e9vbp3QP3Nez92HyLyV9zvbbFAggXaG/MuiwWT+zq71c8DPFMBJx21nyNYWVn7tnuhmExVXQAX\nt8EVH48rrg2V1ngcsW2oiI6nMiIawsw4zRZjajLjMltwhZlxhhlTV5ixzGkye5Z3GnE9cGHGRSmc\npXUKC6vanW8u7iHW3sHtcPiGbny80TptCRefqajwDezq5fjxqu732ppstTXjIiP9HzytXmobN+Bw\nGAHtvbNU2/zJk8ZrvFvUdZU2bXxbwC6X8Xc9ftwox475Tv0tO3y49l6EQPWQuPts3dO4OOOzrays\nOhG+pMSYdxf3Oq+dkxqHb72H27tH2bmn0dHGNu9e5v77N7SHzD31Hr1X30g+72K3+56uWn16+DCm\n7dsxVVYSBoQDTfpG+H9D4f/+rynv0GCN+gZxOp1kZWWxY8cOIiIimDlzJsnJyZ71n3zyCfPmzcNi\nsTBixAhGjRrVbBUWCZiwsKqLx3ht7y1WeLgxlqFDh0DXpCaLpapr/XwzmYyAiY83bobeVO6WYn3d\n/97h6D2tvszhMELP30FU9+OIiKYPjnQ6q4I6LKyqVd3SR1u5zyP0Du2ysto/87rmb775glW7UeG8\ndu1a7HY7y5cvJy8vj+zsbObPnw9ARUUFs2fPZuXKlURHRzN69Gj69+9P+2Dt/hQRaU4mU1UvQkvi\nPiujBQ8a9Mtkqhow2rlzoGvTYI3a1dq0aRP9+vUDIC0tjfz8fM+63bt307VrV9q0aUNERAQ//vGP\n+fLLL5untiIiIiGgUeFss9mwWq2ex2azGcfZ8fU2m424uDjPutjYWGw2WxOrKSIiEjoaFc5Wq5Vi\nr4teOJ1OLGcHwFRfV1xc7BPWIiIiUrdGHXNOT09n/fr1DB06lLy8PFJTUz3rUlJS2L9/P6dOnSIm\nJoavvvqKCRMm1Pl+lWcv4nvkyJHGVEdERKRFceedO/+qa1Q4Dxo0iNzcXDIzM3G5XMyaNYs1a9ZQ\nUlJCRkYGTzzxBBMmTMDlcjFixAg6duxY5/sVFRUBMGbMmMZUR0REpEUqKiryOdvJLShufFFWVkZ+\nfj5JSUmYW9oIRxERkXNUWVlJUVERffr0ISoqqsb6oAhnERERqaJb+oiIiAQZhbOIiEiQUTiLiIgE\nGYWziIhIkGkBt845N/XdlCOUDR8+3HNlty5dujB79uwA1yiwvv32W55//nmWLFnC/v37eeKJJzCZ\nTFx22WX84Q9/IKypNxJoobw/l61bt3LvvffSrVs3AEaPHs3QoUMDW8ELrKKigilTpnDw4EHsdjv3\n3Xcfl1432ZQfAAAgAElEQVR6achvL/4+l86dO4f89lJZWcnTTz/N3r17MZlMTJ8+ncjIyHPeXlpd\nONd1U45QVl5ejsvlYsmSJYGuSlBYsGABq1evJvrsLSVnz57Nww8/zLXXXsu0adNYt24dgwYNCnAt\nL7zqn8uWLVu4++67GT9+fIBrFjirV68mISGBnJwcTp06xe23306vXr1Cfnvx97n87ne/C/ntZf36\n9QAsW7aMjRs3MmfOHFwu1zlvL61uV6+um3KEsu3bt1NaWsr48eMZN24ceXl5ga5SQHXt2pW5c+d6\nHm/ZsoVrrrkGgBtvvJHPPvssUFULqOqfS35+Pp9++iljxoxhypQpIXmd/CFDhvDQQw8B4HK5MJvN\n2l7w/7loe4GBAwcyY8YMAA4dOkR8fHyjtpdWF8513ZQjlEVFRTFhwgQWLlzI9OnTmTRpUkh/Lr/4\nxS8814MH48vFdPa+tbGxsZw5cyZQVQuo6p9L3759efzxx1m6dCmXXHIJ8+bNC2DtAiM2Nhar1YrN\nZuPBBx/k4Ycf1vaC/89F24vBYrEwefJkZsyYwa233tqo7aXVhXNdN+UIZd27d2fYsGGYTCa6d+9O\nQkKC57Kpgs/xn+LiYuLj4wNYm+AxaNAg+vTp45nfunVrgGsUGIcPH2bcuHHcdttt3Hrrrdpezqr+\nuWh7qfLss8/y0UcfMXXqVMrLyz3LG7q9tLpwTk9PZ8OGDQA1bsoRylauXEl2djYAR48exWazkZSU\nFOBaBY/LL7+cjRs3ArBhwwauvvrqANcoOEyYMIHvvvsOgM8//5wrrrgiwDW68I4dO8b48eN57LHH\nuOOOOwBtL+D/c9H2Au+++y5//vOfAYiOjsZkMtGnT59z3l5a3eU73aO1d+7c6bkpR0pKSqCrFXB2\nu50nn3ySQ4cOYTKZmDRpEunp6YGuVkAVFBTwyCOP8Pbbb7N3716mTp1KRUUFPXr0YObMmSF7nXfv\nz2XLli3MmDGD8PBw2rdvz4wZM3wOG4WCmTNn8sEHH9CjRw/PsqeeeoqZM2eG9Pbi73N5+OGHycnJ\nCentpaSkhCeffJJjx47hcDiYOHEiKSkp5/z90urCWUREpKVrdd3aIiIiLZ3CWUREJMgonEVERIKM\nwllERCTIKJxFRESCjMJZREQkyCicRUREgozCWUREJMgonEVERIKMwllERCTIKJxFRESCTFDcS7Gs\nrIz8/HySkpJC7uLxIiISeiorKykqKqJPnz5ERUXVWN+gcP722295/vnnWbJkic/yTz75hHnz5mGx\nWBgxYgSjRo3y3BVqx44dREREMHPmTJKTk+t8//z8fMaMGXMOv5aIiEjLt3TpUr+3kKw3nBcsWMDq\n1auJjo72WV5RUcHs2bNZuXIl0dHRjB49mv79+/P1119jt9tZvnw5eXl5ZGdnM3/+/Dp/hvu+wkuX\nLqVTp07n8nuJiIi0OEeOHGHMmDGe/Kuu3nDu2rUrc+fO5fHHH/dZvnv3brp27UqbNm0A+PGPf8yX\nX35JXl4e/fr1AyAtLY38/Px6K+nuyu7UqRNdunSp9/kiIiKtQW2HcusdEPaLX/wCi6VmhttsNuLi\n4jyPY2Njsdls2Gw2n5trm81mHA5HY+osIiISkho9WttqtVJcXOx5XFxcTFxcXI3lTqfTb7iLiIiI\nf41OzZSUFPbv38+pU6eIiYnhq6++YsKECZhMJtavX8/QoUPJy8sjNTW1OesrItLyuFxgtxvTiAgI\na+azWCsrobgYSkqM4p4PC4P4+KoSEwMmU/P+7JbC4YCiIjhyBE6d8v951bdszBh44IELUt1zDuc1\na9ZQUlJCRkYGTzzxBBMmTMDlcjFixAg6duzIoEGDyM3NJTMzE5fLxaxZs85HvUVaH5cLTp82vkCO\nHTOm7uJ+7HD4ftnGx0NcXM1l7uWxsTW/jF0uKC+vvdjtVfMmk/EeMTFGcc/HxkJ4eGC/6F0uKCsz\nvjzLy42AcjgaNz3XdeXlDftC915XWVlV9/BwI6QjIxtWnE7/P8c9b7c37DNzh3Vt24y7REY27u/h\ncNS9PdVWIiMhMdEobdtWzVd/7J5PSACzuep/5siRqnL4sO9j97KiIuP5jRURAddf3/jXnyOTy9WU\n2jaPgoICBgwYwLp16zQgTM4/lwvOnPENvtOn6/8C8fdF43QaXxIWi+/U3zLvdWVlNYP32LGGf8k2\nVFiY8UVssVTVuaKied7bbK49uCMjz/1zsViML/eGtGDc88EoIsL38/CeN5kaFlbl5f7fOyzM93Ou\naxoTY2yfp0/7ljNnquZ/+MF3p+FCMZl8dz7Ky426nIv4eGNbLi2t82mu+HicHTpR2b4TFe06YW/b\niQprIo7IWBwRMTgiY6kIj6EiPAZ7uDFfbonFbonBbjHmy8OiqXBZuOEG6N69Cb+3l/pyTweDWzOb\nDbZvh23bYOtWY3ryZNU/bm3/0P7mzeaGhVX14nLV3hqoq+VgsTSuVVZcXHur0/txc4dgU8TFQVIS\nXHWVMW3f3pi6i/dji8X4cvX+gq3ti9d7WUVFw1tp3n8fl6vh3X4nThjT8/HZRkdXbYvt2kHXrr7b\nZ0SE8dnUFfx17Syc685VZGTt/zMWCy6X8bG488/9pygpMf4U9Ra7C2d5Ba6yclxl5cbPjInBEhNB\nRKTJ59+nrml5ubG5Hz9eVXweH3NRfLyMylOniaeqROD/bxgVafyKUVHGnyQ6BqLd89GAxUI5kdhN\nkZS5IrETQZkr0qeUOiOxOy1UOk2ejgiXC8JclVgrfyDecYI4x0niHCc98/GVJ/0urzCFUxjXmUJT\nJ47QiUOuzhys7ESBwyiHnB0pOR0Lp4FdTd8Mhw6F//u/pr9PQyicW4MTJ4zg9Q7hrVvhP/8JdM2C\ni3cIVg/ANm0attNQfX1YWM2uz4Z0o0ZGVgVvY7oQg5m7a7O2z6Kuz8VfazwqqsYxWvd+59atRjl2\nzHgLRwU4So2Aczh8i79lYPw53cXd21zfsoqKmsHrPX/6tNFobTwTEHG2xNXz3MYJD4d27Uy07xJN\nux9F065dR9q3N/Z9zGZjn85mM4p7vsh7WaExbQqTyfhZxp/XDLQ9WxrGYoFIi/9/zw6RcEkd/8Le\n+2XVi791ZjPcfHPTft9zoXAOBJcLduyA3Fw4cKBx71FUVBXCR4/WXN+5MwwYAL17w+WXV02Tkoxu\noNqOXdU27w6Uc20Bm0zn1k3sLo09/S4qyjd03QHYvr2xTs4/97dZMzh5ErZ+U7Wpu/c9G7vfGR7u\n+4XrchlBa7c3rbc/LMzYv4uPh0suqZp3F/fj6GijDt7FvQNQV3EPE/D+d3HPV596z0dGGmHrDl3v\neau16UMGnE7j68TdmVNZeW6dFs09Lq41UTiDsRX/7//C668bYdm3L6SlGS2stDTjv60pW3FpKXz1\nlRHGubnw2WdGa7c5dOtm9LW4w/fyy6FXL2PARG3crRIJCu6BvCUlVftN3vtP3vMul7GP4S7R0b6P\nqxfvMVtOp3Gou7b39jdvt1c1dp1O//P+1plMNYPH33z1Zd77nNu2GWN5qrvoImO/03ufs3Pnqvfx\nDl/vx2Fhdf8bu4PaHdbepfoyi8U3gEN1ELT7EHhsLOjijs0rtMN51y4jkP/6V6NfDIxdynffNYpb\n27ZGSHsHdq9etbcOjh71DeJNm3x3y7t1g//6L2PkX8+ejdt9jI83Xhsbe+6vlQZzOIxjc0VFUFjo\ne+i6qMhoLdTXdepvWXm5bxA2rQu0diZTVYdBPeNmgo7738S9z9m7t1Hq2u9sCpOpqtta/1YSaKEX\nzhUVsGYNvPYafPyxsaxdO3jsMbjnHrj0UmPYfV4efPNN1fSTT4ziFhkJV15ZFdYWS1Ug795d9TyL\nxXjO9dcb5Wc/M3b9JSgcPAiffmocv/QOXXcQnzzZtLMvzGbf1pt7PiLC6F6MiTFav97Tupa5jxK4\nW8BlZQ0rLpfvOEB/P8vfz3Ufm/PuhvQ3X/2x0+nbCvWe+lvmbpEmJBhBrP1OCXWhE87/+Q/8z/8Y\n5fBhY9mNN8K998KIEb6Dcjp3Nsp//VfVsjNn4NtvfUP7u++M7mpvCQlGN/PPfmaE8TXXqAs5iBw5\nAuvXG+XTT+H772s+x2Qy9tc6doQ+faBDh5qHsZOSjOVxcTW7U93zZnNodnWKSNO17nCurIQPPzRa\nye+/b+zOt2kDDz5ohPLllzf8veLi4IYbjOJWUWEcGPvmG2P+uuuMfjeNcggahYXwz39WBfL27VXr\n4uLglluMEZjp6UYYJyUZRzF0W3ERCaTWGc4nTsD8+cbxZPewzmuugd/8BjIymq8lGx5uDB7r27d5\n3k+a7Phx3zDesqVqXWwsDBlihPHNNxtHG3TZdxEJRq3zq+l3v4Nly4xv43vuMVrJ6emBrpU0QGWl\n0QlRUmJc/vbkSWPqXepaduhQ1XtFR8OgQUYQ33QTXH21sT8lIhLsWmc4T55sNJGGDzdGNct543Qa\nrdVDh6rKwYNV86dO1T0QqPp8Y0Yth4UZh/oTEoxTbG66ySjXXGMMaBIRaWlaZzi7T3uSelVW+o7q\nrW0E8MmTNYPXXeq7eENt57rGxNR+3mt0dNX17RMSfOerP26OiymIiAST1hnOIczlMlqr+/fDvn1G\n8Z4/ftw3hBt7IS6z2RjQftVVcPHFxtlh7uL9OCFBwSkicq4Uzi2QzWZc/bN68LrnT5/2/7qYGOP0\nH/eVLOu7upR3iY+vCt2LLzbeQyOaRUTOD4VzkHO5jAuZff55Vdm82f+xWavVuKqSuyQn+07bt1cr\nVkSkJVA4BxmbDb78siqIv/ii6sqiYLRif/Yzozu5e3ffIE5MVPiKiLQGCucAcrlgzx7fVvF33/ne\n+zw5GQYONK5vct118KMfaQSyiEhrp3C+wA4fhnXrjMt6r13re15uZCT89KdVQfzTn+oy3CIioUjh\nfJ7ZbLBhgxHEH38M+flV65KSYORIo5v6uuuMrmq1ikVEROHczBwO4w6RH39slM8/rzoPOCoKBg82\nrlo1cKBx1U9dhltERKpTODeDI0dg1SqjdfzJJ/DDD8Zykwl+/GMjiAcNMlrI7nvrioiI1Ebh3AQu\nFyxdCvffXxXI3bsb99YYOBD69zduPSgiInIuFM6NdPw43HcfrFhh3F8jJ8e4lHdKSqBrJiIiLZ3C\nuRE++AAmTDBGXl9/PSxapFAWEZHmo+FI58BmM1rLQ4caFwaZPdu4d7CCWUREmpNazg30+ecwdizs\n3g19+sCSJbrxlYiInB9qOdfDboenn4YbbjCu5vXYY/DVVwpmERE5f9RyrsPWrXDXXfDNN8ZlNBcv\nhhtvDHStRESktVPL2Q+nE+bMgfR0I5jvvtu45rWCWURELgS1nKvZvx/++7/h00+Ny2suWAC33Rbo\nWomISChROHv54APIzITTp41Afv116NAh0LUSEZFQo3D28tRTUFoKCxcaXdm6N7KIiARCveHsdDrJ\nyspix44dREREMHPmTJKTkz3r3333XRYuXEhcXBzDhw9n5MiRAAwfPhyr1QpAly5dmD179nn6FZqH\n0wnbtxunSY0fH+jaiIhIKKs3nNeuXYvdbmf58uXk5eWRnZ3N/PnzAThx4gQvv/wyq1atIj4+nv/+\n7//muuuuIykpCZfLxZIlS877L9BcDhwwWs09ewa6JiIiEurqHa29adMm+vXrB0BaWhr5XjckLigo\noGfPniQkJBAWFsaVV17Jt99+y/bt2yktLWX8+PGMGzeOvLy88/cbNJPt241pr16BrYeIiEi9LWeb\nzebpngYwm804HA4sFgvJycns2rWLY8eOERsby+eff063bt2IiopiwoQJjBw5kn379jFx4kQ+/PBD\nLJbgPcS9Y4cxVTiLiEig1ZuWVquV4uJiz2On0+kJ2TZt2vDkk0/ywAMPkJCQwBVXXEFiYiLdu3cn\nOTkZk8lE9+7dSUhIoKioiM6dO5+/36SJ3C1ndWuLiEig1dutnZ6ezoYNGwDIy8sjNTXVs87hcLB1\n61befPNNXnrpJfbs2UN6ejorV64kOzsbgKNHj2Kz2UhKSjpPv0LzcIez168nIiISEPW2nAcNGkRu\nbi6ZmZm4XC5mzZrFmjVrKCkpISMjAzBGZkdGRnL33XfTtm1b7rjjDp588klGjx6NyWRi1qxZQd2l\nDUa3dnIyxMQEuiYiIhLqTC6XyxXoShQUFDBgwADWrVtHly5dLvjPP30a2rSBwYPho48u+I8XEZEQ\nU1/u6draaDCYiIgEF4UzCmcREQkuCmc0UltERIKLwhldgERERIKLwhmjWzsuDoL4NGwREQkhIR/O\nlZWwc6fRpa27UImISDAI+XDetw/sdnVpi4hI8Aj5cNZIbRERCTYhH84aqS0iIsFG4ayR2iIiEmRC\nPpx37DAGgl16aaBrIiIiYgj5cN6+Hbp3h6ioQNdERETEENLhfPIkFBaqS1tERIJLSIeze6S2BoOJ\niEgwCelw1mAwEREJRgpnFM4iIhJcQjqc1a0tIiLBKKTDeft2SEiADh0CXRMREZEqIRvOFRWwa5fR\npa0bXoiISDAJ2XDeuxccDnVpi4hI8AnZcNZgMBERCVYKZ4WziIgEmZANZ43UFhGRYBWy4bx9O5jN\nkJIS6JqIiIj4CulwTkmBiIhA10RERMRXSIbzsWNw4oS6tEVEJDiFZDhrMJiIiAQzhbOIiEiQCclw\n1khtEREJZiEZzmo5i4hIMAvJcN6xA9q3h3btAl0TERGRmkIunMvLYc8edWmLiEjwqjecnU4n06ZN\nIyMjg7Fjx7J//36f9e+++y633nord955JytWrGjQawJp926orFSXtoiIBK96w3nt2rXY7XaWL1/O\no48+SnZ2tmfdiRMnePnll1myZAl/+9vfWLNmDQUFBXW+JtDcg8EUziIiEqws9T1h06ZN9OvXD4C0\ntDTy8/M96woKCujZsycJCQkAXHnllXz77bd89913tb4m0NyDwdStLSIiwarelrPNZsNqtXoem81m\nHA4HAMnJyezatYtjx45RWlrK559/TklJSZ2vCTSN1BYRkWBXb8vZarVSXFzseex0OrFYjJe1adOG\nJ598kgceeICEhASuuOIKEhMT63xNoO3YAeHh0L17oGsiIiLiX70t5/T0dDZs2ABAXl4eqampnnUO\nh4OtW7fy5ptv8tJLL7Fnzx7S09PrfE0guVxGy/nSSyFI9hVERERqqDeiBg0aRG5uLpmZmbhcLmbN\nmsWaNWsoKSkhIyMDgOHDhxMZGcndd99N27Zt/b4mGBw9Cj/8AP37B7omIiIitas3nMPCwnjmmWd8\nlqV43QT5/vvv5/7776/3NcFAI7VFRKQlCKmLkGiktoiItAQhGc5qOYuISDALqXDW3ahERKQlCKlw\n3r4dOnaEs9dMERERCUohE86lpbBvn7q0RUQk+IVMOO/aZZznrC5tEREJdiETzhoMJiIiLYXCWURE\nJMiETDhrpLaIiLQUIRPO27dDZCQkJwe6JiIiInULiXB2uYyWc2oqmM2Bro2IiEjdQiKcDx0Cm01d\n2iIi0jKERDhrMJiIiLQkCmcREZEgExLhrJHaIiLSkoREOOtWkSIi0pJYAl2BC2H7drj4YoiLC3RN\nRESCR3Z2Nlu2bKGoqIiysjIuueQSEhMTefnll+t97bZt21i3bh3333+/3/UbNmzg8OHDZGRkNHe1\nQ0KrD+fiYjhwAPr3D3RNRESCyxNPPAHAqlWr2LNnD5MmTWrwa3v37k3v3r1rXX/jjTc2uX6hrNWH\n886dxlSDwUQkmD32GKxY0bzvOXIk5OSc++s2btzI888/T3h4OKNGjSIqKoqlS5ficDgwmUy88sor\nfP/99yxbtow5c+YwePBg0tPT2bt3L+3atWPu3Ln8/e9/Z8+ePWRmZvLoo4/SqVMnDhw4wJVXXsn0\n6dM5ceIEkyZNwm630717d7744gs+/vhjn3q88MIL5Ofnc+rUKXr16sXs2bM5ceIEkydP5syZM7hc\nLp599lni4+NrLFuzZg3t27dn9OjR7N69m6ysLJYsWcIvf/lLunXrRnh4OJMnTyYrK4vy8nKKiop4\n+OGHGThwIOvXr+eVV17B5XJxxRVXMH78eB577DFWrlwJwMMPP8z48ePp27dvc/yZ/Gr14ayR2iIi\n5668vJwVZ/cWXnvtNV5//XWio6OZNm0a//rXv+jYsaPnuQcOHGDRokV07tyZzMxMNm/e7PNe+/bt\nY+HChURHRzNw4ECKiopYsGABAwYMYMyYMeTm5pKbm+vzGpvNRnx8PH/9619xOp3ccsstHD16lAUL\nFtC/f39Gjx7N119/zXfffcd3331XY1ltSkpK+O1vf8vll1/OZ599xt133821117L119/zdy5c7np\nppuYMWMGK1asoF27dixYsIDIyEiioqLYtWsX7du3p6Cg4LwGM4RAOGuktoi0BDk5jWvlni/du3f3\nzLdr147JkycTGxvLnj17SEtL83luYmIinTt3BqBz586Ul5f7rO/atStWqxWApKQkysvL2b17N8OH\nDwfg6quvrvHzIyMjOXHiBI888ggxMTGUlJRQUVHB3r17ueOOOwBIT08nPT2dv//97zWWzZ07t97f\nLSkpifnz57Ny5UpMJhMOh4OTJ08SHx9Pu3btAJg4cSIAI0eOZNWqVVx00UUMGzasgZ9i47X60dpq\nOYuInLuwMCMezpw5w8svv8ycOXOYOXMmkZGRuFwun+eaTKY638vf+tTUVL755hsA8vLyaqx3Dyh7\n8cUXeeSRRygrK8PlcpGSkuJpmX/55Zfk5OT4XRYZGUlRUREAW7Zs8fu7vfTSS9x2223k5ORw7bXX\n4nK5aNeuHadPn+bUqVMAzJw5k++++44hQ4aQm5vLxx9/fEHCudW3nLdvh5gY6NIl0DUREWl5rFYr\n6enpZGRkYLFYiI+Pp7CwkC5N/FKdOHEijz/+OB988AEdOnTAYvGNo759+/Lqq68yZswYTCYTl1xy\nCYWFhfzmN79hypQprF69GoBZs2YRGxtbYxkYx4a//PJLrrjiCr91GDJkCM899xyvv/46nTp14uTJ\nk4SFhfGHP/yBe++9l7CwMC6//HKuvPJKTCYTP/nJTzhx4gQJCQlN+t0bwuSqvgsUAAUFBQwYMIB1\n69Y1+Q/uzekEq9Xo0j67gyYiIkHgn//8J4mJifTt25fPPvuM1157jcWLFwe6WnWaPn06gwcP5rrr\nrmvye9WXe6265XzgAJSWqktbRCTYdOnShSlTpmA2m3E6nTz11FOBrlKdxo8fT2JiYrMEc0O06nDW\n8WYRkeCUkpLC8uXLA12NBvvLX/5yQX9eqx4QppHaIiLSErXqcFbLWUREWqKQCOfU1MDWQ0RE5Fy0\n6nDesQO6djVOpRIREWkpWm04nz4Nhw6pS1tEpDZ33XUXn3/+uc+ymTNnei7bWV1BQQGjRo0C4Pe/\n/z12u91n/YYNGzw30/DH+5Kgq1atYt26dU2pfqtW72htp9NJVlYWO3bsICIigpkzZ5KcnOxZv3r1\nav76178SFhbGiBEjuPPOOwEYPny453JtXbp0Yfbs2efpV/DPPRhM4Swi4t/IkSP5+9//7jk9yG63\ns379eh555JF6Xztnzpxz/nlFRUWsWLGCkSNH8qtf/eqcXx9K6g3ntWvXYrfbWb58OXl5eWRnZzN/\n/nzP+ueee4733nuPmJgYbrnlFm655RaioqJwuVwsWbLkvFa+LhqpLSItSgBuSzVkyBDmzJlDaWkp\n0dHRrFu3juuvv56YmBj+/e9/e+7MVFxczAsvvEB4eLjntf379+eDDz6goKCAKVOmEB0dTXR0NG3a\ntAHgb3/7G//4xz8oLS0lMTGRV155hddee41du3Z53td916js7Gw2bdoEwC9/+Ut+/etf88QTTxAR\nEcHBgwcpLCwkOzvb50pflZWVTJs2jSNHjlBYWEj//v35/e9/z759+3j66aepqKggKiqKOXPmcPr0\n6RrLnnvuOYYOHcqNN97Ihg0beP/998nOzubmm2+mR48epKSkcMcdd5CdnU1lZSUnT54kKyuL9PR0\nVqxYwVtvvYXT6aR///6kp6fz9ttve+6DnZmZyUsvveRzc5BzVW+39qZNm+jXrx8AaWlp5Ofn+6zv\n2bMnZ86cwW6343K5MJlMbN++ndLSUsaPH8+4ceP8Xjf1fNNIbRGRukVGRjJw4EDPrRpXrVpFZmYm\nAN9//z05OTksWbKEwYMH8+GHH/p9j+eee44HH3yQN954g6uuugowelxPnTrFG2+8wYoVK6isrGTz\n5s385je/4dJLL+X+++/3vH79+vUUFBTw9ttv8+abb/Lee++x42zr6qKLLmLhwoWMHTu2xjnRhw8f\nJi0tjYULF7Jy5UqWLVsGwLPPPss999zD8uXLGTduHFu3bvW7rDaHDx/m+eefZ8qUKezatYvJkyez\naNEiJk6cyKpVqzh+/DgLFizgzTff5J133sFut5OWlsbOnTv54Ycf+P7770lMTGxSMEMDWs42m83T\nPQ1gNptxOBye66BedtlljBgxgujoaAYNGkR8fDxRUVFMmDCBkSNHsm/fPiZOnMiHH35Y49qp55PC\nWURalADdlmrkyJE899xzXHvttZw+fZrLL78cgI4dO/LHP/6RmJgYjh49Snp6ut/X79u3z3P7xPT0\ndPbs2UNYWBjh4eGeO0odOXIEh8Ph9/W7d+/m6quvxmQyER4ezo9+9CN2794NQO/evQHo1KkTX3/9\ntc/rEhIS2Lx5M1988QVWq9Vz/Hvv3r2enYQBAwYAxnH06svee+89z3t5X8U6MTGRxMREADp06MCr\nr75KVFQUxcXFWK1WDhw4wGWXXUZUVBQAkyZNAmDYsGG89957FBQUeO6Q1RT1tpytVivFxcWex06n\n0xOy27dv59NPP2XdunV88sknnDhxgg8++IDu3bszbNgwTCYT3bt3JyEhwXN3kAtlxw7jutpn72Im\nIheMUnMAAAeJSURBVCJ+9OzZk+LiYhYvXsyIESM8y6dOncqsWbPIzs6mQ4cONe5E5ZaSkuK5u5S7\nZ3X79u2sXbuWP/3pT0ydOhWn04nL5SIsLAyn01nj9e4u7YqKCr755hvPuKa67na1atUq4uLieOGF\nFxg/frzfu1atXr2aJUuW+F0WERHhySXvlrT7jlUAf/zjH3nwwQd59tlnSU1NxeVy0bVrV/bs2ePZ\nGXjwwQc5evQoI0aM4MMPP+TLL7/k5z//eX0fe73qbcqmp6ezfv16hg4dSl5eHqleJw3HxcURFRVF\nZGQkZrOZtm3bcvr0aVauXMnOnTvJysri6NGj2Gw2kpKSmlzZhqqshO+/hyuvhHruZCYiEvJGjBhB\nTk4O69ev9ywbNmwYY8aMITo6mvbt21NYWOj3tU888QSTJ09m4cKFtG3blsjISJKTk4mOjvZ0kScl\nJVFYWMhVV11FRUUFOTk5npbnzTffzL///W8yMjKoqKhgyJAhtd5Fytt1113Ho48+Sl5eHhERESQn\nJ1NYWMjjjz/OtGnTmD9/PlFRUeTk5HDjjTfWWHbgwAGmTJnCmjVr6Natm9+fMWzYMB566CHi4+M9\nd61q27YtEydO5K677sJkMnHzzTd7urBjY2NJS0trll7ieu9K5R6tvXPnTlwuF7NmzWLr1q2UlJSQ\nkZHBW2+9xf/+7/8SHh5O165dmTFjBgBPPvkkhw4dwmQyMWnSpFq7RKD570q1Zw+kpMCYMfC3vzX5\n7UREROp17733MmXKFJ8zmmrT5LtShYWF8cwzz/gsS0lJ8cyPHj2a0aNH13jdCy+8UG/lzhcdbxYR\nkQulrKyMO++8k2uvvbZBwdwQrfKuVApnERG5UKKioli1alWzvmervEKYznEWEZGWrFWG88GDEB4O\nl10W6JqIiIicu1YZzjNmwLvvwtnBgCIiIi1KqzzmfNVVRhEREWmJWmXLWUREpCVTOIuIiAQZhbOI\niEiQCYpjzpWVlQAcOXIkwDURERE5/9x5586/6oIinN0XHx8zZkyAayIiInLhFBUV+b2qWL3X1r4Q\nysrKyM/PJykpCbPZHOjqiIiInFeVlZUUFRXRp08fz01AvAVFOIuIiEgVDQgTEREJMgpnERGRIKNw\nFhERCTIKZxERkSATFKdSNSen00lWVhY7duwgIiKCmTNnNtvNr1u64cOHY7VaAejSpQuzZ88OcI0C\n69tvv+X5559nyZIl7N+/nyeeeAKTycRll13GH/7wB8LCQnPf1ftz2bp1K/feey/dunUDYPTo0Qwd\nOjSwFbzAKioqmDJlCgcPHsRut3Pfffdx6aWXhvz24u9z6dy5c8hvL5WVlTz99NPs3bsXk8nE9OnT\niYyMPOftpdWF89q1a7Hb7Sxfvpy8vDyys7OZP39+oKsVcOXl5bhcLpYsWRLoqgSFBQsWsHr1aqKj\nowGYPXs2Dz/8MNdeey3Tpk1j3bp1DBo0KMC1vPCqfy5btmzh7rvvZvz48QGuWeCsXr2ahIQEcnJy\nOHXqFLfffju9evUK+e3F3+fyu9/9LuS3l/Xr1wOwbNkyNm7cyJw5c3C5XOe8vbS6Xb1NmzbRr18/\nANLS0sjPzw9wjYLD9u3bKS0tZfz48YwbN468vLxAVymgunbtyty5cz2Pt2zZwjXXXAPAjTfeyGef\nfRaoqgVU9c8lPz+fTz/9lDFjxjBlyhRsNlsAaxcYQ4YM4aGHHgLA5XJhNpu1veD/c9H2AgMHDmTG\njBkAHDp0iPj4+EZtL60unG02m6frFsBsNuNwOAJYo+AQFRXFhAkTWLhwIdOnT2fSpEkh/bn84he/\nwGKp6jhyuVyYTCYAYmNjOXPmTKCqFlDVP5e+ffvy+OOPs3TpUi655BLmzZsXwNoFRmxsLFarFZvN\nxoMPPsjDDz+s7QX/n4u2F4PFYmHy5MnMmDGDW2+9tVHbS6sLZ6vVSnFxseex0+n0+bIJVd27d2fY\nsGGYTCa6d+9OQkKC57Kpgs/xn+LiYuLj4wNYm+AxaNAg+vTp45nfunVrgGsUGIcPH2bcuHHcdttt\n3Hrrrdpezqr+uWh7qfLss8/y0UcfMXXqVMrLyz3LG7q9tLpwTk9PZ8OGDQDk5eWRmpoa4BoFh5Ur\nV5KdnQ3A0aNHsdlsJCUlBbhWwePyyy9n48aNAGzYsIGrr746wDUKDhMmTOC7774D4PPPP+eKK64I\ncI0uvGPHjjF+/Hgee+wx7rjjDkDbC/j/XLS9wLvvvsuf//xnAKKjozGZTPTp0+ect5dWd/lO92jt\nnTt34nK5mDVrFikpKYGuVsDZ7XaefPJJDh06hMlkYtKkSaSnpwe6WgFVUFDAI4/8/3buEAdCGAij\n8G+4AScoZ8MTXENQHcdNuEBPguYC1RgQ6za7yZp1A32frGwmfUmbdNS6rtr3XfM86zxPhRBkZtX+\n8/65L9u2KaWkpmnUtq1SSl/PRjUwM+WcFUJ4r03TJDOrel5+7cswDFqWpep5OY5DMUaVUnRdl/q+\nV9d1f58vj4szAAB397hrbQAA7o44AwDgDHEGAMAZ4gwAgDPEGQAAZ4gzAADOEGcAAJwhzgAAOPMC\nsgvL2hpqdk8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x261ae642940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the loss and accuracy curves for training and validation \n",
    "fig, ax = plt.subplots(2,1)\n",
    "ax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\n",
    "ax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n",
    "legend = ax[0].legend(loc='best', shadow=True)\n",
    "\n",
    "ax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\n",
    "ax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\n",
    "legend = ax[1].legend(loc='best', shadow=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "69df90d7-f2fd-497d-a0b2-57c497a3e645",
    "_uuid": "ba77212f82b781b0e5a4e494897b18c31c5e30a3"
   },
   "source": [
    "The code below is for plotting loss and accuracy curves for training and validation. Since, i set epochs = 2 on this notebook .\n",
    "I'll show you the training and validation curves i obtained from the model i build with 30 epochs (2h30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "bacff684-79bb-4ede-83c1-ccb7cf92df77",
    "_uuid": "9f2920cbbfb6f1f566fac4afdd4030c75ee5bf66"
   },
   "source": [
    "<img src=\"http://img1.imagilive.com/0717/mnist_099671_train_val_loss_acc.png\"></img>\n",
    "\n",
    "The model reaches almost 99% (98.7+%) accuracy on the validation dataset after 2 epochs. The validation accuracy is greater than the training accuracy almost evry time during the training. That means that our model dosen't not overfit the training set.\n",
    "\n",
    "Our model is very well trained  !!! \n",
    "\n",
    "<img src=\"http://img1.imagilive.com/0717/accuracies1de.jpg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "de1c65bd-4a88-4351-9f4b-562e72e7e0fd",
    "_uuid": "63698d7d51381b33892ce164b0f21930abb3e937"
   },
   "source": [
    "## 4.2 Confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5688faa0-b33b-4e92-b125-7fa0b37e7df3",
    "_uuid": "3306d29b732341663e50866140dc569360701a81"
   },
   "source": [
    "Confusion matrix can be very helpfull to see your model drawbacks.\n",
    "\n",
    "I plot the confusion matrix of the validation results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "_cell_guid": "11361e73-8250-4bf5-a353-b0f8ea83e659",
    "_execution_state": "idle",
    "_uuid": "16e161179bf1b51ba66c39b2cead883f1db3a9c7"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcAAAAGOCAYAAAAAd9NvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8zvX/x/HHtbFhM0ZGbYZhnfcNEyXCipLlPMwpEilJ\n3xLN0NEhJcm3nL4p5/ZNB0W+WL75dkD5llPOFTamZWoHbLbr8/tDW/mZxmf7XNfncj3v3a7brevj\nuj7vp/c112vvz+f9+bwdhmEYiIiIeBkfdwcQERFxBxVAERHxSiqAIiLilVQARUTEK6kAioiIV1IB\nFBERr6QCKG5TUFDA/Pnz6dq1K506daJDhw5MnTqVvLy8Uu1z2LBhtG/fnkWLFl3y+7dv386IESNM\nt1/WsrKy6N+//wX/vFOnTmRmZrowkcjlw6HrAMVdxo0bx2+//cYLL7xA5cqVOXnyJE888QQBAQFM\nnTrV1D6PHDlC+/bt+e677/D19S3jxK6XkpJCbGws3377rbujiFx2NAIUtzh8+DAfffQREydOpHLl\nygBUqlSJZ555hjvvvBM4O/p54okn6NixI7Gxsbz44ovk5+cDcOONN/Laa6/Rq1cv2rZty1tvvUV2\ndjaDBw8mPz+frl27cujQIa6++moyMjKK2i18npOTw4gRI+jUqRNdunQhMTERp9PJpk2b6Nixo6n2\ni3PjjTfy8ssvExsbS5s2bVi1ahUjRozgrrvuon///pw8eRKAd999lx49etC5c2fatGnDkiVLAHjq\nqac4ffo0nTp1oqCggBtuuIFHH32U9u3bs3379qK/z8yZM+nZsycFBQWkp6dz2223sXHjxrL/4EQu\nJ4aIG6xevdro1q3bX77mySefNJ577jnD6XQaubm5xqBBg4zZs2cbhmEYkZGRxsKFCw3DMIzt27cb\nN9xwg3H69Gnj8OHDxk033VS0j8jISOP48ePnPX///feNQYMGGYZhGPn5+cbYsWONn376ydi4caNx\nzz33mG7//4uMjDTefvttwzAMY/bs2UajRo2MtLQ0o6CgwOjSpYuxYsUKIzs724iLizMyMjIMwzCM\nb7/9tujvUNzf5/333z/v75Ofn2/06dPHmD17tjFgwADjjTfeKPEzEPF2GgGKW/j4+OB0Ov/yNRs2\nbKBv3744HA78/Pzo1asXGzZsKPrzmJgYAK6//nry8vKKRlMXo0mTJuzfv59+/foxZ84cBgwYQJ06\ndSxpv3379gCEh4cTGRlJzZo18fHxISwsjN9++42AgABmzZrFZ599xvTp05k1a9Zf/l2io6PP2+br\n68vUqVOZO3cuDoeDoUOHXnRfiHgrFUBxi6ioKH744Qeys7PP2X7s2DGGDBnC6dOnzyuQTqez6BAk\ngL+/PwAOhwMAo4TT2X+eXFO7dm3Wrl3LkCFDyM7OZuDAgaxevfq89sqi/fLlyxf7/4XS0tLo3Lkz\nqampNGnShJEjR/7l36NSpUrFbj9y5Aj+/v4cPHhQE2NELoIKoLhFzZo1iY2NJSEhoagIZmdn8/TT\nT1O1alUqVKjAbbfdxuLFizEMg7y8PJKSkrj11lsvqZ1q1aqxfft2ANauXVu0fcmSJTz11FPcdttt\njBo1ittuu419+/ad896yaP9i7Nixg2rVqvHQQw/RsmVL1q9fD5yd0VquXDkKCgpKLO6ZmZmMGjWK\nKVOm0LFjR8aOHVvmOUUuNyqA4jYTJkygQYMG9OrVi06dOtGjRw8aNGjA888/D0BiYiIZGRnExsYS\nGxtLvXr1ePDBBy+pjcTERJ599lm6dOnC999/T40aNQDo3LkzBQUFdOjQga5du5KdnX3e5QZl0f7F\naNGiBTVr1uSuu+6ic+fOHD16lGrVqnHw4EFq1KjBddddx913382JEyf+8u/ZunVrWrRowfDhwzl0\n6BCLFy8u86wilxNdBiEiIl5JI0AREfFKKoAiIuKVVABFRMQrqQCKiIhXKufuAACnT59mx44d1KhR\n47K4f6OIiDsU3grvhhtuoEKFCpa39+uvv553Le+lCgwMpGrVqmWU6NLYogDu2LGDPn36uDuGiMhl\nYfHixcXeMags/frrr0Q3a4Ev+SW/+C9UqVKFNWvWuKUI2qIAFl6bld6gL06/ILdm2TK9m1vbFxEx\n61haGgP79yn6TrVSdnY2vuRzzD+afIe50WY54zT89g3Z2dneWwALD3s6/YIo8HfPULhQaGiYW9sX\nESktV55KyvepRIFPRXNvdrp3GootCqCIiHgoB/D7/XBNvdeNVABFRMQ8h8/Zh9n3upEugxAREa+k\nEaCIiJjncJTiEKh7j4GqAIqIiHkefAhUBVBEREqhFCNAN8+C8bhzgB2ja5P6Zu9ztoVWr8Se13tQ\nvbL/ea/v17oBSaPauiTbJ6tW0rRRFFHXX018rx5uW5VbOeyVQTmUw1NymOJw/DEKvOSHCuBFq1+r\nMs/3jcbH549O690ygn8/fTdXVat0zmuDA/yYfn9zpt53s0v6OD09naGDB7I0aTnbdu6hXr0IxiWM\nsb5h5bB1BuVQDk/J4Y08pgBW9PNl7vCWJCz8pmhbreCKdGwaTrfJ6857fZdb6pL26ynGLt7iknzr\n1q6hSXRTGjRsCMCQocNYtnQxrl5vWDnslUE5lMNTcphWOAnG7MONPKYAvvrALcxft5cdhzKKtqWd\nOEWfaf9hT+pv573+zXV7mbx8K6fzSnefuouVknKYsLDaRc9Dw8LIzMwkKyvLJe0rhz0zKIdyeEoO\n00wf/izF5Jky4hEFcPCdV1NQYLDwP/vdHeWCDKez2O2uXt1COeyVQTmUw1NymKYR4PmcTifjx4+n\nZ8+e9OvXj4MHD5reV5/bG9C4fnW+mBzL8tF3UNHPly8mx1Ir2OT95yxQu3Y4aWlHi56npqYSHBxM\nQECAcrgphx0yKIdyeEoOb2RZAVy3bh15eXm88847PP7440yePNn0vtokrqTZqBW0GPMR3aas41Re\nAS3GfETaiVNlmLh0Yu5sx+ZNG9m/bx8A8+bMomNsJ+VwYw47ZFAO5fCUHKZ58CFQy64D3LJlCy1b\ntgTgpptuYseOHVY1ZQshISHMnjef+J7dyTuTR0REfebNX6AcbsxhhwzKoRyeksM0D74TjMOwaKrR\n2LFjadeuHbfffjsArVu3Zt26dZQrd37NTUlJISYmhmPXPeT25ZDSFw1wa/siImalpqbQoV0MycnJ\nhIVZu7Rb4fd26hWxFPgGmtqHb0E2ob985JK8xbFs/BkYGEhOTk7Rc6fTWWzxExERcQfLCmDjxo3Z\nsGEDAN999x2RkZFWNSUiIm5TmvN/l+k5wDvvvJMvvviCXr16YRgGEydOtKopERFxFx/H2YfZ97qR\nZQXQx8eHZ5991qrdi4iIHXjwahAecSG8iIhIWdOsFBERMc+DL4NQARQREfMKl0My+143UgEUERHz\nPHgEqHOAIiLilTQCFBER8zx4FqgKoIiIlEJpljXSOUAREfFUHjwC1DlAERHxShoBioiIeQ5KMQu0\nTJNcMhVAERExz4MPgdqqAG6Z3o3QUNevCfVnwU2Hu7X9Qie+nunuCCIiJdN1gCIiIp7FViNAERHx\nMDoEKiIiXkkFUEREvJIHzwLVOUAREfFKGgGKiEgplOIQqJvHYCqAIiJingdfBqECKCIi5nnwJBid\nAxQREa90WRTAT1atpGmjKKKuv5r4Xj3IzMy0vM3Y1lEc++9UAHx8HLya0JP/LR/L/5aPZdJjXc57\nfZ2rqpP6nyk0vi7c8mzu6A+75rBDBuWwbw4AwzB4YNB9vDLtJbdlsFN/XLLCQ6BmH27k8QUwPT2d\noYMHsjRpOdt27qFevQjGJYyxtM364TWY9FgXfHzOdl/8PTcTWSeE6B4TubnnJFo2aUDXOxoVvd7f\nrxzzX+iPX3nrjzi7oz/smsMOGZTDvjkAdu/axd3tYlj+bpJb2gd79YcZDoejVA938vgCuG7tGppE\nN6VBw4YADBk6jGVLF2MYhiXtVaxQnvnPD2D0tPeKtvn6+hBQ0R9/v3L4ly9H+fK+nM47U/Tn05+K\nY+GKTRz/NduSTH/m6v6wcw47ZFAO++YAmPXGP+g/YCDduse5vO1CduoPM84O5MwWQPdm9/gCmJJy\nmLCw2kXPQ8PCyMzMJCsry5L2Zo7tzbzln7N9b2rRtoUrNnIi8yQH/v0CP66dyA+Hf2HVhh0A3Nfl\nFsqX82X++19akuf/c3V/2DmHHTIoh31zAEyfMZP4vv1c3u6f2ak/vI3HF0DD6Sx2u6+vb5m3NaRH\nS/ILnCz4cOM528cO7cAvJ7KpE/MUDe5KJDioEo/2a8tN14QxuHtLHnlhWZlnuRBX9ofdc9ghg3LY\nN4ddeHx/OEr5cCNLC+DWrVvp18/a365q1w4nLe1o0fPU1FSCg4MJCAgo87b63duMJteHs3HZGD6Y\nOYyK/uXZuGwMvTs05e0Pv+JMfgGZ2adZ9NEmWkVH0qdjM4ICKrD+rcfZuGwMV9aowvwXBnDP7TeW\nebZCruwPu+ewQwblsG8Ou/D0/tA5wGLMnTuXxMREcnNzrWoCgJg727F500b279sHwLw5s+gY28mS\ntlr2e4noHhNp3msynYe/wancMzTvNZkvvztAt3aNAShXzoeOt9/I5u0/Muql5UR1fpbmvSbTvNdk\njqb/xsCxb7Pys+2W5APX9ofdc9ghg3LYN4ddeHp/OChFAXTzENCyaYnh4eG89tprPPnkk1Y1AUBI\nSAiz580nvmd38s7kERFRn3nzF1ja5v/35EvvMW10D757L5ECp8F/Nu/h5bfWujRDITv0h11y2CGD\nctg3h114en+UZiTn7hGgw7BwqlFKSgp///vfSUr66ynGKSkpxMTEsGpNslaE/51WhBeRS5WamkKH\ndjEkJycTFmbtd2nh9/bP1z1MgX9VU/vwzf2VkO//4ZK8xdGt0ERExDRPHgGqAIqIiHmlmc3p5lmg\nKoAiImJeaWZzXq6zQAHCwsJKPP8nIiLyV44fP87tt9/OgQMHOHjwIL179yY+Pp4JEybg/P06yqSk\nJLp27UpcXBzr16+/qP16/IXwIiLiPlZfB3jmzBnGjx9PhQoVAJg0aRIjR45kyZIlGIZBcnIy6enp\nLFy4kGXLlvHPf/6TadOmkZeXV+K+VQBFRMQ0qwvglClT6NWrFyEhIQDs3LmTm2++GYBWrVrx5Zdf\nsm3bNho1aoSfnx+VK1cmPDyc3bt3l7hvFUARETHNypthv/fee1SrVo2WLVsWbTMMo6hwBgQEkJWV\nRXZ2NpUrVy56TUBAANnZJS8+oEkwIiJiS8uXL8fhcPDVV1+xa9cuRo8eTUZGRtGf5+TkEBQURGBg\nIDk5Oeds/3NBvBCNAEVEpHQsuhH24sWLWbRoEQsXLuTaa69lypQptGrVik2bNgGwYcMGoqOjiYqK\nYsuWLeTm5pKVlcWBAweIjIwscf8aAYqIiGmuvhB+9OjRjBs3jmnTphEREUH79u3x9fWlX79+xMfH\nYxgGjz32GP7+/iXuSwVQRERMc1UBXLhwYdH/L1q06Lw/j4uLIy7u0hY21iFQERHxShoBioiIaboX\nqIiIeCfdC1RERLyRRoCXEbusw1frvvNP8rpD2lt93R1BRMQSKoAiImKeB68GoQIoIiKmOc7eC838\ne91IBVBERExzUIoC6OZZMLoOUEREvJJGgCIiYp4ugxAREW/kcFCKc4Blm+VSqQCKiIhpnjwJRucA\nRUTEK2kEKCIipmkE6GafrFpJ00ZRRF1/NfG9epCZmek1Oe5pEsahuWeXAPFxOJjcL5rNL8byv5c7\nMbBtw6LXtby2JuufvZvPX7iHtU+3p3FEdcuz2eFzsUMG5VAOT8lhitnFcEszeaaMeHwBTE9PZ+jg\ngSxNWs62nXuoVy+CcQljvCJHRM3KPBffBJ/ff4saGNOQiJqVuWXMx7QZ9wnD7rqGxhHVKe/rw5vD\nW/LoPzdy29iVvPTBDmYPa2FpNjt8LnbIoBzK4Sk5zCq8F6jZhzt5fAFct3YNTaKb0qDh2dHOkKHD\nWLZ0MYZhXNY5Kvr5MmdYC8Yu3lK0rWOT2izecIACp8FvJ/N4b+NBeraox5kCJ9eOWM62gycAqBsS\nSEZWriW5Ctnhc7FDBuVQDk/J4Y08vgCmpBwmLKx20fPQsDAyMzPJysq6rHNMH9SMtz7dx85DJ/5o\ns3olUo+fLHqemnGSq6pVAiC/wKBGUAW+n9GVZ3s3ZsbKnZbkKmSHz8UOGZRDOTwlh1mePAK0ZBLM\nmTNnSEhIIDU1lby8PIYNG0ZMTIwVTWE4ncVu9/X1taQ9O+S4/45I8p0GizYcIPyKgKLtPsX8MBU4\n//gtMj3zNNeNeI+/1a3Gh0/FsHvCag6kWfOPzA6fix0yKIdyeEoO80pTyC7DQ6ArVqygatWqLFmy\nhHnz5vHcc89Z0QwAtWuHk5Z2tOh5amoqwcHBBAQE/MW7PDtHfMsIGtWrzn9f6EDSqDZU9PPlvy90\n4EjGSWoFVyx63VXBFTmScZKgiuXpGP3Hb5hbf8pgx6Ffub521TLPVsgOn4sdMiiHcnhKDrM8eQRo\nSQG86667ePTRRwEwDMPS32Ri7mzH5k0b2b9vHwDz5syiY2wny9qzQ46YCau59amPaTl2FXFT13Mq\nr4CWY1fx8TeH6duqPr4+DqpUKk/X5nVZueUwBU6DmQ80p1nDGgBcE1qFhlcG8c2B45bkA3t8LnbI\noBzK4Sk5vJElh0ALf3PJzs5mxIgRjBw50opmAAgJCWH2vPnE9+xO3pk8IiLqM2/+Asvas3OOfybv\npV7NQD6feA9+5XyY/+k+vtj9MwB9XvmMSf2iKe/rIPeMkwde/5wjGSdL2KN5dugPO2RQDuXwlBym\nefC9QB2GRVONjh49ysMPP0x8fDzdu3f/y9empKQQExPDqjXJhIaGWRHH42hFeBG5VKmpKXRoF0Ny\ncjJhYdZ+lxZ+b59pMw4qVTO3k5MZlF//nEvyFseSEeAvv/zCoEGDGD9+PLfccosVTYiIiA2U5mbY\n7h4BWnIOcNasWWRmZvL666/Tr18/+vXrx+nTp61oSkRExBRLRoCJiYkkJiZasWsREbERhwefA9TN\nsEVExLTS3Azb3QsCqgCKiIhpnjwC9PhboYmIiJihEaCIiJjmoBSHQN08BFQBFBER80pR/wxNghER\nEU/l4+PA4WOukhk+Doq/FbhrqACKiIhppZkEqkkwIiIibqARoIiImFaqZY10HaCIiHgqTz4EqgIo\nIiKmefIIUOcARUTEK2kEaFN2WYfvivi33B2BX5bc5+4IYmMWLWl6yUyPgjye+RGgoQvhRUTEU3nw\nvbBVAEVExLzSnAN096hZ5wBFRMQraQQoIiKm6RCoiIh4pbMF0Owh0DIOc4lUAEVExDRPHgHqHKCI\niHgljQBFRMQ0T54FqgIoIiKm6RCom32yaiVNG0URdf3VxPfqQWZmptuyGIbBA4Pu45VpL7ktgzv6\no2PTcI68FX/OttDqldg7qwfVK/sXbbu7SRiH3uzNly/eW/QIrGDd72F2+dlQDnvmWLp4Ec2a3ESz\n6Ea0adWCLVu+cUsOu/SHOY6iUeClPtx9N2yPL4Dp6ekMHTyQpUnL2bZzD/XqRTAuYYxbsuzetYu7\n28Ww/N0kt7QP7umP+rUq80K/aHz+tCp071b1WfNMB66qFnDOa5tFhjDjox3c+uSKokf26XxLctnl\nZ0M57Jlj7549JDz1JB98/AmbvvmW0U+NpXdcN5fnsEt/eCOPL4Dr1q6hSXRTGjRsCMCQocNYtnSx\nW+4POOuNf9B/wEC6dY9zeduFXN0fFf18mfdIK556++uibbWCKxLbNJxuk9ae9/rmV4dw+/VX8t/J\nHVnzzN20uLamJbnAPj8bymHPHP7+/rw+ay5XXnklAI2bRHMsLY28vDyX5rBLf5hVeAjU7MOdPP4c\nYErKYcLCahc9Dw0LIzMzk6ysLIKCglyaZfqMmQCs/zTZpe3+mav7Y8aQW3lz3R52HDpRtC3txCni\nX15f7OszsnJZuuEAH319iFuuDmHZk225ZdQKjmScLPNsdvnZUA575qhTty516tYFzp66GDPqce7p\neC9+fn4uywD26Q+zNAmmGAUFBSQmJvLjjz/icDh45plniIyMLPN2DKez2O2+vr5l3pYncGV/PNDu\navILnCxcv5/wGoEX9Z4/F8av9vzMpr3ptI26ikX/2V/m+ezys6Ec9sxRKCcnhyH3DyQl5TAffvyJ\ny9u3W39cKk2CKcb69We/6JYtW8bIkSN55ZVXLGmndu1w0tKOFj1PTU0lODiYgICAv3jX5cuV/dGn\ndQOa1L+CL1+8l/eeuoOKfr58+eK91AquWOzrq1Ty44kuN56zzQGcKSj+C6C07PKzoRz2zAFw+NAh\n2rZqga+vL6vXfkrVqlVdnsFO/eFtLCuAd9xxB8899xwAR44csWwoH3NnOzZv2sj+ffsAmDdnFh1j\nO1nSlidwZX+0TljJzU98yK1PrqDrpHWcyivg1idXkHbiVLGvzzp1hiHtr6FTszoARNWtRnSDK1j3\nXaol+ezys6Ec9syRkZFBuztac2/nLixYvJSKFYv/xc1qdukPs8zOAC3VSvJlxNJzgOXKlWP06NGs\nXbuWGTNmWNJGSEgIs+fNJ75nd/LO5BERUZ958xdY0pYnsHN/OA2Dni9+ykuDmjG2x03kOw0GTP+M\n41m5lrRnl75QDnvmmDv7DQ4fOsSKDz9gxYcfFG1f9e91VK9e3WU57NIfZnnyIVCH4YKpRunp6cTF\nxbFy5UoqVap03p+npKQQExPDqjXJhIaGWR1HLoFWhBe7s8tsSXePZgBSU1Po0C6G5ORkwsKs/S4t\n/N6u3nsKvpWvMLWPgqxfOL50tEvyFseyQ6AffPABs2fPBqBixYo4HA58fDz+qgsREblMWHYItF27\ndjz11FP06dOH/Px8EhISqFChglXNiYiIG+gyiGJUqlSJV1991ardi4iITdjg6K8pHn8hvIiIuI8n\njwB1Uk5ERLySRoAiImKalZdBFHdHMX9/f8aMGYPD4aBhw4ZMmDABHx8fkpKSWLZsGeXKlWPYsGG0\nadOmxPZVAEVExLSzBdDsIdC//vM/31Fs06ZNvPLKKxiGwciRI2nWrBnjx48nOTmZm266iYULF7J8\n+XJyc3OJj4+nRYsWJd7XVQVQRERMs3IEeMcdd9C6dWvgjzuKffnll9x8880AtGrVii+++AIfHx8a\nNWqEn58ffn5+hIeHs3v3bqKiov5y/zoHKCIitlV4R7HnnnuO2NhYDMMoGnEGBASQlZVFdnY2lStX\nLnpPQEAA2dnZJe/bstQiInLZ83E48DE5BLzY902ZMoUnnniCuLg4cnP/uHViTk4OQUFBBAYGkpOT\nc872PxfEC7Z/6ZFFRETOsnJB3OLuKHbDDTewadMmADZs2EB0dDRRUVFs2bKF3NxcsrKyOHDgwEUt\nv6cRoIiImFeaVR1KeF9xdxSrX78+48aNY9q0aURERNC+fXt8fX3p168f8fHxGIbBY489hr+/f4nN\nqwCKiIgtXeiOYosWLTpvW1xcHHFxcZe0fxVAERExzQfwMTkAdPc5OBVAERExzZNvhaYCKH/JDmvx\nBbcZ7+4IAJxY/6y7I0gx3P0l6u08eUFcd49ARURE3EIjQBERMc3x+39m3+tOKoAiImKaj6MUk2Dc\nfAhUBVBERMyz8DpAq6kAioiIaZoEIyIi4mE0AhQREdNccTNsq6gAioiIaZ58CFQFUERETHNQijvB\nuPkyiMviHOAnq1bStFEUUddfTXyvHmRmZiqHF+aIbXkNx1YnFD0f0rkpX/7zQb5d+AhvjuuGX3lf\nAOqHVWPdzPv538Lh/Hf2ECLDr7A0F9jnMwEwDIMHBt3HK9NeclsGu/SHcni3CxbAmTNn/uXDLtLT\n0xk6eCBLk5azbece6tWLYFzCGOXwshz1w6ox6aH2RecUOrW6lmHdmnPPyLdp3H8mFf3KMSLuVgDe\nGteduR9spnG/mTz35nqWPt/Lslxgn88EYPeuXdzdLobl7ya5pX2wT38oR9mwcj1Aq3n8CHDd2jU0\niW5Kg4YNARgydBjLli7GMAzl8JIcFf3LM39cd0bPXF20rc9dN/HqO19wIusUhmHwyMsfseTf33HV\nFZWJrHMFSck7AFizaR8BFcpzU+SVZZ6rkF0+E4BZb/yD/gMG0q37pS0bU5bs0h/KUTYcjj8mwlzq\nw90F8ILnAIcPH170/ydPnuTQoUNERkZy+vRpKlWq5JJwFyMl5TBhYbWLnoeGhZGZmUlWVhZBQUHK\n4QU5Zo6KZd6HX7P9wLGibQ1qV6dGcCAfvtSPK6+ozBdbDzL2jTXcEFGTo79knfPlkpqeSWiNIL7b\ne7RMcxWyy2cCMH3G2aM36z9Ndmm7f2aX/lCOsuH4/WH2ve5U4gjwq6++olOnTjz00EP88ssvtG3b\nls8///yidn78+HFuv/12Dhw4UOqgF2I4ncVu9/X1taxN5bBPjiGdm5Jf4GTBqm/P2V7e15eY6Pr0\nHZ9Ei8GzqRZUkWceuAOfC9x7qcBp3W/bdvlM7MIu/aEcUmIBnDZtGkuWLCEoKIiQkBAWLVrEiy++\nWOKOz5w5w/jx46lQoUKZBL2Q2rXDSUv74zf31NRUgoODCQgIsLRd5bBHjn53N6LJNaFsfHMYH0zt\nS0X/8mx8cxgAKzZ8T9bJXM7kF7B0zTaa3VCbw8d+o2a1wHP2cdUVQaT+/FuZ5vozu3wmdmGX/lCO\nslG4HqDZhzuVWACdTic1atQoet6gQYOL2vGUKVPo1asXISEh5tNdhJg727F500b279sHwLw5s+gY\n28nSNpXDPjlaDp1D9IB/0HzQG3QetYhTuWdoPugNXl++ka5tbqCC39mj/LEtr2HLrlRS0zP54cgJ\nesTcAMAdNzfAaRjs+OHnMs9WyC6fiV3YpT+Uo2wU3gzb7MOdSrwOsFatWqxfvx6Hw0FmZiaLFy/m\nqquu+sv3vPfee1SrVo2WLVsyZ86cMgtbnJCQEGbPm098z+7knckjIqI+8+YvsLRN5bB/jtnvbya4\nckW+/OfatoMSAAAgAElEQVSD+Pr48N3eIwyf+REA/Z9O4vUnOzG6/+2czsunz/h3LJ1w4O6+sBu7\n9IdylA1PXhHeYZTwL//48eO88MILfPnllzidTpo3b05iYuJfjuz69OlT1Cm7du2ibt26vPHGG+eM\nJP8sJSWFmJgYVq1JJjQ0rHR/I7nsaEV4kYuTmppCh3YxJCcnExZm7Xdp4fd21CMz8a9q7khf7q8/\ns+214S7JW5wSR4DVq1dn2rRpZGdnU65cuYs6p7d48eKi/+/Xrx9PP/30BYufiIh4rsv6Vmh79uxh\nzJgxHDlyBICIiAimTJlCeHi45eFERMTePPkQaIkFcMKECYwcOZLbb78dgLVr15KQkMCiRYsuqoGF\nCxeWLqGIiNiWA/OTWWx/HWBubm5R8QO48847yc7OtjSUiIiI1S5YAI8cOcKRI0e45pprmDNnDhkZ\nGfz2228sWrSI6OhoV2YUERGb8uTrAC94CLRv3744HA4Mw2DTpk0sW7as6M8cDgeJiYkuCSgiIvbl\nybdCu2AB/PTTT12ZQ0REPNBlvSL8Dz/8wJIlSzh58iSGYeB0OklJSTnnUgcRERFPU+IkmMcee4yg\noCB27drFtddey/Hjx2n4+7IdIiLi3Tx5PcASR4BOp5MRI0aQn5/PddddR69evejVy9oFREVExDN4\n8nWAJY4AK1asSF5eHnXr1mXnzp34+fmRm5vrimwiImJ3pRn92X1F+HvvvZcHH3yQ1q1bs2jRIgYP\nHkzNmjVdkU1ERMQyJR4C7du3L507dyYwMJCFCxeyfft2brvtNldkExERm7ssZ4HOnDnzgm/as2cP\nw4cPtySQiIh4jsv6Ztgi7maXZYiu/vtH7o4AwO6XO7o7AuD+CQxiDw7M/yy4+yfoggVQIzwREbmc\naQQoIiKm+XARsyn/4r3upAIoIiKmXdbXAQKcPHmS3bt3YxgGJ0+etDqTiIh4CIfj7HqAZh7uPo1c\nYgH86quv6NSpEw899BDp6em0bduWzz//3BXZRERELFNiAZw2bRpLliwhKCiIkJAQFi1axIsvvuiK\nbCIiYnNmR3+FD3e6qHuB1qhRo+h5gwYNLA0kIiKew5PPAZZYAGvVqsX69etxOBxkZmayePFirrrq\nKldkExERmyvNSM7dI8ASD4E+++yzfPTRRxw9epQ77riDXbt28eyz9rgwWURExKwSR4DVq1dn2rRp\nrsgiIiIexpNvhVbiCLBt27bExMSc97CTT1atpGmjKKKuv5r4Xj3IzMxUDuVwS4YBLeuy9qnWrBlz\nO3MfaEr1QD8A+t1Wh5WjWpGc0Jrp/RrhV+7sP71bGlZn5ahWrB59O8seuYVrrwqyNN/SxYto1uQm\nmkU3ok2rFmzZ8o2l7V2IHX42ChmGwQOD7uOVaS+5LYOd+uNSOX6/GbaZh7vPAZZYABcuXMiCBQtY\nsGABb775Jr1796Zbt26uyHZR0tPTGTp4IEuTlrNt5x7q1YtgXMIY5VAOl2e4oXYVHmhbn66vfE67\nyZ/x0885PH7PNdwVVYv7WtUj/h9fccek/1ChvC/3t46gcoVyzL4/mokffs9dUz5j7Dvb+cfAJkXF\nsazt3bOHhKee5IOPP2HTN98y+qmx9I5z/b9lO/xsFNq9axd3t4th+btJbmkf7NUfZviU8uFOJbYf\nGhpa9KhTpw6DBw9m3bp1rsh2UdatXUOT6KY0aNgQgCFDh7Fs6WIMw1AOL8/h6gw7Dv9G6+c+Jet0\nPv7lfKhZtQIncvLoenNt5q7/gd9OnsEwICFpG+99nULdGgFknsrni72/AHDg52yyT+fTuG6wJfn8\n/f15fdZcrrzySgAaN4nmWFoaeXl5lrR3IXb42Sg0641/0H/AQLp1j3N524Xs1B9mnL0ZtsmHm7OX\neA7w66+/Lvp/wzDYt2+frVaET0k5TFhY7aLnoWFhZGZmkpWVRVCQtYeTlMPeOdyRId9p0O7GWkzp\n/Tfy8guYtmoP8x5oytaDfrw9rBk1gyrw9Q/HmfjhLnJy8wnwL0fLa2rw393pRIVXIfLKyoQE+VuS\nrU7dutSpWxc4+295zKjHuafjvfj5+VnS3oXY4Wej0PQZZ5d9W/9pskvb/TM79Ye3KbEAzpgxo+j/\nHQ4HwcHBTJ48+aJ23qVLFwIDAwEICwtj0qRJJmNemOF0Frvd19e3zNtSDs/K4a4Ma7ansWZ7Gr1u\nCWfhsGYUOA1aXlODwXO/JvdMAS/3bcSojtfw7Hs7eWDuZp7oeA0Jna5l8/4Mvtz7C2cKrP3NPycn\nhyH3DyQl5TAffvyJpW0Vxw4/G3bi6f1xWS6IW+juu+8mPj7+knecm5uLYRgsXLjQVLCLVbt2OF9v\n3lT0PDU1leDgYAICAixtVznsn8PVGepcUYkaQRX45ocMAJI2HmJizyj2pWWxemsa2afzAfjg6xRG\n3BWJwwE5uQX0eu2ron0kJ7Tmp/QcS/IBHD50iO5d7uXqa65l9dpPqVixomVtXYgdfjbsxNP747Ke\nBbpkyRJTO969ezenTp1i0KBB9O/fn++++87UfkoSc2c7Nm/ayP59+wCYN2cWHWM7WdKWcnhWDldn\nCAmqwMz7GhMccPaQYufoMPYczWTplwe5p9GV+Jc/+8+tXVQtth36FcOAtx68mRtrVwGgw01XcqbA\nYNcRa2YAZmRk0O6O1tzbuQsLFi91S/EDe/xs2Imn94cn3wz7ou4E079/f/72t7/h7//HuYmSFsyt\nUKEC999/Pz169OCnn37igQceYPXq1ZQrV7YrMIWEhDB73nzie3Yn70weERH1mTd/QZm2oRyemcPV\nGb7+IYOZa/bxziO3kO80+Pm30wyZ+zWpJ05RpZIfK0e1wsfhYEfKbzz/wTYARiz4lim9/0Z5Xx9+\nzjzNA/O+LqEV8+bOfoPDhw6x4sMPWPHhB0XbV/17HdWrV7es3f/PDj8bdqL+cB+HUcJUo5kzZxa7\nvaQCmJeXh9PppEKFCgB0796d1157rWgG2p+lpKQQExPDqjXJhIaGXWx2EZe6+u8fuTsCALtf7uju\nCID77+Mo50tNTaFDuxiSk5MJC7P2u7Twe7vTM28SWL2mqX1kHz/GhxMGuSRvcS44HHv//ffp0qVL\niYXuQt5991327t3L008/zbFjx8jOzj7nptoiIuL5LstzgAsWlG4I3r17d7KysujduzePPfYYEydO\nLPPDnyIi4l6X9XJIZvn5+fHyyy9btXsREZFSuWAB3LdvX7H3/DQMA4fDQXKy+y4cFRERe3D8/p/Z\n97rTBQtgnTp1mDNnjiuziIiIh3GU4lCmu88BXrAAli9fntDQUFdmERERD+NDKRbELdMkZdh+48aN\nXZlDRETEpS44Ahw/frwrc4iIiAdylGJdv5Led+bMGRISEkhNTSUvL49hw4bRoEEDxowZg8PhoGHD\nhkyYMAEfHx+SkpJYtmwZ5cqVY9iwYbRp06bE9nVdgoiImFaayxlKet+KFSuoWrUqU6dO5ddff6Vz\n585cc801jBw5kmbNmjF+/HiSk5O56aabWLhwIcuXLyc3N5f4+HhatGhR4konKoAiImKalRfC33XX\nXbRv3x44ewWCr68vO3fu5OabbwagVatWfPHFF/j4+NCoUSP8/Pzw8/MjPDyc3bt3ExUV9Zf7d/c5\nSBERkWIFBAQQGBhIdnY2I0aMYOTIkUWX4hX+eVZWFtnZ2VSuXPmc92VnZ5e4fxVAEREx7exlEA5T\nj4sZOR49epT+/fvTqVMnYmNj8fH5o2zl5OQQFBREYGAgOTk552z/c0G8EBVAERExzcpbof3yyy8M\nGjSIUaNG0b17dwCuu+46Nm06u37ihg0biI6OJioqii1btpCbm0tWVhYHDhwgMjKyxOw6BygiIqZZ\neQ5w1qxZZGZm8vrrr/P6668DMHbsWJ5//nmmTZtGREQE7du3x9fXl379+hEfH49hGDz22GPnLN93\nISqAIhdpz7RYd0cAoEaft90dAYD0xQPcHUEuc4mJiSQmJp63fdGiRedti4uLIy4u7pL2rwIoIiKm\n+eDAx+Q9Pc2+r6yoAIqIiGmevB6gCqCIiJjmoBQ3wy7TJJdOs0BFRMQraQQoIiKmFV7TZ/a97qQC\nKCIipukcoIiIeCVPHgHqHKCIiHgljQBFRMQ0Tz4EelmMAD9ZtZKmjaKIuv5q4nv1IDMzUzmUwzYZ\n3JWjY3RtUuf3PmdbaPVK7HmjB9Ur/3GbqJbX1+Kziffw5YuxfPp8B5rUv8LybN78udg5hxkOzhYS\nMw9dBlFK6enpDB08kKVJy9m2cw/16kUwLmGMciiHLTK4K0f9WpV5vl80Pn+6QKt3qwj+/fTdXFWt\nUtG28r4+vP1oKx6Z8xW3PvkRL763jbnDb7M0mzd/LnbOYVbhivBmH+7k8QVw3do1NIluSoOGDQEY\nMnQYy5YuxjAM5fDyHHbI4I4cFf18mTu8JQkLvinaViu4Ih2bhtNt8rpzXnumwEnksH+x7acMAOqG\nBJKRlWtJrkLe+rnYPYc38vgCmJJymLCw2kXPQ8PCyMzMJCsrSzm8PIcdMrgjx6sP3ML8dXvZcSij\naFvaiVP0efk/7En97bzX5xcY1KhSgT1v9OD5vtFMX7HDklyFvPVzsXsOsxylfLiTxxdAw+ksdruv\nr69yeHkOO2RwdY7B7a6mwGmw8D/7L+l96b+d5uph/yJm3CpeH9aCBlcGlXm2Qt74uXhCDrPMLoZb\nmssnyiy7lTufPXs2PXv2pGvXrvzrX/+ypI3atcNJSzta9Dw1NZXg4GACAgIsaU85PCeHHTK4Okef\n2xvQuH51vpgSy/Ixd1DRz5cvpsRSK7hisa8Pqlie2KbhRc+3/pjBjoMnuK521TLPVsgbPxdPyGGW\nRoDF2LRpE99++y1Lly5l4cKFpKWlWdJOzJ3t2LxpI/v37QNg3pxZdIztZElbyuFZOeyQwdU52oxd\nSbMnVtBi9Ed0m7yOU3kFtBj9EWknThX7+gKnwT8evJXmV9cA4JqwqkSGVuGb/b9Ykg+883PxhBze\nyLLrAD///HMiIyN5+OGHyc7O5sknn7SknZCQEGbPm098z+7knckjIqI+8+YvsKQt5fCsHHbIYKcc\nxcnJzaf3S+uZPOBmyvv6kHumgEEzNnAk46RlbdqlP5SjbHjydYAOw6KpRomJiRw5coRZs2aRkpLC\nsGHDWL16dbHTXlNSUoiJiWHVmmRCQ8OsiCNy2dCK8HIhqakpdGgXQ3JyMmFh1n6XFn5vPzpjCcEh\ntUzt48TPabw6It4leYtj2QiwatWqRERE4OfnR0REBP7+/mRkZFC9enWrmhQRERcrvKjd7HvdybL2\nmzRpwn//+18Mw+DYsWOcOnWKqlWtO7EuIiJyKSwbAbZp04avv/6a7t27YxgG48eP95hpvSIicnFK\nc0cXd98JxtKbYVs18UVEROyhNJczuPsyCK0GISIipp2dBWp2BFjGYS6RCqCIiJimSTAiIiIeRiNA\nERExrzTLGl3Ok2BEROTypkkwIiLilTz5Vmg6BygiIl5JI0ARETHNBwc+Jg9mmn1fWVEBFBER0zz5\nEKgKoIiImOb4/T+z73UnnQMUERGvpBGgiIexyzp8wXe/6O4IAJz4RPccdicdAhUREa/kKMUkGHcf\nAlUBFBER0zx5BKhzgCIi4pU0AhQREdMclGIEWKZJLp0KoIiImObJl0GoAIqIiGk+jrMPs+91J50D\nFBERr6QRoIiImKZDoCIi4p1KcRmEu2fBXBaHQD9ZtZKmjaKIuv5q4nv1IDMz06tzGIbBA4Pu45Vp\nL7ml/UJ26A87ZPD2HLG3NuDYB4+et33ZhM68MvyOoucdmtcndfkjbJw1oOgRWNHP0mze/LmUFUcp\n/3Mnjy+A6enpDB08kKVJy9m2cw/16kUwLmGM1+bYvWsXd7eLYfm7SS5v+8/s0B92yODtOeqHBjNp\nSBt8/t9sh7/H3cytN4Sds635daFMf/drmj/4dtEj+1SeZdm8+XORszy+AK5bu4Ym0U1p0LAhAEOG\nDmPZ0sUYhuGVOWa98Q/6DxhIt+5xLm33/7NDf9ghgzfnqOhfjvmj72H07PXnbG/1t3DujK7HvI+/\nO2d78+tDaX1TOF/8oz/rpvWmxY3nFsiy5q2fS1krnAVq9uHW7O5tvvRSUg4TFla76HloWBiZmZlk\nZWV5ZY7pM2YS37efS9ssjh36ww4ZvDnHzJHtmbdyK9t/+Llo25XVA3npobYMnPwxBc5zv+AzMk8x\ne8W3tHh4AeP/uYF3nu5C6BWBlmQD7/1cypqD0hwGdS+PL4CG01nsdl9fX6/MYRd26A87ZPDWHENi\nbyK/wMmCf28v2lbO14cFCbGMeuNT0jJyzntPr2c+YMUX+wD4cmcqG3em0rZJ3TLPVsgbPxcrFN4L\n1OzDnSybBfree+/x/vvvA5Cbm8uuXbv44osvCAoKKtN2atcO5+vNm4qep6amEhwcTEBAQJm24yk5\n7MIO/WGHDN6ao1+7G6noX46NswbgV86Xin7l+GXFSHx8HEx5sA0ANYMD8PXxwd+vHE/NXs+Qexsx\ndenGon04HA7O5BdfHMqCN34uci7LRoBdu3Zl4cKFLFy4kOuvv57ExMQyL34AMXe2Y/Omjezfd/Y3\nx3lzZtExtlOZt+MpOezCDv1hhwzemqPlIwuJHjKf5g++Teex73IqL5+q90wj6O6Xiya4zPt4K8s/\n281D01aTdSqPB+9tROfbIgH4W/0Qoq+uxdqvf7QkH3jn52IFRykf7mT5dYDbt29n//79TJgwwZL9\nh4SEMHvefOJ7difvTB4REfWZN3+BJW15Qg67sEN/2CGDclwcp9Ogx4T3mfZwDIn9W5DvdNLvhY84\nnnnKsjbt0h92yWGWw+HAx+SxTIebj4E6DIunGg0fPpy+ffvSvHnzC74mJSWFmJgYVq1JJjTU2plf\nIlI2tCK8/aSmptChXQzJycmEhVn7XVr4vf3Sm+9To+ZVpvaRfuwITwzq4pK8xbF0EkxmZiY//vjj\nXxY/ERERd7D0EOjXX3/NLbfcYmUTIiLiTqU5mXe5zgIF+PHHH90yrBUREdfQzbAvYPDgwVbuXkRE\n3Kw01/O5+zpAj78QXkRExAwthyQiIqZ58ClAFUARESkld1cyk1QARUTENE+eBKNzgCIi4pU0AhQR\nEdM0C1RERLySK26GvXXrVvr1O7vO6cGDB+nduzfx8fFMmDAB5+/LSSUlJdG1a1fi4uJYv379X+2u\niAqgiIiYZ3EFnDt3LomJieTm5gIwadIkRo4cyZIlSzAMg+TkZNLT01m4cCHLli3jn//8J9OmTSMv\nL6/EfasAioiIbYWHh/Paa68VPd+5cyc333wzAK1ateLLL79k27ZtNGrUCD8/PypXrkx4eDi7d+8u\ncd8qgCIiUgoO0/9dzBCwffv2lCv3x3QVwzCKllEKCAggKyuL7OxsKleuXPSagIAAsrOzS9y3JsGI\niIhprp4E4+Pzx7gtJyeHoKAgAgMDycnJOWf7nwvihagAiogpdlmHr/4j77s7AgAHXuvi7ghu4eo7\nwVx33XVs2rSJZs2asWHDBpo3b05UVBTTp08nNzeXvLw8Dhw4QGRkZIn7UgEUERHzXFwBR48ezbhx\n45g2bRoRERG0b98eX19f+vXrR3x8PIZh8Nhjj+Hv71/ivlQARUTE1sLCwkhKSgKgXr16LFq06LzX\nxMXFERcXd0n7VQEUERHTPPlWaCqAIiJimiffCUYFUERETPPk5ZB0HaCIiHgljQBFRMQ8Dx4CqgCK\niIhpmgQjIiJeyZMnwVwW5wA/WbWSpo2iiLr+auJ79SAzM1M5lKOIYRg8MOg+Xpn2klvaB/v0BXhf\nf9x3ewSfjosheVwMbz7YnOqV/ZjzwM2sSWhT9Ng1rSPzhzUH4NbIK1j9VBvWjm3Lv0bexnWhQZZl\n+zM7fC7exuMLYHp6OkMHD2Rp0nK27dxDvXoRjEsYoxzKAcDuXbu4u10My99NcnnbhezSF+B9/XFj\neFUevLMBnaZ+Rsxzyfz4czZPxl7HkLmbaTdxPe0mrmfU4m/JPHmGscu2UrlCOeYObcbz7+3gzhc+\nZczS75j1wM34lbP2q9IOn0tpWLkWoJU8vgCuW7uGJtFNadCwIQBDhg5j2dLFGIahHMrBrDf+Qf8B\nA+nW/dLuEFGW7NIX4H39sf3Qr9w2fi1Zp/PxL+dDraoVOJHzxzpx5X0dTB/QhAn/2saRE6eoFxJI\n1qkzfL4nHYADx7LJOpVPk3rVyjzbn9nhcykVD62AHl8AU1IOExZWu+h5aFgYmZmZZGVlKYdyMH3G\nTOL79nNpm/+fXfoCvLM/8p0G7f92Jd9MuotmDa/gnS8PFv1Z7xZ1OfbbaVZvPQrADz9nE+BfjlbX\nhgDwtzpVufqqyoRUqWBJtkJ2+FzMMr8YkvnJM2XF4wug4XQWu93X11c5lMMW1Bfnckd//HvrUW4c\ntYppH+9m8YgWRZMvHmjbgFdX7Sl6XfbpfAa+sZFH7opk7di2dG8Wzhd7fuFMQfGZxbNZVgDPnDnD\n448/Tq9evYiPj+fAgQOWtFO7djhpaUeLnqemphIcHExAQIAl7SmHZ+WwA/XFuVzZH3VrBNC0fvWi\n58u+/ImwapWoWsmP68Oq4Ovr4Kt9vxT9ucMBJ3Pz6fHK59z5wqeMS9pGnSsC+OnnkhdX9VaFs0DN\nPtzJsgL42WefkZ+fz7Jly3j44YeZPn26Je3E3NmOzZs2sn/fPgDmzZlFx9hOlrSlHJ6Xww7UF+dy\nZX+EVKnAG/c3JTjAD4CuN9dmz5FMTuTkcUvkFXzx+7m+QoYBC4bfSlR4VQA6Nr6KMwVOvk9136xd\nuzN7+s8OpwEtuw6wXr16FBQU4HQ6yc7OPmdJ+7IUEhLC7Hnzie/ZnbwzeURE1Gfe/AWWtKUcnpfD\nDtQX53Jlf2zef5wZq/fw7t9bUlDgJO230wyatRGAeiGBpBw/ed57hr/5NVP7NqK8rw8/Z57m/tkb\nLcl22fDgO8E4DIumoh09epSHHnqIkydPcuLECWbNmkXjxo2LfW1KSgoxMTGsWpNMaGiYFXFE5DKl\nFeH/kJqaQod2MSQnJxMWZu13aeH39rx3VlLzyqtM7ePY0SMM7nmPS/IWx7JDoG+99Ra33XYb//73\nv/nwww8ZM2YMubm5VjUnIiJucHYA6IlzQC08BBoUFET58uUBqFKlCvn5+RQUFFjVnIiIuIEn3wrN\nsgJ43333kZCQQHx8PGfOnOGxxx6jUqVKVjUnIiJu4MGnAK0rgAEBAbz66qtW7V5ERKRUtBqEiIiY\n58FDQBVAERExTesBioiIdyrNHV0u1zvBiIiI2JlGgCIiYpoHnwJUARQRkVLw4AqoAigiIqZ58iQY\nnQMUERGvpBGgiIiYpluhiYiIV/LgU4AqgCIiYp6DUowAyzTJpVMBFNuzaMnKS+Zw9/EaKZYd1uED\nqDVgkbsj4JP7KzXcHcKDqACKiEgpeO5BUBVAERExTZNgRETEK3nu+E/XAYqIiJfSCFBERMzz4NUg\nVABFRMQ0T74VmgqgiIiY58EnAVUARUTENA+uf5oEIyIi3umyKICfrFpJ00ZRRF1/NfG9epCZmakc\nygHA0sWLaNbkJppFN6JNqxZs2fKNyzOAPfrCTjng7B1+Hhh0H69Me8ltGdzRH/c0CePQvDgAfBwO\nJveLZvPUWP73cicGxjQsel3VAD/mPNSCDS90YPPUWHreVs/ybGYUXgdo9uFOHl8A09PTGTp4IEuT\nlrNt5x7q1YtgXMIY5VAO9u7ZQ8JTT/LBx5+w6ZtvGf3UWHrHdXNpBrBHX9gpB8DuXbu4u10My99N\nckv74J7+iKhZmefim+Dz+zf/wJiGRNSqzC2jP6bNuE8Ydtc1NI6oDsAbQ2/lSMZJWo1dRedJyUzp\nH81V1SpZms8MRyn/cyePL4Dr1q6hSXRTGjQ8+5vTkKHDWLZ0scvvH6kc9svh7+/P67PmcuWVVwLQ\nuEk0x9LSyMvLc1kGsEdf2CkHwKw3/kH/AQPp1j3O5W0XcnV/VPTzZc5DLRi7eEvRto7RtVn82QEK\nnAa/nczjva8O0vO2elQN8KP1jbWY8t42AI5knCRm/GpOZOdakq1UHKV8uJHHT4JJSTlMWFjtoueh\nYWFkZmaSlZVFUFCQcnhxjjp161Knbl3g7OG2MaMe556O9+Ln5+eS9gvZoS/slANg+oyZAKz/NNml\n7f6Zq/tj+v3NeOvTfew8dOKPNqtXIjXjZNHz1IyTXB9elYialTn26yke7nAtd/ztKvzL+fLayu85\nkJZV5rm8mccXQMPpLHa7r6+vcigHADk5OQy5fyApKYf58ONPXN6+XfrCLjnswpX9cf8dkeQXGCz6\n7ADhVwQUbfcp5iRYgdOgfDkf6oZUJuvUGe56Zg31agbyybj2HEjLYutPGWWerzQ0C7QYeXl5PP74\n48TFxTFo0CB++uknS9qpXTuctLSjRc9TU1MJDg4mICDgL96lHN6S4/ChQ7Rt1QJfX19Wr/2UqlWr\nurR9sE9f2CWHXbiyP+JbRdAoojr/ndiBpCfbUNHPl/9O7MCRjJPUqlqx6HVXVavIkYyTpJ04Oypc\nsuEHAH48ls3GvT/TpH71Ms9WWpoEU4ykpCQqVapEUlISiYmJPPfcc5a0E3NnOzZv2sj+ffsAmDdn\nFh1jO1nSlnJ4Vo6MjAza3dGaezt3YcHipVSsWLHkN1nADn1hpxx24cr+iBm/mlvHfEzLhFXEvbie\nU3kFtExYxcffHKbv7fXx9XFQpVJ5ujavy8pvDnMwPYfvfjxO75YRANQIqsDNDWvw7Y/2Gv2BZ0+C\nsewQ6P79+2nVqhUAERERHDhwwJJ2QkJCmD1vPvE9u5N3Jo+IiPrMm7/AkraUw7NyzJ39BocPHWLF\nhx+w4sMPirav+vc6qld33W/SdugLO+WwCzv0xz/X7aVeSCCfT7oHv3I+zE/exxe7fwag7yuf8dJ9\nN9AvETUAAA45SURBVDMwpiE+Dgcvvr+db3847tJ8lzuHYdGUp3feeYetW7fywgsvsHXrVnr37s2O\nHTuKPb6ekpJCTEwMq9YkExoaZkUc8WBaEV48gW1WhN/+GsnJyYSFWftdWvi9/d7KtVx1VaipfRw5\nkkrXe+50Sd7iWHYItFu3bgQGBhIfH8/atWu5/vrrvfZku4iI2I9lh0C3b9/OLbfcQkJCAtu3b+fI\nkSNWNSUiIm7ioBQrwpdpkktnWQGsU6cOr776KrNmzaJy5cq88MILVjUlIiJyySwrgNWqVeOtt96y\navciImIDWg9QRES8Ummu53P3vDIVQBERMU13ghEREfEwGgGKiIh5HjwEVAEUERHTztY/s5Ng3EsF\nUERETPPkSTA6BygiIl5JI0ARETHNg08BqgCKiEgpWFgBnU4nTz/9NHv27MHPz4/nn3+eOnXqmGzs\nfDoEKiIipVCatQD/ugKuW7eOvLw83nnnHR5//HEmT55cpsltMQIsKCgA4FhampuTiB1pOSTxBD65\nv7o7Aj55mcAf36mu8PMx89/bJb13y5YttGzZEoCbbrqJHTt2mG6rOLYogOnp6QAM7N/HzUlERMyp\n4e4Af5Kenl6mhwqLExgYSJUqVUr9vV2lShUCAwOL/bPs7Oxz/szX15f8/HzKlSub0mWLAnjDDTew\nePFiatSooTUDRURMKigoID09nRtuuMHytqpWrcqaNWvIzs4u1X4CAwOpWrXqBf8sJyen6LnT6Syz\n4gc2KYAVKlQgOjra3TFERDye1SO/P6tateoFi1dZaNy4MevXr6dDhw589913REZGlun+HYZdTrCI\niIj8SeEs0L1792IYBhMnTqR+/fpltn8VQBER8Uq6DEJERLySCqCIiHglFUAREfFKKoAi4lKadiB2\n4dEF0DAM9uzZw969e90dxRacTif/+c9/WLdunVtzGIbBjh072Llzp1szbNiwwW3tF3I6nYwdO5Zv\nv/3W7Tn+9a9/ub1PDMOgS5cuzJgxw+05du3axe7du92eY+/evXz//fduzeGtbHEdoBmGYTBs2DCC\ng4PJyMggNDSU8ePHuy3P22+/zYABA9zWvmEYPPzww9SqVYv//e9/bNiwgWeffdYtOQo/lxMnTtCq\nVSvi4+NdnmPLli088sgjvPTSS7Rp06ZML569WE6nk1GjRhEVFUWjRo1wOp34+Lj+d07DMBg0aBDR\n0dHk5eWdd3cNV3E6nYwfP55KlSoRHBxclM3Vt5gr/BkNCgoiMzOTW265xS3/dp1OJw8//DChoaEc\nPnyY2rVrk5iY6PIc3sxjR4BJSUlUr16dSZMmMWPGDL7//nueeeYZt2TJyclhyZIlTJs2zS3tw9kC\nXLVqVSZMmMC7775LZmYmWVlZLs+xZMkSqlSpwqRJk4iLiyMjI4MDBw64NIPT6aRGjRqEhYWxYMEC\nNm/ezOHDh8nLy3NpjokTJ+J0OhkwYACPP/44f//73xk3bpzLc2zYsIHIyEiGDx/ON998w9SpUxk/\nfrzLD0UmJCRQq1YtnnvuOdatW0dGRoZb7q/61ltvUaVKFV588UX69+9Pbm4uKSkpLs/x9ttvExQU\nRGJiIq+//jqffvopEyZMcHkOb+axBbB+/fo4HA6OHTuGv78/CxYs4Pvvv3dLEdq+fTvVqlUjNTWV\nhIQEl7cPEBYWRs2aNTl9+jS//vorx48fd8u5lrCwMKpUqQLA2rVrSU5OJiEhgREjRrgsg4+PD1dc\ncQV9+/Zl7NixTJ06lT59+vDTTz+5LANAr169OHjwIL1796ZFixY8/fTT/Prrrzz//PMuzRESEkJK\nSgrPP/88LVq04NFHHyUtLc2lR0wyMzNp27Ytw4cPp379+jRo0IDTp08DZ39hcaWGDRty6tQp9uzZ\nw4oVK1i9ejWjRo3i0UcfdWmOevXqERgYyPHjx/H19eXRRx9lx44dvPnmmy7N4c08ugBWrFiRrVu3\nkpGRgZ+fHzNmzODUqVMuz1KvXj3i4+OZPHkyubm5jBs3zuUZGjduTM+ePalQoQIA+fn5BAUFsWLF\nChYtWuSyHE2aNGH48OEA/9fenYdE1XcBHP+ajTY5RZllEWVPC2mZtlIUbSZFRhqFBmWlfxQh9keJ\nrZMaGq2SpFFQEW06BZlTmS1qqNlmC1Fo2KZYOZZWDuOko47vH+KAb+Xz9D69M9Wcz3/eO3Pv4Xq5\nZ+7v/u45zJo1i/Pnz3PmzBnMZjM1NTVWi8NkMvH27VsMBgMODg64ubnx7t07q15shw4dysqVKxk4\ncCBz5syhR48eJCUlodfrrXqeDhw4kL/++osPHz7g6emJq6srhw4dorq6mtraWqvE0L17d2bPnm35\nu3PnzuzZswdo/cFizR9ro0ePZt68eWzfvp2ysjLS09NJS0ujoaGByspKq8Xh5eWFyWQiLS0NjUZD\nTk4OUVFRGI1Gq8Vg737bBNizZ08WL15MXl4eBQUFvH37locPH/Ly5UsaGhqsGou7uzv+/v4oFAq2\nbNlCU1MT69ats2oMrq6u9OvXDwClUom3tze5ubmcO3eOiRMnWi0OlUpF9+7dAfD396exsZG8vDz0\nej3Ozs5Wi6Nbt258+vSJvXv3kpCQQGxsLBqNxuoXl5kzZ7J69WoUCgU1NTUUFhZafWjaxcWF4OBg\nTCYTN2/epKSkhPz8fGpra63+bLQt0W3YsIEePXqQkZEBWLfVlEqlYs6cOYSGhjJq1Cjq6+u5ceMG\nBoMBFxcXq8Xh7u7OqlWr6Nu3L5WVlSxfvhxHR0dKS0tpbGyU2bJW8NuXQnv9+jWXLl2itLSU+vp6\n1q9fz7Bhw2wa08ePH0lKSiIyMpI+ffpYff86nY4ZM2bg6+vLrl27GDRokNVjALhy5Qo5OTlUV1ez\nZcsWhg4datX9FxcX8/nzZyZPngyA0Wika9euVo0BWi/6586dIysrC7PZzObNm21yjlZUVKDVann0\n6BEKhYK1a9cyfPhwq8fR0tJCS0sL6enpvH//nvDwcJRKpdXjqKioICUlBaVSSVlZGWq12urnaJvi\n4mLy8vLIyclhx44dNr+G2YvfPgFC63CfXt/aCNLV1dXG0bSy1Yw/gC9fvqBWq1mzZo3Nkh+09vLS\n6/U4Ojri7u5uszja/he2mHHYxmAw8OXLF8twrK2YzWbq6uowm82WZ7W2UltbS0tLy/+1m8DfaZss\n1rlzZ5ueo/X19ZSXl9O1a1cGDBhgszjszR+RAMXXTCYTTk5Otg5DCCF+WZIAhRBC2KXfdhKMEEII\n8W9IAhRCCGGXJAEKIYSwS5IAxS/rzZs3eHt7ExQUxIIFC5g3bx7h4eHodLr/eZvp6els3LgRgJUr\nV1JVVfXdz+7fv5/79+//0Pa/9VpBcnIyycnJHX7Pz8/vh8px/ZNtCiE6JglQ/NL69OmDVqslIyOD\nzMxMvL29iY+P/ynbPnz4cIdT34uKimhubv4p+xJC/Hp+224Qwj6NHz+e3NxcoPWuycfHh5KSElJT\nUykoKOD48eOYzWZGjhxJbGwszs7OZGRkcPDgQVQqFf3797e8DO/n58eJEyfo3bs327Zt48GDBygU\nCiIiIjCZTDx9+hS1Wk1KSgpdunSx1PLs0qULW7duZcSIEbx584bo6GiMRiO+vr5/G/+pU6fQarWW\ndwKTkpIYMmQIACkpKTx79gxnZ2e2bduGp6cn1dXVxMTEoNPpcHBwICoqyvJivxDi35E7QPHbaGxs\nJCsri7Fjx1qWTZs2jatXr/Lx40fOnj2LRqNBq9XSq1cvjh49SlVVFXv37uX06dOcOXOGurq6r7Z7\n8uRJjEYjWVlZHDt2jAMHDhAQEIC3tzcJCQkMHz6cDRs2EB0dzfnz54mPj2ft2rUAxMfHs3DhQrRa\nbbu4vsVgMJCdnc3Jkye5dOkS/v7+pKamWtZ7eHiQkZFBRESEZZh2+/btLFq0iPT0dA4ePEhMTAwG\ng+FnHE4h7J7cAYpf2vv37wkKCgJaX+738fEhKirKsr7truvu3buUl5cTEhICtCbLESNG8OjRI8aM\nGWOpvjJ//nzu3LnTbh9FRUWEhITQqVMnevfuTWZmZrv1dXV1PH36lE2bNlmWGY1GPn36xL1790hM\nTAQgMDCww35uKpWKxMREMjMzKSsro6CgAC8vL8v64OBgAKZPn050dDR6vZ5bt27x6tUrSwPZpqYm\nKioqfuAICiG+RxKg+KW1PQP8nrYC283NzcydO9eSgOrq6mhubub27dvtOkB8q/jzfy8rLy+3FBaH\n1vJhTk5O7eLQ6XSWEl5ttSQcHBw6LLVWWVnJsmXLCA0NZdq0abi5uVFSUmJZ7+jo2O7zCoUCs9ls\n6fUIUFVVhZubG9nZ2d/djxDin5EhUPFHmDhxItevX7f0QYyLi+P48eOMGzeOx48fU1VVhdls5vLl\ny199d8KECWRlZdHS0kJNTQ2hoaGYTCYcHR1pbm6mW7duDBo0yJIACwsLWbp0KQCTJ0/mwoULAFy7\ndq3DZrdPnjzBw8ODsLAwfH19yc/PbzfJ5uLFi0BrH8XBgwejVCqZNGmSZZj0xYsXBAYG2qTllxB/\nIrkDFH8ET09PIiMjWbFiBWazGS8vL1atWoWzszNqtZqwsDCUSuU3q/0vWbKEhIQEAgMDAdi6dSsq\nlYqpU6cSGxvLrl272LNnD3FxcRw5cgSFQsG+fftwcHAgJiaG6OhoNBoNo0aN6rCdzpQpU0hLSyMg\nIAAnJyd8fHx4/vy5ZX1ZWRlBQUG4uLiwc+dOANRqNTExMcyfPx+A3bt3o1KpfuahE8JuSS1QIYQQ\ndkmGQIUQQtglSYBCCCHskiRAIYQQdkkSoBBCCLskCVAIIYRdkgQohBDCLkkCFEIIYZckAQohhLBL\n/wF+J0342YvgLgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x261adc2bdd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Look at confusion matrix \n",
    "\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "\n",
    "# Predict the values from the validation dataset\n",
    "Y_pred = model.predict(X_val)\n",
    "# Convert predictions classes to one hot vectors \n",
    "Y_pred_classes = np.argmax(Y_pred,axis = 1) \n",
    "# Convert validation observations to one hot vectors\n",
    "Y_true = np.argmax(Y_val,axis = 1) \n",
    "# compute the confusion matrix\n",
    "confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n",
    "# plot the confusion matrix\n",
    "plot_confusion_matrix(confusion_mtx, classes = range(10)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "1b8a5cdc-9122-4e31-b9fa-0f6b57d33fc8",
    "_uuid": "ecb928433299b163ecc1f6c4e66d4ddcf38fe898"
   },
   "source": [
    "Here we can see that our CNN performs very well on all digits with few errors considering the size of the validation set (4 200 images).\n",
    "\n",
    "However, it seems that our CNN has some little troubles with the 4 digits, hey are misclassified as 9. Sometime it is very difficult to catch the difference between 4 and 9 when curves are smooth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "ef54d686-6f79-4d96-a5a0-a64657bd742e",
    "_execution_state": "idle",
    "_uuid": "afd59cae1115188b77abd3471e5e89790cef80a0"
   },
   "source": [
    "Let's investigate for errors. \n",
    "\n",
    "I want to see the most important errors . For that purpose i need to get the difference between the probabilities of real value and the predicted ones in the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "_cell_guid": "7b0f31b8-c18b-4529-b0d8-eb4c31e30bbf",
    "_execution_state": "idle",
    "_uuid": "e7a3d6449b499a29db224e42e950f21ca1ec4e36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAFiCAYAAAC3VNLzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYVHX7P/D3OODGiKTiUirh/qipkfuaFqEEAl/tUjQ0\n4RG/6qNShiKRmCL6hCVGkmnPNxOX3DLMrURTCpesTEGLykdckUVFVmWAz+8Pf4wOZ2CGcebADO/X\ndXVdfm7PnHMP3nHPOeczn6MQQggQERGRbOrVdAJERER1DZsvERGRzNh8iYiIZMbmS0REJDM2XyIi\nIpmx+RIREcnMIpvv9evX8Y9//ANeXl6a/8aOHYtdu3Y98b5nzJiBr776CgDg5eWF3NzcSrfNy8vD\nlClTqn2MQ4cOwc/PTxI/ffo0PDw89L6+a9euuHPnTrWOGRISgv/85z96tzt8+DA8PT3h5eUFPz8/\nXL16tVrHsVasOfPVXGpqKvz8/ODt7Y3/+Z//QUpKSrWOY81Yd9ZbdzayHs2EGjZsiPj4eM04IyMD\nHh4e6NmzJ7p162aSYzy+f13u3buH5ORkkxyrNrh//z6Cg4MRHx8PJycnbNy4EREREVi/fn1Np1Yr\nsOZMr6ioCAEBAVi+fDlGjBiBhIQEvP322zh06FBNp1ZrsO5MrzbUncU234patWoFJycnpKWl4eLF\ni9i1axeKioqgUqkQFxeHnTt3Ytu2bSgrK4ODgwPeffdddOzYERkZGQgJCUFmZiaefvpp3L59W7PP\nrl274uTJk2jWrBk+/fRT7NmzBzY2NnBycsLKlSuxaNEi3L9/H15eXvjqq6+QlpaG5cuXIycnB6Wl\npfDz88P48eMBAGvWrME333wDBwcHODk56X0/ly9fxtKlS1FYWIjMzEx069YN0dHRaNCgAQAgOjoa\nycnJKCsrQ1BQEEaOHAkAlb7PyiQnJyMsLAzx8fEoLS2FEAJ5eXkAgIKCAs3xSIo19+Q1l5SUhHbt\n2mHEiBEAgJdeeglt27Y1+t+kLmDdWUndCQt07do10adPH63Yr7/+Kvr16ydu3rwpdu/eLfr16yfy\n8vKEEEKcPn1aTJo0SRQWFgohhPjhhx/EmDFjhBBCzJo1S6xevVoIIURaWpro06eP2L17txBCiC5d\nuojbt2+LhIQE8corr4icnBwhhBCRkZEiNjZWKw+1Wi3c3d1FSkqKEEKI3NxcMWbMGHH27Flx+PBh\n4e7uLvLy8oRarRaBgYHi9ddfl7yvU6dOiVdffVUIIcTKlSvF119/LYQQori4WHh4eIhDhw5p8vr0\n00+FEEKkpqaK/v37i9u3b1f5PhcuXCg+++wzvT/bPXv2iB49eoghQ4aIQYMGibS0NL2vqQtYc+ap\nufXr14s5c+aIRYsWCR8fHzF16lTN+yHWnTXXncWe+ZZ/CgOA0tJSPPXUU4iKikKbNm0APPwkp1Kp\nAADHjh3DlStXMHHiRM3r7927h5ycHJw4cQILFy4EADg5OWHAgAGSY508eRKjR49G06ZNAQCLFi0C\n8PB+TLm0tDRcvXoVoaGhWjlevHgRly5dgqurqyafcePGIS4ursr3FxwcjKSkJGzYsAFpaWnIzMxE\nYWGh5u99fX0BAF26dEHHjh1x9uxZ/PLLL5W+T0OkpqZi7dq1OHDgANq3b49NmzZhzpw5iI+Ph0Kh\nMGgf1ow1Z/qaKykpwfHjx7Fp0yb07t0bCQkJCAwMxPfff4/69esbtA9rx7qzzrqz2OZb8T5IRY0b\nN9b8uaysDF5eXggODtaMMzMz0bRpUygUCojHlre2sZH+SJRKpVbzyc3NlUxOKC0thb29vVZO2dnZ\naNKkCaKiorSOoVQq9b6/t956C6WlpRgzZgxefPFFpKena+2jXr1Hc+WEELCxsanyfRrixx9/hIuL\nC9q3bw8AmDx5MlasWIG7d++iWbNmBu3DmrHmTF9zLVu2RIcOHdC7d28AwMsvv4ywsDBcu3atykuI\ndQnrzjrrziJnO1fXkCFDsH//fmRmZgIAtm3bhqlTpwIAhg0bhu3btwMAbt68idOnT0teP3jwYBw+\nfBj5+fkAgJiYGGzcuBE2Njaa+6TOzs5o0KCBpiDT09Ph4eGBlJQUDBs2DIcOHUJubi7Kysr0Tm4A\nHjbC2bNnw93dHQqFAufOnUNpaanm7/fs2QMAuHDhAq5cuYLevXtX+T4N0b17d5w5cwbZ2dkAgISE\nBLRt25aN1wisOcMMHz4cN27c0Mw0PXPmDBQKBe/7Gol1Z5jaUHcWe+ZbHcOGDcP06dPh7+8PhUIB\nlUqFjz/+GAqFAuHh4Vi0aBHGjBmD1q1b65w9OGLECPz999+ayx+dOnXCsmXL0KhRI3Tv3h1jxozB\ntm3bEBsbi+XLl+Ozzz5DSUkJ5s2bhxdeeAHAw0u648aNg729Pbp164a7d+9WmfObb76J2bNno2nT\npmjUqBH69eun9bWfa9euwdvbGwqFAh9++CEcHByqfJ+VeXwSwqBBgxAQEAA/Pz/Y2tqiadOmiI2N\nNeZHXuex5gyrOUdHR6xduxbvvfceioqKUL9+fcTExHCin5FYd5ZTdwoh+EhBIiIiOdWJy85ERES1\nCZsvERGRzNh8iYiIZFYnJlwBQEREBM6cOQMAuHTpEp555hk0bNgQALB9+3bNn03l+vXr8PT0xNmz\nZ6vcbtSoUVizZg2ee+45g/cdExODu3fvYvHixVVuFxcXh3Xr1qFFixYAADs7O2zdutXg49CTqYs1\nV+78+fOYNGkSEhMTOVteZnWx7izxd12dab5hYWGaP48aNQqrVq2qVhFYorNnzyIkJASenp41nUqd\nVBdrDgDu3LmDJUuWQK1W13QqdVJdrDtL/F3Hy87/X8+ePTFv3jy4ubkhOTlZ8jSNx8dHjx7Fa6+9\nBm9vb0ycOFHvJ77s7GzMmjULEyZMwKhRo+Dn56e1rurWrVvh4+ODV199VetpJdU9TkZGBry8vJCR\nkQHgYUHu27cP3t7eCAgIQGpqarV/LmQ+1lhzZWVlCA4OxptvvlntnwfJwxrrzhJ/19WZM1991Go1\nRo4ciTVr1lS5XVpaGlavXo1Nmzbhqaeewl9//YVp06bhu+++01pp5nH79+9Hnz59EBgYCCEEAgMD\nER8fD39/fwBAgwYNsGfPHmRkZMDb2xu9e/eGra1tpcepTKtWrTRfai8sLESHDh0wY8YMuLi44MCB\nA5g+fToOHjwIOzs7I39KZErWVnPAw0X1e/XqhWHDhhnxEyE5WFvdWervOjbfx/Tt21fvNklJScjM\nzMQbb7yhiSkUCly9erXSx3tNnToVP//8Mz7//HOkpaXhr7/+0ixrBkCzPmmrVq0wdOhQnDx5Ekql\nstLjGKJx48Zaz7R0d3fHJ598guTkZAwcONCgfZD5WVPNHTt2DOfPnzfoWapUs6yp7iz1dx2b72Mq\n+zRXXFys+XNZWRkGDRqE6OhoTSw9PR0tW7asdL9RUVE4f/48xo0bhwEDBqCkpETv2qWlpaWVHufw\n4cN638uNGzdw9OhRrQdZl++bag9rqrndu3fj1q1b8PHx0cSmTp2KyMhIq7/naGmsqe4s9Xcd7/lW\nolmzZpqHRz9eAAMHDkRSUhIuXboEADh+/DjGjh2LBw8eVLqvH3/8EVOnToW3tzeaN2+OEydO6Fy7\n9ObNmzhx4gQGDRpk1HEe16hRI0RHR+P8+fOa1xcVFaFXr17V+CmQnCy95mJiYnDw4EHEx8drLgl+\n8cUXbLy1nKXXnaX+rqvdHw1qUFhYGJYuXQp7e3sMHjwYjo6OAIDOnTtj6dKleOuttzSfrj755JNK\nP0kCwOzZs/H+++8jNjYWSqUSLi4uWpdUHjx4AB8fH6jVaoSFhcHZ2RkAqn2cjIwMBAYGYv369WjV\nqhWio6OxePFiqNVqqFQqrF27lo9pq8WsoebI8lhD3Vni7zqu7UxERCQzXnYmIiKSGZsvERGRzNh8\niYiIZGb2CVf3799HSkoKHB0doVQqzX04quVKS0uRlZWFnj17mnyN2cex7qicXDUHsO7oEX11Z/bm\nm5KSgsmTJ5v7MGRhtmzZYtAX/Y3FuqOKzF1zAOuOpCqrO7M33/Jp61u2bEHr1q3NfTiq5W7duoXJ\nkydr6sJcWHdUTq6aA1h39Ii+ujOq+ZaVlWHJkiVITU1F/fr1ERERAScnJ53bll96ad26Ndq2bWvM\n4cgKmfuSHOuOKpLjMjDrjiqqrO6MmnCVkJCA4uJibN++HfPnz8fKlSufKDkiIqK6xKjm+8svv2ie\nWtKnTx+kpKSYNCkiIiJrZlTzzc/Ph0ql0oyVSiVKSkpMlhQREZE1M6r5qlQqFBQUaMZlZWW1/gkS\nREREtYVRzdfFxQWJiYkAgN9++w1dunQxaVJERETWzKjTVVdXVyQlJWHixIkQQiAyMtLUeREREVkt\no5pvvXr1sHTpUlPnQkREVCfwRi2RBQsPD9ca6/pQ/MMPP0hiQ4cONVtORKQfH6xAREQkMzZfIiIi\nmbH5EhERyYzNl4iISGaccEVkwb7//nutcb160s/Tx44dk8Q44YqoZvHMl4iISGZsvkRERDJj8yUi\nIpIZmy8REZHMOOGKyEIUFxdLYoWFhXpf5+PjY450iJ7YtWvXJLHnn39eEmvevLkklpSUpDW+c+eO\nZJva/NAfnvkSERHJjM2XiIhIZmy+REREMmPzJSIikpnRE658fHygUqkAAG3btsWKFStMllRdlpeX\nJ4l9+OGHktiSJUsksePHj2uNhw8fbrK8qOb9/vvvktjZs2f1vs7Jyckc6RBV6b///a8k9tFHH2mN\nN2/eLNkmJyfHoNhzzz2nNfb19ZVso+t3Z21hVPN98OABhBCIi4szdT5ERERWz6jLzn/88QeKiorg\n7++PKVOm4LfffjN1XkRERFbLqDPfhg0bIiAgAK+99hrS0tIwffp0HDp0CDY2/NowERGRPkZ1S2dn\nZzg5OUGhUMDZ2RkODg7IyspCmzZtTJ2fxbh9+7bWeP/+/Qa9bvXq1VrjixcvSrYpKSmRxHQ9vebl\nl1/WGjs7O0u2SU1NNSgvskwTJkyQxBo2bFgDmVBdkp6eLokNGTJEEsvKytIaCyEk2ygUCklM12IZ\nf/75p9b40KFDkm1q8z1foy4779q1CytXrgQAZGRkID8/H46OjiZNjIiIyFoZdeY7fvx4LFq0CL6+\nvlAoFIiMjOQlZyIiIgMZ1THr16+PDz74wNS5EBER1QlcZIOIiEhmdfpa8c2bN7XGarVass2pU6ck\nscWLF0tiN27c0BoXFRU9YXbVV1paqjW+dOmSZJtvv/1WEnNzczNbTiSvdu3aSWK8JUSmVvF3ZXR0\ntGSbipOrAOnTicoXanpcbGysJNaqVStJzN3dXWtccfJqbcczXyIiIpmx+RIREcmMzZeIiEhmbL5E\nREQyqzMzMTp16iSJZWdna40rTlgCgMLCQrPlZG66Vo8pLi6ugUyIyJoEBwdrjT/++GODXlfxqUa6\nVmTTJT8/XxJLTk7WGleczFXb8cyXiIhIZmy+REREMmPzJSIiklmduedb8alDAJCXl1cDmcjHzs5O\nEtP1dBAiourYu3ev1ljX/JJnn31WEjP0Hm9Fuhbj0BWzJDzzJSIikhmbLxERkczYfImIiGTG5ktE\nRCQzq5xwpeupQ5Y8uapZs2aS2Pz58yWxik+0+e9//yvZpmvXrqZLjGT1/vvv13QKRAAAhUJR5RgA\nRowYIVc6FsmgM99z587Bz88PAHDlyhX4+vpi0qRJCA8PR1lZmVkTJCIisjZ6m++GDRsQFhaGBw8e\nAABWrFiBoKAgbN26FUIIHDlyxOxJEhERWRO9zbd9+/aIiYnRjC9cuID+/fsDAIYPH44TJ06YLzsi\nIiIrpLf5urm5wcbm0a1hIYTm+r6dnZ1F30slIiKqCdWecFWv3qN+XVBQAHt7e5MmZApRUVGSmK4V\nWCyFricr6fq5e3h4aI2bNm1qtpxIfrqe7FKRrol4RDXhzp07NZ1CrVbtrxp1794dp0+fBgAkJiai\nb9++Jk+KiIjImlW7+S5cuBAxMTGYMGEC1Go13NzczJEXERGR1TLosnPbtm2xY8cOAICzszM2b95s\n1qSIiIisGVe4IiIikplVrnDl4uIiiZ06daoGMjGN+/fvS2Jz5syRxD744AOtsa778du3bzddYmQ2\nuhavKS4ulsQqTiR8/JsJRKZQvsbD40pLS/W+7uLFi+ZIx2rwzJeIiEhmbL5EREQyY/MlIiKSmVXe\nIPr2228lsTZt2khiuhavsGRpaWla45s3b0q28fHxkcQ+//xzrbGDg4NJ86Lqu3TpkiT23XffSWIV\nnxzTuHFjk+Wg676zrv9nHl94x9Q5UM3bs2ePJHb9+nW9r7t9+7Yk9ueff2qNu3TpYnxiFo5nvkRE\nRDJj8yUiIpIZmy8REZHM2HyJiIhkZpUTrlQqlSRWFxcf0LUow969eyWxhIQErfH48ePNlhOZVvfu\n3bXGDRs2NNm+dU2u0vWkrHbt2mmNf/31V8k2zZo1M1leJC9dixY1adJEa5ybmyvZJicnRxI7e/as\n1pgTroiIiEg2bL5EREQyY/MlIiKSGZsvERGRzAyahXTu3DmsWrUKcXFxuHjxImbMmIFnn30WAODr\n6wt3d3dz5mgSum7s//zzz2Y7nq4JJn5+fpJYYGCg3n3FxsZKYmvXrjUuMR2io6O1xpxwVffk5eVJ\nYlevXjXotcOGDdMac3KVddH171lxYp+u+tHlX//6l9Z46NChkm2eeeaZamRnufQ23w0bNmDv3r1o\n1KgRAODChQuYNm0a/P39zZ4cERGRNdJ72bl9+/aIiYnRjFNSUnDs2DFMnjwZoaGhyM/PN2uCRERE\n1kZv83Vzc9P6jmyvXr2wYMECbNmyBe3atTPp5U8iIqK6oNorT7i6usLe3l7z52XLlpk8KXNYtWqV\nJPbyyy9rjUtKSkx2PF1PAtF1f8MQUVFRkpgQQhLTdW/YEBUXRfj7778l23Tq1MmofZN5vfjii0a9\nLi4uTmu8YsUKyTapqakG7Wv27NlG5UCWYd26dZJYVlaWUfu6e/eu1njw4MGSbX7//XdJzBqflFXt\n2c4BAQE4f/48AODkyZPo0aOHyZMiIiKyZtU+812yZAmWLVsGW1tbtGjRwmLOfImIiGoLg5pv27Zt\nsWPHDgBAjx498OWXX5o1KSIiImvGRTaIiIhkVmce9VNxIQAAeOqpp7TGxk4iAB7OAn9c586djd5X\nRQ0aNJDE5syZI4kZO+Gq4hfmObnKcmzZskVrXFBQINlG1+S/AwcOaI1LS0sl2ygUCoNyqDiZcdOm\nTZJtrHHCjDW6f/++JBYeHi6JVayN5s2bS7a5ffu23uNdv35dEtNVi9aIZ75EREQyY/MlIiKSGZsv\nERGRzNh8iYiIZFZnJlyZUv/+/SWxhIQErbGdnZ3Jjqdr5a2lS5eabP+tW7c22b7INJo2bSqJtWrV\nShL75ptvqhwbasmSJZLY5MmTJTEPDw9JrOKErilTpki24SQsy7BmzRqjXvfRRx9JYrrqhx7hmS8R\nEZHM2HyJiIhkxuZLREQkMzZfIiIimdXpCVfTp0/XGkdGRhr0up9++kkSmzVrltZY18QFBwcHvfvW\nNbnqww8/lMS2bdumd1+GMuXkLTKNli1bSmL9+vWTxPbt22fU/gcNGqQ1XrBggWSbiiufAYCnp6ck\n9sEHH2iNda2QZmtrW90UqQa89NJLklhZWZkkVq+e9nnb4sWLJdsMHDhQEjt58qTefYeEhEhi1vjc\neJ75EhERyYzNl4iISGZsvkRERDKr0/d8dd1DM9bmzZu1xkqlUrJNxftsgPQeS0REhGSbL7744gmz\ne8Te3l4S0/XEJ6p9dC1kcObMGa1xRkaGZBtdT45p1qyZ1ljXk7N0GTJkiCRW8Z7vpEmTJNvwnq9l\n6Nu3ryRW8f4uIH2q0aVLlyTb6IpVfJ0h+7ZWVTZftVqN0NBQ3LhxA8XFxZg5cyY6deqEkJAQKBQK\ndO7cGeHh4Tp/gERERKRblc137969cHBwQFRUFHJycuDt7Y1u3bohKCgIAwYMwOLFi3HkyBG4urrK\nlS8REZHFq/KUdfTo0Zg3bx4AQAgBpVKJCxcuaNY2Hj58OE6cOGH+LImIiKxIlc3Xzs4OKpUK+fn5\nmDt3LoKCgiCE0FyTt7OzQ15eniyJEhERWQu9E67S09Mxe/ZsTJo0CZ6enoiKitL8XUFBgc4JPJai\n4tOJ2rVrJ9nm2rVrRu1b1yQpU06cMoSue/HlVzIep+tpOVT7ODk5SWKnT5/WGut6GlKTJk0kMWMn\ntbzyyiuS2PPPP6811vVUo8TEREnMkn93ED2pKs98s7Oz4e/vj+DgYIwfPx4A0L17d83/8ImJiTpn\nxxEREVHlqmy+69atQ25uLmJjY+Hn5wc/Pz8EBQUhJiYGEyZMgFqthpubm1y5EhERWYUqLzuHhYUh\nLCxMEq/4nVYiIiIyHL+gS0REJLM6vcJV69attcanTp2SbDNgwABJTNeKQbVBxUk077zzjmSbJUuW\nyJQNyUHXJEFzatSokST2888/y5oDyaviamiAdBKfrpXVHjx4oHffuiYIzpgxoxrZWS6e+RIREcmM\nzZeIiEhmbL5EREQyq9P3fCuqeA8YAH766SdJrOLiHID894FtbKT/dKGhoVrj8PBwudIhIiv166+/\nSmIVF+bR9XtS15yTpKQkrbGueTZdunSpbooWiWe+REREMmPzJSIikhmbLxERkczYfImIiGTGCVd6\n6Hrij67JBStWrNAax8TEmCyHuXPnSmKLFi2SxFq2bGmyYxIRAYYt5DJ06FBJ7Pjx4+ZIx2rwzJeI\niEhmbL5EREQyY/MlIiKSGZsvERGRzKqccKVWqxEaGoobN26guLgYM2fORJs2bTBjxgw8++yzAABf\nX1+4u7vLkWutoWsSVnR0dJVjIiKiclU2371798LBwQFRUVHIycmBt7c3Zs+ejWnTpsHf31+uHImI\niKxKlc139OjRcHNzAwAIIaBUKpGSkoLLly/jyJEjcHJyQmhoKFQqlSzJEhERWYMq7/na2dlBpVIh\nPz8fc+fORVBQEHr16oUFCxZgy5YtaNeuHdauXStXrkRERFZB74Sr9PR0TJkyBV5eXvD09ISrqyt6\n9uwJAHB1dcXFixfNniQREZE1qbL5Zmdnw9/fH8HBwRg/fjwAICAgAOfPnwcAnDx5Ej169DB/lkRE\nRFakynu+69atQ25uLmJjYxEbGwsACAkJQWRkJGxtbdGiRQssW7ZMlkSJiIisRZXNNywsDGFhYZL4\nl19+abaEiIiIrB0X2SAiIpIZmy8REZHM2HyJiIhkxuZLREQkMzZfIiIimbH5EhERyYzNl4iISGZs\nvkRERDKrcpENUygtLQUA3Lp1y9yHIgtQXgfldWEurDsqJ1fNPX4M1h3pqzuzN9+srCwAwOTJk819\nKLIgWVlZcHJyMuv+AdYdPWLumis/BsC6o0cqqzuFEEKY88D3799HSkoKHB0doVQqzXkosgClpaXI\nyspCz5490bBhQ7Mdh3VH5eSqOYB1R4/orTthga5duya6desmxo4dq/nP09NT7Ny584n3HRgYKHbv\n3i2EEGLs2LHi3r17lW6bm5sr/Pz8qn2MgwcPitdff10SP3XqlHj11Vf1vr5Lly7i9u3b1TrmwoUL\nxWeffaZ3uz/++EO8/vrrwsvLS/j4+Ijk5ORqHcdasebMV3Pfffed8PDwEGPHjhWvv/66uHLlSrWO\nY81Yd+aru3KHDx8Wzz//fLWOYQpmv+xsLg0bNkR8fLxmnJGRAQ8PD/Ts2RPdunUzyTEe378u9+7d\nQ3JyskmOVRsUFRUhICAAy5cvx4gRI5CQkIC3334bhw4dqunUagXWnOndv38fwcHBiI+Ph5OTEzZu\n3IiIiAisX7++plOrNVh35pOWloZ///vfEOa9AKyTxTbfilq1agUnJyekpaXh4sWL2LVrF4qKiqBS\nqRAXF4edO3di27ZtKCsrg4ODA95991107NgRGRkZCAkJQWZmJp5++mncvn1bs8+uXbvi5MmTaNas\nGT799FPs2bMHNjY2cHJywsqVK7Fo0SLcv38fXl5e+Oqrr5CWlobly5cjJycHpaWl8PPz0zwHec2a\nNfjmm2/g4OBg0H2ny5cvY+nSpSgsLERmZia6deuG6OhoNGjQAAAQHR2N5ORklJWVISgoCCNHjgSA\nSt9nZZKTkxEWFob4+HgkJSWhXbt2GDFiBADgpZdeQtu2bY3+N7F2rLknr7nS0lIIIZCXlwcAKCgo\n0ByPdGPdPXndAQ9PNoKDgxESEoK3337b6H8Po8l+rm0C165dE3369NGK/frrr6Jfv37i5s2bYvfu\n3aJfv34iLy9PCCHE6dOnxaRJk0RhYaEQQogffvhBjBkzRgghxKxZs8Tq1auFEEKkpaWJPn36aC7F\nlF/ySEhIEK+88orIyckRQggRGRkpYmNjtfJQq9XC3d1dpKSkCCEeXqYZM2aMOHv2rDh8+LBwd3cX\neXl5Qq1Wi8DAQL2XYlauXCm+/vprIYQQxcXFwsPDQxw6dEiT16effiqEECI1NVX0799f3L59u8r3\nacilmPXr14s5c+aIRYsWCR8fHzF16lTN+6nrWHPmqTkhhNizZ4/o0aOHGDJkiBg0aJBIS0vT+5q6\ngnVnvrp7++23xc6dO3X+jOVgsWe+5Z/CgIc3tp966ilERUWhTZs2AB5+klOpVACAY8eO4cqVK5g4\ncaLm9ffu3UNOTg5OnDiBhQsXAgCcnJwwYMAAybFOnjyJ0aNHo2nTpgCARYsWAQCuX7+u2SYtLQ1X\nr15FaGioVo4XL17EpUuX4Orqqsln3LhxiIuLq/L9BQcHIykpCRs2bEBaWhoyMzNRWFio+XtfX18A\nQJcuXdCxY0ecPXsWv/zyS6Xv0xAlJSU4fvw4Nm3ahN69eyMhIQGBgYH4/vvvUb9+fYP2Yc1Yc6av\nudTUVKxduxYHDhxA+/btsWnTJsyZMwfx8fFQKBQG7cPase5MX3dbtmyBjY0Nxo8fr/Xe5GSxzbfi\nfZCKGjcl9LEaAAAY9klEQVRurPlzWVkZvLy8EBwcrBlnZmaiadOmUCgUWtf7bWykPxKlUqn1iyA3\nNxe5ubla25SWlsLe3l4rp+zsbDRp0gRRUVFaxzBkFuRbb72F0tJSjBkzBi+++CLS09O19lGv3qP1\nUYQQsLGxqfJ9GqJly5bo0KEDevfuDQB4+eWXERYWhmvXrlV5OaeuYM2ZvuZ+/PFHuLi4oH379gAe\nfkVnxYoVuHv3Lpo1a2bQPqwd6870dbdnzx7Nhxq1Wq358/r169GqVSuD9vGk6sQKV0OGDMH+/fuR\nmZkJANi2bRumTp0KABg2bBi2b98OALh58yZOnz4tef3gwYNx+PBh5OfnAwBiYmKwceNG2NjYaO5Z\nOTs7o0GDBpqCTE9Ph4eHB1JSUjBs2DAcOnQIubm5KCsr0zu5AXj4S2n27Nlwd3eHQqHAuXPntL6s\nvWfPHgDAhQsXcOXKFfTu3bvK92mI4cOH48aNG0hJSQEAnDlzBgqFgvd9jcCaM0z37t1x5swZZGdn\nAwASEhLQtm1bNl4jse4Ms2vXLuzbtw/x8fFYv3695gOOXI0XsOAz3+oYNmwYpk+fDn9/fygUCqhU\nKnz88cdQKBQIDw/HokWLMGbMGLRu3Vrn7MERI0bg77//1lz+6NSpE5YtW4ZGjRqhe/fuGDNmDLZt\n24bY2FgsX74cn332GUpKSjBv3jy88MILAB5eXhs3bhzs7e3RrVs33L17t8qc33zzTcyePRtNmzZF\no0aN0K9fP1y9elXz99euXYO3tzcUCgU+/PBDODg4VPk+K/P4JARHR0esXbsW7733HoqKilC/fn3E\nxMRwAowRWHOG1dygQYMQEBAAPz8/2NraomnTpoiNjTXmR05g3Rlad7WB2RfZICIiIm114rIzERFR\nbcLmS0REJDM2XyIiIpnViQlXABAREYEzZ84AAC5duoRnnnlGs9j19u3bTb7g+vXr1+Hp6YmzZ89W\nud2oUaOwZs0aPPfccwbvOyYmBnfv3sXixYur3C4uLg7r1q1DixYtAAB2dnbYunWrwcehJ1MXa67c\nmjVrcO/ePYO3J9Opi3V34MABrF27FkqlEq1bt0Z4eDieeeYZg49TE+pM8w0LC9P8edSoUVi1alW1\nisASnT17FiEhIfD09KzpVOqkulhzt27dQmRkJI4fP45x48bVdDp1Ul2ru7S0NISHh2Pz5s3o2rUr\nzpw5g7lz52L37t01nVqVeNn5/+vZsyfmzZsHNzc3JCcno2vXrrhz547m7x8fHz16FK+99hq8vb0x\nceJEvZ/4srOzMWvWLEyYMAGjRo2Cn5+f1rqqW7duhY+PD1599VXs2rVLE6/ucTIyMuDl5YWMjAwA\nD5vvvn374O3tjYCAAKSmplb750LmY401t2vXLrzwwguYNm1atX8eJA9rq7s//vgD3bp1Q9euXQEA\n/fr1w40bN2ps5SpD1ZkzX33UajVGjhyJNWvWVLldWloaVq9ejU2bNuGpp57CX3/9hWnTpuG7777T\nWmnmcfv370efPn0QGBgIIQQCAwMRHx8Pf39/AECDBg2wZ88eZGRkwNvbG71794atrW2lx6lMq1at\nNN9hKywsRIcOHTBjxgy4uLjgwIEDmD59Og4ePAg7Ozsjf0pkStZWcwDwr3/9C8DDy4VUO1lb3XXv\n3h1//vknfv/9d/zjH//A0aNHkZOTg6ysrFq9QBCb72P69u2rd5ukpCRkZmbijTfe0MQUCgWuXr1a\n6eO9pk6dip9//hmff/450tLS8Ndff2mWcASgWZ+0VatWGDp0KE6ePAmlUlnpcQzRuHFj/Oc//9GM\n3d3d8cknnyA5ORkDBw40aB9kftZUc2Q5rKnu2rdvj8jISISHh6O4uBgvvfQSunXrBltbW4NeX1PY\nfB9T2ae54uJizZ/LysowaNAgREdHa2Lp6elo2bJlpfuNiorC+fPnMW7cOAwYMAAlJSV61y4tLS2t\n9DiHDx/W+15u3LiBo0ePws/PT7Jvqj2sqebIclhT3RUXF8PJyQk7duwA8PABMV988UWtPusFeM+3\nUs2aNdM8PPrxAhg4cCCSkpJw6dIlAMDx48cxduxYPHjwoNJ9/fjjj5g6dSq8vb3RvHlznDhxQufa\npTdv3sSJEycwaNAgo47zuEaNGiE6Ohrnz5/XvL6oqAi9evWqxk+B5GTpNUeWydLrrri4GL6+vkhP\nTwcAbNy4ES+88AIcHByq8VOQH0+DKhEWFoalS5fC3t4egwcPhqOjIwCgc+fOWLp0Kd566y3NJ7dP\nPvmk0k+SADB79my8//77iI2NhVKphIuLi9YllQcPHsDHxwdqtRphYWFwdnYGgGofJyMjA4GBgZon\nc0RHR2Px4sVQq9VQqVRYu3YtHw1Yi1lDzZHlsYa6W7ZsGaZPn47S0lJ07NgRK1asMNFPx3y4tjMR\nEZHMeNmZiIhIZmy+REREMmPzJSIikpnZJ1zdv38fKSkpcHR0hFKpNPfhqJYrLS1FVlYWevbsafI1\nZh/HuqNyctUcwLqjR/TVndmbb0pKCiZPnmzuw5CF2bJli0Ff9DcW644qMnfNAaw7kqqs7szefMun\nrW/ZsgWtW7c29+Golrt16xYmT56sqQtzYd1ROblqDmDd0SP66s6o5ltWVoYlS5YgNTUV9evXR0RE\nBJycnHRuW37ppXXr1rV+xRGSj7kvybHuqCI5LgOz7qiiyurOqAlXCQkJKC4uxvbt2zF//nysXLny\niZIjIiKqS4xqvr/88guGDRsGAOjTpw9SUlJMmhQREZE1M6r55ufnQ6VSacZKpRIlJSUmS4qIiMia\nGdV8VSoVCgoKNOOysjI+LYeIiMhARjVfFxcXJCYmAgB+++03dOnSxaRJERERWTOjTlddXV2RlJSE\niRMnQgiByMhIU+dFRERktYxqvvXq1cPSpUtNnQsREVGdwLWdiYiIZMbmS0REJDM2XyIiIpmx+RIR\nEcmMzZeIiEhmbL5EREQyY/MlIiKSGZsvERGRzNh8iYiIZMbmS0REJDM+ishCvffee5LYCy+8oDX2\n8PCQKx2qJRQKhVGvE0KYOBMiqgrPfImIiGTG5ktERCQzNl8iIiKZsfkSERHJzOgJVz4+PlCpVACA\ntm3bYsWKFSZLyhoUFhZKYra2tpLY33//rTV2cnKSbNO4cWNJbPXq1ZJYgwYNtMYXLlyQbNOiRQtp\nslTrGDtxiogsg1HN98GDBxBCIC4uztT5EBERWT2jLjv/8ccfKCoqgr+/P6ZMmYLffvvN1HkRERFZ\nLaPOfBs2bIiAgAC89tprSEtLw/Tp03Ho0CHY2PBrw0RERPoY1S2dnZ3h5OQEhUIBZ2dnODg4ICsr\nC23atDF1fjXu559/lsS+/vprSezIkSNa4zt37ki20XXv9vz581rjTp06SbZp1qyZJFZQUCCJ5efn\nVzkGeM+XdNN1j5kLb5CpJSYmao2nT58u2eavv/6SxPbs2SOJeXl5mS6xGmDUZeddu3Zh5cqVAICM\njAzk5+fD0dHRpIkRERFZK6POfMePH49FixbB19cXCoUCkZGRvORMRERkIKM6Zv369fHBBx+YOhci\nIqI6gYtsEBERyaxOXyuuOKFk06ZNkm10TQgoKysz6ngtW7aUxJRKpdb4+vXrkm0uXbpk1PGIiGrK\n6dOnJbE33nhDa3zlyhXJNrom/82fP18Sq5MTroiIiMh4bL5EREQyY/MlIiKSGZsvERGRzOr0hKuK\nTx4KCAgw6HXTpk2TxKZOnar3df3795fEUlJStMZNmjSRbJORkSGJubm5SWKdO3fWGrdq1UpvTkSV\n4apXVJmSkhKt8dtvvy3ZZv369ZLYgwcPjDpecXGxUa+rzXjmS0REJDM2XyIiIpmx+RIREcmsTt/z\nNVavXr0ksaFDhxq1LxcXF73bVLyXCwC///67JFa/fn2tcaNGjYzKiYionFqtlsRGjhypNT5x4oRk\nG11zBoz1/vvvm2xftQXPfImIiGTG5ktERCQzNl8iIiKZsfkSERHJjBOuHmNrayuJ6Zps0K9fPznS\nqVL79u1rOgUisjK6nqo2atQoScyQJ63NnTtXEqv4e0vX4hyNGzeWxF5++WW9x7M0Bp35njt3Dn5+\nfgAePgLK19cXkyZNQnh4uNGP1yMiIqqr9DbfDRs2ICwsTLMs2IoVKxAUFIStW7dCCIEjR46YPUki\nIiJrorf5tm/fHjExMZrxhQsXNGsUDx8+XOf3u4iIiKhyepuvm5sbbGwe3RoWQmi+PG1nZ4e8vDzz\nZUdERGSFqj3hql69R/26oKAA9vb2Jk1ITo9/qACATp06SbbRtZLUmTNnJLGBAweaLjGiWoBPMLIu\nFf89dU2uGjRokCSWnp6ud9/vvPOOJLZkyRJJ7MCBA3r39fzzz0tiLVq00Ps6S1Ptrxp1794dp0+f\nBgAkJiaib9++Jk+KiIjImlW7+S5cuBAxMTGYMGEC1Gq1zufKEhERUeUMuuzctm1b7NixAwDg7OyM\nzZs3mzUpIiIia8YVroiIiGRWp1e4Kikp0Rrrmlyli7e3tznSIaoxnFxl/SpOsHr22WcNep2dnZ0k\ntnPnTq0xbz9WH898iYiIZMbmS0REJDM2XyIiIpnV6Xu+xmrevLnJ9nX//n2t8UcffSTZZuPGjZKY\nrnssoaGhWmNHR8cnS45qjK57sOUry5mDofvmvWHLYOgCGoaYP3++JGbIPV5dT4Tbt2+f3tfl5uZK\nYleuXJHEnJyc9O6rNuOZLxERkczYfImIiGTG5ktERCQzNl8iIiKZccLVYwydTGLspJN169ZJYjNn\nztQaP/7UqKqkpqZKYra2tlrjyMhIyTYVn+REtZM5J1eR9XN1dZXEDHk6kS4NGjSQxHx8fLTGly5d\nkmyja8LVn3/+qfd4ycnJkpiup8YZ+35qC575EhERyYzNl4iISGZsvkRERDJj8yUiIpKZQbNvzp07\nh1WrViEuLg4XL17EjBkzNE/E8PX1hbu7uzlzNJuKEwl0Pa0oPj5eEtM1Gaa4uFhrfPToUck2CxYs\nkMQqTrAaMmSIZJunn35aEjt37pwk9uGHH2qN27VrJ9lmzpw5khgRWRddE64MmeykyzvvvKN3G7lX\nZLMGepvvhg0bsHfvXjRq1AgAcOHCBUybNg3+/v5mT46IiMga6b3s3L59e8TExGjGKSkpOHbsGCZP\nnozQ0FDk5+ebNUEiIiJro7f5urm5aX03tFevXliwYAG2bNmCdu3aYe3atWZNkIiIyNpUe8UFV1dX\n2Nvba/68bNkykycll4pfAk9MTDTodaWlpZLY3Llztca6nkSky6pVq7TGs2bNkmyj60vumZmZkljF\ne8NLly6VbPPPf/5TEiu/pUCkj677eHzSUe3z73//WxKr+BSgnTt3Srb56aefjDqesTUwZcoUSczQ\n352WrtqznQMCAnD+/HkAwMmTJ9GjRw+TJ0VERGTNqn3mu2TJEixbtgy2trZo0aKFRZ/5EhER1QSD\nmm/btm2xY8cOAECPHj3w5ZdfmjUpIiIia8ZFNoiIiGRWpx9xU1ZWpjW+e/euQa+bNGmSJHbw4EGt\nsa5JUhEREZJYxYlaSqXSoBwcHR0lscDAQK3x+vXrJdt8/vnnkpiuSV5EZLl0TaKcP3++1rji7x4A\nKCoqMlkOH330kSQWHh6uNe7fv7/JjmdpeOZLREQkMzZfIiIimbH5EhERyYzNl4iISGZ1esJVxUlR\nY8aMkWxTcSJVZbEOHTpojffv3y/ZpnPnztVNsVK6Vhp6fBnQyuhanYtqHz4lhszN1tbWoJixGjdu\nrHcbFxcXkx3P0vDMl4iISGZsvkRERDJj8yUiIpIZmy8REZHM6vSEq4oTlJydnQ163ZAhQySxiutd\nt2nTxvjEzOi5556r6RSIyMoUFhZKYhUflwoADRs21Bp3797dbDnVdjzzJSIikhmbLxERkczYfImI\niGRWp+/5zps3T2v8f//3fwa9TtfTiVq0aGGSnAyl6x7Lr7/+qjXu27evZJvBgwebLSciqpt27twp\niWVkZEhi3t7eWmN7e3uz5VTbVdl81Wo1QkNDcePGDRQXF2PmzJno1KkTQkJCoFAo0LlzZ4SHh6Ne\nPZ5AExERGarK5rt37144ODggKioKOTk58Pb2Rrdu3RAUFIQBAwZg8eLFOHLkCFxdXeXKl4iIyOJV\neco6evRozaVZIQSUSiUuXLigeQDy8OHDceLECfNnSUREZEWqbL52dnZQqVTIz8/H3LlzERQUBCGE\nZoF3Ozs75OXlyZIoERGRtdA74So9PR2zZ8/GpEmT4OnpiaioKM3fFRQUWPQN84pfAm/evLlkmx9/\n/FESGzlypCRWccJVZGSkZBtdT02qSNckhe3bt0ti0dHRkphardYah4eHS7apX7++3hyoduKTjqi2\nSk5ONmi7nj17mjkTy1HlmW92djb8/f0RHByM8ePHA3i4Isnp06cBAImJiTpn1BIREVHlqmy+69at\nQ25uLmJjY+Hn5wc/Pz8EBQUhJiYGEyZMgFqthpubm1y5EhERWYUqLzuHhYUhLCxMEt+8ebPZEiIi\nIrJ2/IIuERGRzOr0Cle2trZa48WLF0u2KSgokMR27dolie3bt09rHBgYaFAOFSfRPMkEmjVr1miN\n//nPfxq9L7IMpqwfImN99dVXBm3XoUMHM2diOXjmS0REJDM2XyIiIpmx+RIREcmsTt/zNYSdnZ0k\nNnXqVEns9ddf1xrrWvlr69atkticOXP05jB37lxJ7J133pHEmjRpojXmghp1j7kX4tC1f6p7Ll++\nrDVOS0uTbDNgwABJrHy9COKZLxERkezYfImIiGTG5ktERCQzNl8iIiKZccKViSiVSq2xg4ODZJtZ\ns2YZFCMyJU6SIlNbvny53m10LfKjawJrXcUzXyIiIpmx+RIREcmMzZeIiEhmbL5EREQyq3LClVqt\nRmhoKG7cuIHi4mLMnDkTbdq0wYwZM/Dss88CAHx9feHu7i5HrkREVAtUnEyVlZUl2eb555+XKx2L\nVGXz3bt3LxwcHBAVFYWcnBx4e3tj9uzZmDZtGvz9/eXKkYiIyKpU2XxHjx4NNzc3AA+/rqBUKpGS\nkoLLly/jyJEjcHJyQmhoKFQqlSzJEhERWYMq7/na2dlBpVIhPz8fc+fORVBQEHr16oUFCxZgy5Yt\naNeuHdauXStXrkRERFZB74Sr9PR0TJkyBV5eXvD09ISrqyt69uwJAHB1dcXFixfNniQREZE1qfKy\nc3Z2Nvz9/bF48WIMGjQIABAQEIB3330XvXr1wsmTJ9GjRw9ZEiUiotph4MCBWuP4+PgaysRyVdl8\n161bh9zcXMTGxiI2NhYAEBISgsjISNja2qJFixZYtmyZLIkSERFZiyqbb1hYGMLCwiTxL7/80mwJ\nERERWTsuskFERCQzNl8iIiKZsfkSERHJjM2XiIhIZmy+REREMmPzJSIikhmbLxERkczYfImIiGRW\n5SIbplBaWgoAuHXrlrkPRRagvA7K68JcWHdUTq6ae/wYrDvSV3dmb77lD1mePHmyuQ9FFiQrKwtO\nTk5m3T/AuqNHzF1z5ccAWHf0SGV1pxBCCHMe+P79+0hJSYGjoyOUSqU5D0UWoLS0FFlZWejZsyca\nNmxotuOw7qicXDUHsO7oEX11Z/bmS0RERNo44YqIiEhmbL5EREQyY/MlIiKSGZsvERGRzNh8iYiI\nZGb27/kCQFlZGZYsWYLU1FTUr18fERERZv++nSmcO3cOq1atQlxcHK5cuYKQkBAoFAp07twZ4eHh\nqFev9n12UavVCA0NxY0bN1BcXIyZM2eiU6dOFpG7qbHu5MO6e4g1Jx+Lrzkhg2+//VYsXLhQCCHE\n2bNnxf/+7//Kcdgnsn79euHh4SFee+01IYQQM2bMEKdOnRJCCPHuu++K7777ribTq9SuXbtERESE\nEEKIu3fvihEjRlhM7qbGupMP6+4h1px8LL3mZPlI8Msvv2DYsGEAgD59+iAlJUWOwz6R9u3bIyYm\nRjO+cOEC+vfvDwAYPnw4Tpw4UVOpVWn06NGYN28eAEAIAaVSaTG5mxrrTj6su4dYc/Kx9JqTpfnm\n5+dDpVJpxkqlEiUlJXIc2mhubm6wsXl0VV4IAYVCAQCws7NDXl5eTaVWJTs7O6hUKuTn52Pu3LkI\nCgqymNxNjXUnH9bdQ6w5+Vh6zcnSfFUqFQoKCjTjsrIyrX9sS/D4fYOCggLY29vXYDZVS09Px5Qp\nU+Dl5QVPT0+Lyt2UWHfyYt2x5uRmyTUnS/N1cXFBYmIiAOC3335Dly5d5DisSXXv3h2nT58GACQm\nJqJv3741nJFu2dnZ8Pf3R3BwMMaPHw/AcnI3NdadfFh3D7Hm5GPpNSfL2s7lMwD//PNPCCEQGRmJ\njh07mvuwT+z69et46623sGPHDly+fBnvvvsu1Go1OnTogIiIiFq5cHpERAQOHjyIDh06aGLvvPMO\nIiIian3upsa6kw/r7iHWnHwsveb4YAUiIiKZ1dIvQBEREVkvNl8iIiKZsfkSERHJjM2XiIhIZmy+\nREREMmPzJSIikhmbLxERkcz+H7lzvS1WGPo+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x261ad5dd7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display some error results \n",
    "\n",
    "# Errors are difference between predicted labels and true labels\n",
    "errors = (Y_pred_classes - Y_true != 0)\n",
    "\n",
    "Y_pred_classes_errors = Y_pred_classes[errors]\n",
    "Y_pred_errors = Y_pred[errors]\n",
    "Y_true_errors = Y_true[errors]\n",
    "X_val_errors = X_val[errors]\n",
    "\n",
    "def display_errors(errors_index,img_errors,pred_errors, obs_errors):\n",
    "    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n",
    "    n = 0\n",
    "    nrows = 2\n",
    "    ncols = 3\n",
    "    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n",
    "    for row in range(nrows):\n",
    "        for col in range(ncols):\n",
    "            error = errors_index[n]\n",
    "            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n",
    "            ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n",
    "            n += 1\n",
    "\n",
    "# Probabilities of the wrong predicted numbers\n",
    "Y_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n",
    "\n",
    "# Predicted probabilities of the true values in the error set\n",
    "true_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n",
    "\n",
    "# Difference between the probability of the predicted label and the true label\n",
    "delta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n",
    "\n",
    "# Sorted list of the delta prob errors\n",
    "sorted_dela_errors = np.argsort(delta_pred_true_errors)\n",
    "\n",
    "# Top 6 errors \n",
    "most_important_errors = sorted_dela_errors[-6:]\n",
    "\n",
    "# Show the top 6 errors\n",
    "display_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)\n",
    "print(len(sorted_dela_errors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d21c4caf-e699-4647-8ef4-e60e868607ae",
    "_execution_state": "idle",
    "_uuid": "afc408bd5545a6a2b2d4e04989890546263cb642"
   },
   "source": [
    "The most important errors are also the most intrigous. \n",
    "\n",
    "For those six case, the model is not ridiculous. Some of these errors can also be made by humans, especially for one the 9 that is very close to a 4. The last 9 is also very misleading, it seems for me that is a 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "_cell_guid": "05ff3b9f-c3bb-4cec-a8c2-2c128e8f15b3",
    "_execution_state": "idle",
    "_uuid": "7f17e7bf0a54a01a52fef2d554780f6bc6580dc6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# predict results\n",
    "results = model.predict(test)\n",
    "\n",
    "# select the indix with the maximum probability\n",
    "results = np.argmax(results,axis = 1)\n",
    "\n",
    "results = pd.Series(results,name=\"Label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "_cell_guid": "b5f1f39f-13b8-439a-8913-0f120e3d47a9",
    "_execution_state": "idle",
    "_uuid": "369dfaab09240f3f12bcff91953ffd315ab84985",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n",
    "\n",
    "submission.to_csv(\"cnn_mnist_datagenswitch.csv\",index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('ModelSave.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "** you found this notebook helpful or you just liked it , some upvotes would be very much appreciated - That will keep me motivated :)**\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}