03C_Keras_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #03-C\n",
    "# Keras API\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Tutorial #02 showed how to implement a Convolutional Neural Network in TensorFlow. We made a few helper-functions for creating the layers in the network. It is essential to have a good high-level API because it makes it much easier to implement complex models, and it lowers the risk of errors.\n",
    "\n",
    "There are several of these builder API's available for TensorFlow: PrettyTensor (Tutorial #03), Layers API (Tutorial #03-B), and several others. But they were never really finished and now they seem to be more or less abandoned by their developers.\n",
    "\n",
    "This tutorial is about the Keras API which is already highly developed with very good documentation - and the development continues. It seems likely that Keras will be the standard API for TensorFlow in the future so it is recommended that you use it instead of the other APIs.\n",
    "\n",
    "The author of Keras has written a [blog-post](https://blog.keras.io/user-experience-design-for-apis.html) on his API design philosophy which you should read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following chart shows roughly how the data flows in the Convolutional Neural Network that is implemented below. See Tutorial #02 for a more detailed description of convolution.\n",
    "\n",
    "There are two convolutional layers, each followed by a down-sampling using max-pooling (not shown in this flowchart). Then there are two fully-connected layers ending in a softmax-classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Flowchart](images/02_network_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras. Note the long import-statements. This might be a bug. Hopefully it will be possible to write shorter and more elegant lines in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import InputLayer, Input\n",
    "from tensorflow.python.keras.layers import Reshape, MaxPooling2D\n",
    "from tensorflow.python.keras.layers import Conv2D, Dense, Flatten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and TensorFlow version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.8-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data/MNIST/train-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/train-labels-idx1-ubyte.gz\n",
      "Extracting data/MNIST/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "data = input_data.read_data_sets('data/MNIST/', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of:\n",
      "- Training-set:\t\t55000\n",
      "- Test-set:\t\t10000\n",
      "- Validation-set:\t5000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(len(data.train.labels)))\n",
    "print(\"- Test-set:\\t\\t{}\".format(len(data.test.labels)))\n",
    "print(\"- Validation-set:\\t{}\".format(len(data.validation.labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data.test.cls = np.argmax(data.test.labels, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We know that MNIST images are 28 pixels in each dimension.\n",
    "img_size = 28\n",
    "\n",
    "# Images are stored in one-dimensional arrays of this length.\n",
    "img_size_flat = img_size * img_size\n",
    "\n",
    "# Tuple with height and width of images used to reshape arrays.\n",
    "# This is used for plotting the images.\n",
    "img_shape = (img_size, img_size)\n",
    "\n",
    "# Tuple with height, width and depth used to reshape arrays.\n",
    "# This is used for reshaping in Keras.\n",
    "img_shape_full = (img_size, img_size, 1)\n",
    "\n",
    "# Number of colour channels for the images: 1 channel for gray-scale.\n",
    "num_channels = 1\n",
    "\n",
    "# Number of classes, one class for each of 10 digits.\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_images(images, cls_true, cls_pred=None):\n",
    "    assert len(images) == len(cls_true) == 9\n",
    "    \n",
    "    # Create figure with 3x3 sub-plots.\n",
    "    fig, axes = plt.subplots(3, 3)\n",
    "    fig.subplots_adjust(hspace=0.3, wspace=0.3)\n",
    "\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Plot image.\n",
    "        ax.imshow(images[i].reshape(img_shape), cmap='binary')\n",
    "\n",
    "        # Show true and predicted classes.\n",
    "        if cls_pred is None:\n",
    "            xlabel = \"True: {0}\".format(cls_true[i])\n",
    "        else:\n",
    "            xlabel = \"True: {0}, Pred: {1}\".format(cls_true[i], cls_pred[i])\n",
    "\n",
    "        # Show the classes as the label on the x-axis.\n",
    "        ax.set_xlabel(xlabel)\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot a few images to see if data is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUMAAAD5CAYAAAC9FVegAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHihJREFUeJzt3XmUFNXZx/HvA0LYVQQFFWdOwAVCFBWDu0aBKCogccG4\nEGM0osEtAaNx1xglKBzRE7YD4QQNigKCUVFAEV8EJIIi4wYiCsRlhLggIsJ9/5i5XdUzPXtXVU/7\n+5zjmequ6qpnvPSdp27dxZxziIj80DVIOgARkVygylBEBFWGIiKAKkMREUCVoYgIoMpQRARQZSgi\nAqgyFBEBVBmKiACwS00ObtOmjSssLIwolNzzwQcfUFxcbEnHESeVcf5TGWdWo8qwsLCQZcuW1T6q\neqZ79+5JhxA7lXH+UxlnpttkERFUGYqIAKoMRUQAVYYiIoAqQxERoIZPk0Vqa8SIEQBs3boVgDfe\neAOAxx9/vNyxgwcPBuCoo44C4MILL4wjRPmBU2YoIoIyQ4nYueeeC8C0adMy7jcr3xd2zJgxAMyd\nOxeAE044AYD99tsvihAlQe+++y4ABx54IAAPPPAAAEOGDIk9FmWGIiIoM5QI+GwQKs4IDzroIABO\nOeUUAN5///3UvlmzZgGwevVqAKZMmQLAjTfemP1gJVHLly8HoEGDkrxsn332SSwWZYYiIigzlCzy\n411nzJhRbl/Xrl2BIOtr06YNAC1atADgu+++Sx3bo0cPAF5//XUAPv/884gilqStWLECCP4dDBgw\nILFYlBmKiBBDZuj7kY0fPx6AvffeO7WvSZMmAJx//vkAtGvXDoBOnTpFHZZE4L///S8AzrnUez4j\nnDNnDgDt27fP+FnfDxHgrbfeStt3+umnZzVOSd7KlSsBGD16NAAXXXRRkuEAygxFRIAYMsOhQ4cC\nJRMsVsT3K2vVqhUAXbp0ycq1O3ToAMCwYcOAH+bcdXE644wzgOApMEDLli0BaN26daWfffTRR1Pb\n4fZDyU/vvPMOAFu2bAHSeyAkRZmhiAiqDEVEgBhukydMmAAE3STCt8BFRUVA0PHyxRdfBGDx4sVA\nMPzqww8/rPD8jRo1AoKuGr4RP3wef7us2+R4FBQUVPvYv/3tb0AwLCvMd7HxPyV/DB8+HChZggBy\n47upzFBEhBgyw5NPPjntZ5gfiuVt3rwZCDJF/9fi1VdfrfD8P/rRj4BgoLcf5gWwadMmADp27Fir\n2CU6Tz31FAC33HILANu2bUvt22uvvQC45557AGjWrFnM0UkUwg9R/Xfaf2+bN2+eREhplBmKiJBj\nw/F23313AE466aS09zNllWU98cQTQJBdAhx88MEADBw4MFshSpb4oXvhjNDz3Sz81F2SHxYsWFDu\nvbZt2yYQSWbKDEVEyLHMsDY+/fRTAK644gogfSiYb4+qqsOvxKd///5AMDzPGzRoUGr7rrvuijUm\niYdf6iHMD4jIBcoMRUTIg8zwoYceAoIMcbfddkvt80+qJHm+/+eiRYuAoK3QtxnddNNNqWP9dE6S\nH1555RUAJk2alHrv0EMPBaBXr16JxJSJMkMREepxZvjyyy8DQV8078knn0xt++mjJHl+0s7i4uK0\n9/30beoLmr/mzZsHpPf08H2M/TR+uUCZoYgIqgxFRIB6fJv89NNPA8Hcdz179gTgqKOOSiwmKc+v\neeKHWHonnngiAHfccUfcIUnM/CQtYWeffXYCkVROmaGICPUwM9y6dSsAzz77LBBM1HD77bcDwZRe\nkpzwanZ33303UH726m7dugHqRpPPPv74YwAWLlwIpE+icuaZZyYSU2WUGYqIUA8zQz8ZqG+DOvXU\nUwE4+uijE4tJ0t13332p7aVLl6bt88Px1FaY//7xj38A8MknnwDBdzVXKTMUEaGeZIZ+IlCAO++8\nE4Bdd90VgJtvvjmRmKRi999/f4X7/PBJtRXmv3Xr1qW99lP05SplhiIi5Hhm6J9KXnXVVan3vv/+\newD69OkDqF9hfePLtDpP/X3274/dvn07AF988UW5Y/1Qr5EjR2Y8V8OGDVPb9957L6DlBKI2e/bs\ntNenn356QpFUjzJDERFUGYqIADl6m7xjxw4gmNli7dq1qX2dOnUCggcpUr/4dWmq45xzzgGgffv2\nQNBFY+rUqXWKwa++F55DUbLHd7L25VVfKDMUESFHM8M1a9YAwQpqYb7bhua/y13+4RbAzJkza32e\nxx57rMpj/MOVBg3S/6737dsXCNbeDjv22GNrHZNUbcaMGUDwsNPPap3rqx0qMxQRIccyQ99Js3fv\n3mnvjxgxIrWd64/nBaZPn57aHj58OFB+ogavqKgIqLwd8JJLLgGgoKCg3L5f/vKXAHTu3Ll2wUrW\nfPPNNwA888wzae/76brC3ZtykTJDERFyLDMcO3YsUH4YT7itwcxijUnqprrr4j7yyCMRRyJR8+23\nfoXKfv36AXD11VcnFlNNKDMUESFHMkPfL+nBBx9MOBIRqS2fGfp1kusbZYYiIuRIZujXQP7qq6/S\n3vejTTTdk4hETZmhiAiqDEVEgBy5TS7Lr5w2b948AFq3bp1kOCLyA6DMUESEHMkMb7jhhrSfIiJx\nU2YoIgKYc676B5t9Bqyr8sD8UeCca5t0EHFSGec/lXFmNaoMRUTylW6TRURQZSgiAkT8NNnM9gDm\nlb5sB+wAPit9/TPnXOYZP+t2zS5AeD6ojsANzjnNAhGBhMq4AJgM7Ak44O8q3+gkUcal150M9AE2\nOOe6RXGNtOvF1WZoZrcBXzvnRpR530rj2BnBNRsBG4DDnHPrs31+SRdXGZvZ3sCezrkVZtYKWA6c\n6px7Nxvnl4rF+T02sxOArcC4OCrDRG6TzayTmRWZ2cPAKqCDmf0vtH+gmU0o3d7LzKab2TIzW2pm\nR9bgUr2At1QRxi/KMnbObXTOrSjd/hJ4G9gnut9GMon6e+ycWwBsiuwXKCPJNsODgJHOuS6UZG8V\neQAY7pzrDpwD+P+5PcxsTBXXGAj8KxvBSq1EXsZm9mOgK/BqdkKWGorjexyLJEegrHHOlV8LtLye\nwIGh6f53N7OmzrklwJKKPmRmTYDTgOvqHKnUVtRl3Ap4AhjinPu6ztFKbURaxnFKsjLcEtreCYQX\nN2kS2jZq10h7GrDEOVdcy/ik7iIrYzNrDEwHJjnnZtUpSqmLqL/HscmJrjWlja6bzWx/M2sAnBna\nPRe40r8ws+o2pJ6HbpFzRjbLuLSx/h/ACufcAxGEK7UQ0fc4NjlRGZa6HpgDLALCDzyuBI4xszfM\nrAi4FCpvazCzlsDPgZnRhiw1lK0yPoGSP3a9zGxF6X+/iDh2qZ5sfo+nAQuBLma23sx+HWXgGo4n\nIkJuZYYiIolRZSgigipDERFAlaGICKDKUEQEqGGn6zZt2rjCwsKIQsk9H3zwAcXFxVb1kflDZZz/\nVMaZ1agyLCwsZNmy6oy8yQ/du3dPOoTYqYzzn8o4M90mi4igylBEBFBlKCICqDIUEQFUGYqIAKoM\nRUSAZCd3rdCWLSXzRQ4dOhSAMWOCGX78Y/Jp06YBUFBQEHN0IpKPlBmKiJCjmeHGjRsBGD9+PAAN\nGzZM7fOdRWfPng3A73//+5ijk9p47bXXABgwYABQMiqgtp577rnUdufOnQHo0KFD7YOTxPjvcd++\nfQEYPXo0AIMHD04dE/7+R0mZoYgIOZYZfvbZZwAMGjQo4Ugk2+bMmQPAtm3b6nyuWbOC9Z8mTpwI\nwNSpU+t8XonP559/DqRngABDhgwB4JJLLkm917Rp01hiUmYoIkKOZIYPPFCywNnMmSXrN736atXr\ngS9cuBAAv4bLIYccAsDxxx8fRYhSS99//z0ATz/9dNbOGR54f//99wNBD4TmzZtn7ToSnZdeegmA\nDRvS150/77zzAGjSpEm5z0RNmaGICDmSGV5zzTVAzZ4aTZ8+Pe3nfvvtB8Bjjz2WOubwww/PVohS\nSy+88AIAixYtAuD666+v8zk3bdqU2l61ahUA33zzDaDMMJeF24vvuuuujMdceOGFAJQsjR0vZYYi\nIqgyFBEBEr5N7tOnDxA8BNmxY0eVn2nTpg0Q3A6tW7cOgLVr1wJwxBFHpI7duXNn9oKValu5cmVq\ne+DAgQB06tQJgBtvvLHO5w93rZH644033kht+0743i67lFRFp556aqwxhSkzFBEhgcxwwYIFqe23\n334bCBpLK3qAcvnll6e2e/fuDcCuu+4KwPz58wH4y1/+Uu5zf//734HyHTslWuGy8A82pkyZAkCL\nFi1qfV7/4CT8byiJhnapHf+wM5NevXrFGElmygxFRIgxM/QD830bEkBxcXHGY303mbPOOguAW2+9\nNbWvWbNmacf6KbzGjh1b7pzDhg0D4NtvvwWCSR0aNWpUu19CKvX4448D6R2sfVthuC23tnx3jHA2\neOKJJwKw22671fn8Eq1wRu81btwYgLvvvjvucMpRZigiQoyZ4fbt24GKs0EIhtI9+uijQPDkuDI+\nM/RPKa+77rrUPj9Ey2eIfpqgjh071ih2qR4/4a7//w7Zaa/1dxWPPPIIEDx5BLjpppsAZfu5zHe4\nf+WVV8rt83d63bp1izWmTJQZioiQI8PxfHvSpEmTgOplhGX5rO/hhx9Ovbd06dIsRCdV+eKLLwBY\nvHhxuX1XXHFFnc8/btw4IJjirUuXLql9J510Up3PL9GqbOKVXOrpocxQRIQEMsNMo0yWLFlS5/P6\nUSzhUSdlR7b4p9K+z5tkhx+Av379eiCYhilb1qxZk/a6a9euWT2/RCtTZuif/mfjziFblBmKiKDK\nUEQEiPE22a99HNVKV36VreXLl6feKzvM7/bbb4/k2j90LVu2BILuEeGJGvwQutatW9f4vJ9++ikQ\ndNnxjjnmmFrFKfF6+eWXgaBLVJgfTrvvvvvGGlNllBmKiBBjZvjUU09l9Xy+m0VRURFQ+XAe31VH\nHXOj4Vcv80Pv/LA8gNNOOw1I7wyfyZtvvpna9g9M/PRsZSdjaNBAf8PrA78Cnn+QGZYLEzOUpX9V\nIiLkSKfr2vDTRD300EMVHlNYWAjA5MmTgWACCInGbbfdBqRnAv6OIDxBRyZt27ZNbftMsKKhmxdf\nfHFdwpSYlG3rDU+mcdlll8UdTpWUGYqIUA8zQ79UgJ8YtjJ+2NZxxx0XaUxSonPnzkD6CoX+6X7Z\njtNl+enawgYNGgSU7yTv2yglN/nO92WfIoefHGdjSrdsU2YoIkKMmWFliz4988wzaa8vvfRSADZu\n3Fjheaoz3Xu2n2BLzR166KFpP2vixz/+ccb3w/0Yf/rTn9YuMImMn7Kr7FPkfv36JRFOtSkzFBFB\nlaGICBDjbbKft8zPOh3mO+aWHaqXaeiev82uzkp6Ur/526yyt1u6Nc5tvrO15wc9XHPNNUmEU23K\nDEVEiDEzHDBgAADDhw9PvVfZeihV8X9tfHeO8ePHA9C+fftan1Nyi39IprWR65c5c+akve7QoQMQ\nTM6Qq5QZiogQY2boV7HzK98BzJw5E4BRo0bV+Hx//vOfgWAtZMk/fr1rT52tc5tfAXP16tVp7zdp\n0gTI/YlSlBmKiJDAcDy/NnJ4u3fv3kCwCpqfqPWMM84A4He/+13qM/7JYniFNMlPfrVEP8D/lltu\nSTIcqYKfWs0PtVu1ahUA+++/f2Ix1YQyQxERcmSihlNOOSXtpwgEGca1114LaI3kXOf7/vrp9Xwv\ngMMOOyyxmGpCmaGICDmSGYpk4tuOpX7Ze++9AZg4cWLCkdSMMkMREVQZiogAqgxFRABVhiIigCpD\nERFAlaGICACWabX7Cg82+wxYF104OafAOde26sPyh8o4/6mMM6tRZSgikq90mywigipDERFAlaGI\nCBDx2GQz2wOYV/qyHbAD+Kz09c+cc99FdN0+wEigITDWOfe3KK4jyZVx6bV3AV4D3nfO9Y/qOj90\nCX6PJwN9gA3OuW5RXCPtenE9QDGz24CvnXMjyrxvpXHszNJ1GgHvAD8HPgaWAb90zr2bjfNLxeIq\n49B5hwHdgGaqDOMRZxmb2QnAVmBcHJVhIrfJZtbJzIrM7GFgFdDBzP4X2j/QzCaUbu9lZtPNbJmZ\nLTWzI6s4/ZHAW865dc65bcBjQL+ofhfJLOIyxswKgF7ApKh+B6lc1GXsnFsAbIrsFygjyTbDg4CR\nzrkuwIZKjnsAGO6c6w6cA/j/uT3MbEyG4/cBPgq9Xl/6nsQvqjIGGAUMBdQ3LFlRlnGskpzPcI1z\nblk1jusJHBhaO3d3M2vqnFsCLIksOsmGSMrYzPoDHznnVphZz+yFK7WQN9/jJCvDLaHtnUB4pfAm\noW2jZo20G4AOodf7UvlfLIlOVGV8NDDAzPqWnqeVmU12zg2qU7RSG1GVcexyomtNaaPrZjPb38wa\nAGeGds8FrvQvzKyqhtTFQBczKzCzH1GSks/KdsxSM9ksY+fcMOfcvs65QuAC4DlVhMnL8vc4djlR\nGZa6HpgDLKKknc+7EjjGzN4wsyLgUqi4rcE5tx24CngeKAKmOOfeiTp4qZaslLHktKyVsZlNAxZS\nktysN7NfRxm4xiaLiJBbmaGISGJUGYqIoMpQRARQZSgiAtSwn2GbNm1cYWFhRKHkng8++IDi4mKr\n+sj8oTLOfyrjzGpUGRYWFrJsWXU6m+eH7t27Jx1C7FTG+U9lnJluk0VEUGUoIgKoMhQRAVQZiogA\nqgxFRABVhiIigCpDEREg2cldRUQA2Lx5MwAffvhhhccUFBQAMHLkSAC6du0KwAEHHADAIYccUqcY\nlBmKiJBwZvjpp58CcM455wBw9NFHA3DZZZcBJT3ls+GLL74A4KWXXgLglFNOAaBRo0ZZOb+I1MxT\nTz0FwOzZswF48cUXAXjvvfcq/MyBBx4IlAyvA9i2bVva/p0767ZKqTJDERESyAx92wDAT37yEyDI\n3Pbaay8g+xnhYYcdBkBxcTFAalzm/vvvn5XrSPV9+eWXAPzpT38CYNWqVQDMnTs3dYwy9vywZs0a\nAB566CEAxo0bl9q3detWAGoy0/4770S7eocyQxERYswMfVbm2wcBPv/8cwCuvLJk0azRo0dn9Zp3\n3XUXAGvXrgWCv0zKCOM3ZcoUAG666Sag/FNDnzEC7LHHHvEFJpFZv75kPahRo0bV6TwHHXQQEDw9\njooyQxERYswMX3vtNSB4ahR2yy23ZO06b775Zmp7xIgRAJx5Zsnyreeee27WriPV47ODa6+9Fgju\nEMzS59ocMmRIavvBBx8EoHXr1nGEKLXgyxGCzO/YY48Fgt4ajRs3BmDXXXcFoEWLFqnPfP311wD8\n4he/AIKsr0ePHgAceuihqWObNm0KQPPmzbP8W6RTZigigipDEREghttk37H6iSeeKLdv4sSJALRt\n27bO1/G3x7169Sq3b8CAAQC0bNmyzteRmvFNFf5hWUWmTp2a2n7mmWeA4GGLv4X2t12SnC1btgDp\n37PXX38dgJkzZ6Yde9RRRwGwfPlyIL3LnH+Atu+++wLQoEHyeVnyEYiI5IDIM8M//OEPQNC1wneA\nBjj77LOzdp2XX34ZgI8//jj13sUXXwzABRdckLXrSNXWrVuX2p40aVLaPj+Y3newf/7558t93neW\n91nl+eefD0C7du2yH6xUy3fffQfAr371KyDIBgFuvPFGAHr27Jnxs5kGUey3335ZjrDulBmKiBBD\nZui7UPif++yzT2pfXdqA/HCeu+++GwiG/IS7bPg2SYnXihUrUtu+M/Xxxx8PwIIFCwD49ttvAXjk\nkUcA+Otf/5r6zOrVq4Egy+/Xrx8QtCWqy018fBcY/z3zEyuE2/mHDh0KQLNmzWKOLruUGYqIkMBE\nDX7qHoDevXsDsNtuuwEwePDgKj/vO237n4sXL07bn812SKmd8NRKPlP3na69Jk2aAPCb3/wGgMcf\nfzy1zw/w94P4fcahp8nx80+I77nnHiCYYHXhwoWpY3yn6vpOmaGICDFkhldffTUA8+fPB2Djxo2p\nfb79yGcATz75ZJXn88eWHc7VsWNHIGjbkOT861//Kvfev//9bwD69++f8TN+WrVMjjzySCB9OJfE\nY9GiRWmv/TA53z8wnygzFBEhhszw8MMPB2DlypVA+pPGZ599FoDhw4cDsOeeewIwaNCgCs934YUX\nAnDwwQenve+XDPAZoiTnvPPOS237bP/VV18F4O233waCfw8zZswA0if99W3I/j0/9Zov+y5dukQW\nu6QLt+VC8ET/9ttvT73Xt29fIH1yhfpImaGICKoMRUQAsJqsQdC9e3dXWUN3HN5//30guB3u1q0b\nAM899xyQnUkfvO7du7Ns2TKr+sj8kY0y3rRpU2rbl5MfYlfRA7DwwH/fgf70008H4N133wWCVRPH\njBlTp/jCVMaVKztoIpOGDRsCcPnllwPBnIQfffQRAJ06dQKCNY/C/Bo4flKHKB7MVLeMlRmKiJDw\nusm1cccddwDBXyr/8CWbGaHUTXi43LRp0wA466yzgPIZ4lVXXQXAvffem/qM75Dtp17zQ/XmzJkD\nBJ2yQQ/MovbHP/4RgPvuu6/CY3bs2AEEGb3/WRP+4emJJ54IpE/pFhdlhiIi1JPM0GcXAJMnTwag\nVatWgFZSy3V+WiffRcNPzOC7z/hM32eDYTfffDMAb731FhB00/GfgeDfg0TDD8Pzq1r66dS2b9+e\nOsavc+MzxNrwk0D773p4JTw/yW/UlBmKiFBPMkPf0TPstNNOA9Ini5Xc5TPEiiYAzcSviuZXNfSZ\n4QsvvJA6xj+51rRe0fBPio844gggeLIfNm/ePCDIFm+77TYAli5dWuPr+bbk//znPzX+bF0pMxQR\noR5mhn7tVP+US/Kfb6+aNWsWkP6k0a+xnM21t6VmTj755LTXfsitzwwbNWoEBMtwAFx66aUAjBw5\nEgjakpOkzFBEBFWGIiJAjt8m+2FX4RXv/KpqenDyw+HX1B02bBiQvj6vb6wfOHAgAAcccEC8wUk5\nfgZ7v2qef7DiZx8CeO+994BgxvqywmslxUWZoYgI9SQzDA8S79OnT9oxX331FRDMfZeL67FKdvhJ\nOe68887Ue/5B2g033AAE63P7bjkSv86dOwNBl6hHH3203DHh7lEAu+xSUhX5LnPh4ZlxUWYoIkKO\nZ4aZ+L8gPgPwj+b98B0Nz8p/F110UWp77NixAEyfPh0I2qLKzoQu8fFZ+ahRo4Dg7i3ckfqTTz4B\noLCwEAjK1LcBJ0GZoYgI9TAzHD9+PAATJkwA4Le//S0QDOqX/Beerm3u3LlAsJ6vn1ggFzrx/tD5\nnh9+rfR//vOfqX2vvPIKEGSCfgqvJCkzFBEhxzPD0aNHA3Drrbem3jv++OMBGDx4MAC77747AI0b\nN445OskFvveAXzbAD9krKioCtJJeLvGrG5bdzhXKDEVEyPHM8LjjjgNg/vz5CUciuc5PHnvIIYcA\nsHr1akCZoVSfMkMREVQZiogAOX6bLFJdfk2ctWvXJhyJ1FfKDEVEUGUoIgKoMhQRAcD8alTVOtjs\nM2BddOHknALnXNuqD8sfKuP8pzLOrEaVoYhIvtJtsogIqgxFRICI+xma2R7AvNKX7YAdwGelr3/m\nnPsuwmvvArwGvO+c6x/VdX7okipjM7sOuKT05Rjn3OgoriOJlvF6YHPp9bY553pEcZ3U9eJqMzSz\n24CvnXMjyrxvpXHszPL1hgHdgGaqDOMRVxmbWTdgMnAk8D3wHPAb55x6XEcszu9xaWXY1Tn3v2yd\nszKJ3CabWSczKzKzh4FVQAcz+19o/0Azm1C6vZeZTTezZWa21MyOrMb5C4BewKSofgepXMRl3BlY\n7Jzb6pzbDrwEnBnV7yKZRf09jluSbYYHASOdc12ADZUc9wAw3DnXHTgH8P9ze5jZmAo+MwoYCuhR\nebKiKuOVwAlm1trMmgOnAh2yG7pUU5TfYwfMN7P/mNklFRyTNUmOTV7jnFtWjeN6AgeGlgvd3cya\nOueWAEvKHmxm/YGPnHMrzKxn9sKVWoikjJ1zb5rZ/cBc4GtgOSXtShK/SMq41JHOuQ1m1g543sze\ncs4tykLMGSVZGW4Jbe8ELPS6SWjbqFkj7dHAADPrW3qeVmY22Tk3qE7RSm1EVcY458YB4wDMbDiw\nug5xSu1FWcYbSn9+bGZPAj8DIqsMc6JrTWmj62Yz29/MGpDe/jMXuNK/KG08r+xcw5xz+zrnCoEL\ngOdUESYvm2VcesyepT8Lgb7A1GzGKzWXzTI2sxZm1qJ0uzklzwDezH7UgZyoDEtdD8yhpOZfH3r/\nSuAYM3vDzIqAS6HKtgbJTdks45mlx84ELnfOfRlh3FJ92Srj9sD/mdnrwFJghnNubpSBazieiAi5\nlRmKiCRGlaGICKoMRUQAVYYiIoAqQxERQJWhiAigylBEBFBlKCICwP8D3P5bzM0W5d8AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc310557978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the first images from the test-set.\n",
    "images = data.test.images[0:9]\n",
    "\n",
    "# Get the true classes for those images.\n",
    "cls_true = data.test.cls[0:9]\n",
    "\n",
    "# Plot the images and labels using our helper-function above.\n",
    "plot_images(images=images, cls_true=cls_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function to plot example errors\n",
    "\n",
    "Function for plotting examples of images from the test-set that have been mis-classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_example_errors(cls_pred):\n",
    "    # cls_pred is an array of the predicted class-number for\n",
    "    # all images in the test-set.\n",
    "\n",
    "    # Boolean array whether the predicted class is incorrect.\n",
    "    incorrect = (cls_pred != data.test.cls)\n",
    "\n",
    "    # Get the images from the test-set that have been\n",
    "    # incorrectly classified.\n",
    "    images = data.test.images[incorrect]\n",
    "    \n",
    "    # Get the predicted classes for those images.\n",
    "    cls_pred = cls_pred[incorrect]\n",
    "\n",
    "    # Get the true classes for those images.\n",
    "    cls_true = data.test.cls[incorrect]\n",
    "    \n",
    "    # Plot the first 9 images.\n",
    "    plot_images(images=images[0:9],\n",
    "                cls_true=cls_true[0:9],\n",
    "                cls_pred=cls_pred[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PrettyTensor API\n",
    "\n",
    "This is how the Convolutional Neural Network was implemented in Tutorial #03 using the PrettyTensor API. It is shown here for easy comparison to the Keras implementation below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if False:\n",
    "    x_pretty = pt.wrap(x_image)\n",
    "\n",
    "    with pt.defaults_scope(activation_fn=tf.nn.relu):\n",
    "        y_pred, loss = x_pretty.\\\n",
    "            conv2d(kernel=5, depth=16, name='layer_conv1').\\\n",
    "            max_pool(kernel=2, stride=2).\\\n",
    "            conv2d(kernel=5, depth=36, name='layer_conv2').\\\n",
    "            max_pool(kernel=2, stride=2).\\\n",
    "            flatten().\\\n",
    "            fully_connected(size=128, name='layer_fc1').\\\n",
    "            softmax_classifier(num_classes=num_classes, labels=y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequential Model\n",
    "\n",
    "The Keras API has two modes of constructing Neural Networks. The simplest is the Sequential Model which only allows for the layers to be added in sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Start construction of the Keras Sequential model.\n",
    "model = Sequential()\n",
    "\n",
    "# Add an input layer which is similar to a feed_dict in TensorFlow.\n",
    "# Note that the input-shape must be a tuple containing the image-size.\n",
    "model.add(InputLayer(input_shape=(img_size_flat,)))\n",
    "\n",
    "# The input is a flattened array with 784 elements,\n",
    "# but the convolutional layers expect images with shape (28, 28, 1)\n",
    "model.add(Reshape(img_shape_full))\n",
    "\n",
    "# First convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "                 activation='relu', name='layer_conv1'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Second convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "                 activation='relu', name='layer_conv2'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Flatten the 4-rank output of the convolutional layers\n",
    "# to 2-rank that can be input to a fully-connected / dense layer.\n",
    "model.add(Flatten())\n",
    "\n",
    "# First fully-connected / dense layer with ReLU-activation.\n",
    "model.add(Dense(128, activation='relu'))\n",
    "\n",
    "# Last fully-connected / dense layer with softmax-activation\n",
    "# for use in classification.\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Compilation\n",
    "\n",
    "The Neural Network has now been defined and must be finalized by adding a loss-function, optimizer and performance metrics. This is called model \"compilation\" in Keras.\n",
    "\n",
    "We can either define the optimizer using a string, or if we want more control of its parameters then we need to instantiate an object. For example, we can set the learning-rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.optimizers import Adam\n",
    "\n",
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a classification-problem such as MNIST which has 10 possible classes, we need to use the loss-function called `categorical_crossentropy`. The performance metric we are interested in is the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Now that the model has been fully defined with loss-function and optimizer, we can train it. This function takes numpy-arrays and performs the given number of training epochs using the given batch-size. An epoch is one full use of the entire training-set. So for 10 epochs we would iterate randomly over the entire training-set 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "55000/55000 [==============================] - 5s - loss: 0.2261 - acc: 0.9335        \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.callbacks.History at 0x7fc30e732f98>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x=data.train.images,\n",
    "          y=data.train.labels,\n",
    "          epochs=1, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Now that the model has been trained we can test its performance on the test-set. This also uses numpy-arrays as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9152/10000 [==========================>...] - ETA: 0s\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b"
     ]
    }
   ],
   "source": [
    "result = model.evaluate(x=data.test.images,\n",
    "                        y=data.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print all the performance metrics for the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.0618685603024\n",
      "acc 0.9801\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can just print the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 98.01%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "images = data.test.images[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the true class-number for those images. This is only used when plotting the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_true = data.test.cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as One-Hot encoded arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViI\nsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL\n19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpT\nRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fn\nc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF98\n8QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/Pe\nTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JL\naDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnD\nqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqo\nSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOje\nvXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfA\nKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U\n78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih5\n8/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5Xhrd\nunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzm\nGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH75\n5QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlW\nAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8\neDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcA\nmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oS\nUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2m\nh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQp\nsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtX\nL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV\n6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwx\nScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qE\nMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJ\njzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueee\ne/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/J\nfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv\n1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGCl\nYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim2\n0L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXM\nP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f\n//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUp\nIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/v\nYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHK\nLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591\nE1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk5\n7l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTu\ny3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRi\nlrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tn\ncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj7\n7mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHT\ngEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA\n6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTc\nD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7\nyhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7\n+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRD\nzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+Y\nqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHj\nGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0ef\nVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD9\n3T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId\n5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGR\nBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZR\npikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7X\nY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0\nqMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrr\nr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LL\nwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBF\nZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpv\nL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377\nbZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/nj\nQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWent\nvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCN\nCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdm\nzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4p\nrngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+F\nR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzV\nq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD\n8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDt\nJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQ\ntuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZg\nn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOg\niyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIi\nCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRT\nT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh\n9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMA\neP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxT\nRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCO\ncPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1\nukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcX\nL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqaw\nOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1\nzitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSki\nkoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0R\nkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvV\nzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m\n9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l\n73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OA\nvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7p\nFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7\nv+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+\n4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9a\nKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHs\nYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7Hxj\nG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7\n+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX\n87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3ef\nZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0u\nLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4\nDrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd\n5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eyg\np5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3\ndvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XG\nx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B31\n5roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYe\neI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqj\nap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSp\nk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajS\nFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801\nAOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZ\nZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu\n0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73\nvwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBM\nU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDu\nXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCC\nXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJE\naYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrr\nA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADA\nhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70A\nmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuX\np9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5\n+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+\nqvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcD\nmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFD\nBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaj\nu+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcf\nbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4\nYxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7Dzzjtn\nvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3N\nkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz2\n2GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa8\n4qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1w\ne77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOB\nzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccd\nAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7+\n+uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRh\nFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosy\nTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM44\n4wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAA\ntttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZ\nuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoK\nQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6c\nCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDp\np58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJ\noKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJW\nE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8\nF8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZP\nZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWz\nw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/D\nF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X\n1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCP\nmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY\n4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9\nPmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j7\n18CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2\nZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72B\nL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXx\nq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKm\nKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zu\nxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YL\nwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEe\nZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1\nSl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeF\nu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+g\niTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYP\nW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+\nFr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2br\nkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerG\nVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bp\nyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Ch\nmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYp\nHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoi\nIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc30d870dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            cls_true=cls_true,\n",
    "            cls_pred=cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=data.test.images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XvcVWP+//HXJ6GDUweUUg2JTENIGeVYclYh9cPDaWIo\n0vg6zjAyCBkkSk5TKIepSI7JsZFjKbkrJENTFDFJhkjX74+9rr32vk97r3uf7/v9fDx63GuvvQ6f\nu+ve1/6sa13rusw5h4iIpKdeoQMQESklqjRFRCJQpSkiEoEqTRGRCFRpiohEoEpTRCQCVZoiIhGo\n0hQRiUCVpohIBPUz2bl58+auXbt2WQqlNMydO3e1c27bQseRLyrj2k9lHE1GlWa7du2YM2dOJoco\nOWb2eaFjyCeVce2nMo5Gl+ciIhGo0hQRiUCVpohIBKo0RUQiUKUpIhJBRnfPc+2DDz4A4NBDDwVg\n9erVALz77rvxbbp06ZL/wESkzlKmKSISQVFmmn/4wx8AeOihhwDYsGEDAB06dACgRYsWhQlMROo8\nZZoiIhEUZaY5Y8YMoGKG+fzzzwPQunXrwgQmWfe73/0OgLKyMgAOOeQQAF5++eWCxSRSHWWaIiIR\nFFWmOXjwYABWrVoFwK677grAc889B8SekZXaYdiwYQAsXLgwaf0BBxxQiHAkh3xvl3vuuQeAjz/+\nGICdd945vs3xxx8PQLdu3QDYdtviHS9FmaaISARFlWlOnjwZgF9//RWAf/7zn4AyzNrkwgsvBODO\nO+8EwDkHwGGHHQbAX//61yr3vffeewH4v//7vyq36d+/PwD3339/5sFKRr788ksATjzxRACWLVsG\nQP36sWpn1qxZ8W3Hjx8PhP2ub731VqA4rzyUaYqIRFDwTPMf//hHfHnNmjUADBgwAICOHTtWus8X\nX3wRXy5/l9U/PbTDDjtkNU6pucWLF8eXJ06cCMDGjRuB8CriuOOOA2CTTTapsP/dd98NwNChQwH4\n+eefk973d9wBTjrppCxFLZmqVy+Wk33//fcAbLPNNgA88sgjQPh5B7jiiisA4uN6Tp8+HVCmKSJS\n8lRpiohEUPDL87Vr18aX/Q0g3+3ANxj7Lkc33XQTAJ9++ml8n+XLlycdz3d8b9y4MQDNmzePv+cv\n73xj829+85ss/RZSnb59+8aXv/32WyD8v3/mmWeAik0x48aNiy//6U9/AsLLct915ZhjjgFg6623\njm/bqFGjrMYuNbf99tsD4SW2v+T2n+uBAwfGt+3RowcAY8eOBeCuu+4CYP/99wegX79+eYg4Pco0\nRUQiKHimOWbMmArr/I2gp59+Ggi7kaxfvz7l8cpnnh999FF8efbs2QDsvvvuScdXl6bc8J2Yv/rq\nqwrvDRo0CKj6Zp/vbgbw008/AeGNhD333BOAli1bZi9YyZlTTjkFCDNNPyDP8OHD49v499577z0A\nfvjhh6SfxUSZpohIBAXLNB944AEAPvvsswrv/f3vfwfgiSeeAMIM07d7XHzxxfFtW7VqVe15Hnvs\nsfjyww8/DMCiRYsAuOOOOwC45ZZbIscvqY0aNQpI7lrSp08fILkME913330AvPXWWxXe8x2eu3bt\nmtU4Jbd8O7OZAWEn97POOqvCtg0aNADCv4NTTz01HyFGokxTRCSCgmWaflAOf8c80W233Zb02t+F\nmzBhAgA77bRT2ufZZ5994stHH300EHaG9o/y7bfffkDYdiqZWbp0KRB2ZE+05ZZbArDZZpsB8Mor\nrwAwd+5cIGzn+vHHH+P7+B4R3bt3z03AklO+l8OUKVMAmD9/PgDXXXddfBv/OO2+++4LwGmnnZbP\nECNRpikiEkHB755XZ7vttgPg0UcfBaJlmJXZZZddgDBz9dmubz9TppkdTz31FBA+PpdoxYoVABxx\nxBFAOGhDYmZZnu8R4a8UfB/Onj17ZiliyQc//Jv/eeONN8bf++WXX4DizjA9ZZoiIhEUZabpByA9\n99xzATjooIOyclw/iIfPXH3bph9GTHfRc8+3YdbEJ598AsCQIUOAcCBj/3cipW/TTTctdAgpKdMU\nEYlAlaaISARFeXle/vI529q3b5+T40r69t57byDs+Pz+++8D4c0jP0slhGNs+q4q/tHYv/3tb0Dy\nYA7+Jp8Ur2+++QYIuxklatasWb7DiUyZpohIBEWZae622245Pf7q1atzevy6zmd+fjCOF198Mf5e\nr169gPCRWD+En+/U7Efuvv322+P7+Kx02rRpQDhUnJ+Dxs8dBHDllVdm81eRHPBd0jZs2BBft/nm\nmwNhR/hipkxTRCSCosw0c8V3oB0xYkTS+hNOOKEQ4dRabdu2Tfp5+OGHp9ynurZIP8jw6aefDoRD\n+vnH8hIHLFamWfxWrlxZYV1lg3cUK2WaIiIRFCzTPPLIIwG4+uqr4+v8EHCTJk0CwsFKmzRpkpVz\n+iHh/PzqfvCI6ubRlvw4//zzgXD6i+r4aRJ8punvxgK88MILAPTu3TvbIUqGfDn52UUTldIjzMo0\nRUQiKFim6fvhJbYn+kGCL730UiAciNRnIf5xOT8xU3X8nTn/6B2E02h4J554IhBOfyHFI3Hg4vI6\nd+4MwBZbbAHAunXr4u8lTronxcVPe1LZwOOJk+MVO2WaIiIRqNIUEYmg4F2OEm/C+DlEpk6dCoSz\nGfr5yt9++20geS7z8g499FAgnM3Q31SC8BEtP3dNrh7TlOg6deoEhI+4nnfeefH3ysrKALjqqqsA\n2HnnnYFwvqHEMhbJNWWaIiIRFDzT3GuvveLLDz30EBDeCBo5ciRQedZYlcTH7wBatGgRX/bzz5xz\nzjk1D1hyws8D5DPMK664Iv6en8vJXxn4xyp9FzWRfFKmKSISQcEzzcr47kg+8/RZxw033ABUn3H6\njMVnk4ldmvwAElK8LrroIgBmzpwZX/f8888DcMABBwDh7KGVzY0ukmvKNEVEIijKTLM83/ncZ57+\np9Re/sEGCNu2R48eDSjDlMJSpikiEkFJZJpS97Rq1Sq+fNlllwGwceNGILyb7iVOd+EHM5bi44f/\n69KlCxAOOA3QvXt3IBycOrFNu9go0xQRiUCZphQ9P1/9HXfckfRTSkvTpk0BePbZZ4GwXCHsc+sH\n5ylmyjRFRCJQpSkiEoEuz0Ukr7bddlsgnLOr1CjTFBGJQJWmiEgEqjRFRCIw51zNdzb7Gvg8e+GU\nhLbOuW0LHUS+qIxrP5VxNBlVmiIidY0uz0VEIlClKSISQbWVppk1M7P5wb+VZrYi4fVmuQrKzJqa\n2eNm9qGZLTazrim2H2RmXwdxLTazszI8/0Qz65tiGzOzsWb2iZktMLPOmZyzUApYxheb2UIzKzOz\nSWa2eYrtr0uI7QMzOzrD87+ebpmZ2QAzcyrjyOe9KCjjhWZ2QRrbF+Jz/Fsze9PM1pvZsHSOW23n\ndufcN0Dn4ODDgXXOub+XO6kRaxvdmM4J03QHMN05d3xQqA3T2GeSc26YmbUAysxsunNudUKc9Z1z\nG7IY47HAjs659mbWAxgDdM/i8fOiEGVsZm2Bc4FOwHpgCtAfmJhi15udc6PMrBPwiplt5xIa5XNQ\nxpjZVsBgYE6qbYtVgcq4M3A60AXYALxgZk875/6dYtd8f45XAxcAJ6a7Q40uz82svZktMrNJwEJg\nRzNbk/D+QDO7L1jePsga55jZO2a2X4pjNwW6OecmADjnfnbOfZdubM65lcBnQJsgO3nQzGYDE8ys\nvpndGsSxwMwGBeesF2SNH5rZTKDqOYJDfYAHg3O+DrQws1pzxzWXZRzYFGhA7Iu7EfBFurE558oA\nA5oE2cRdZvYOMMLMtjCzCUEc88zs2CDGRmY2OchgpgbnTseI4F+tm8Utx2XcEXjLOfejc+4XYBbQ\nL8U+cfn6HDvnVjnn5hCr2NOSSZvmbsBtzrndgRXVbDcaGOmc6wKcBPhC6GZm4yrZfifg6+A/aZ6Z\n3WNmjdINyszaA22BTxPi7OmcOxU4B/jKOdcV2BcYYmZtiH3L/AbYHTgT2D/heNeb2VGVnKoV8J+E\n18uDdbVJTsrYOfc5cDux/78viZXJy+kGZWb7Az85574NVrUE9nPOXQr8FXg+KONDgVvMrAFwPvBf\n51xH4Dpgr4TjjbdKLr3NbF9gO+fcjHRjK0G5+hx/ABxksaa2xsCRwI7pBpXHz3FkmTx7vjSooVPp\nBeway/6BWHbQ0Dn3NvB2FTF1IZYyzyV2qX4JcE2K85xiZgcTywgGOefWBOd80jn3U7BNb6CjmQ0M\nXm8N7AIcCDwSXJosN7NX/UGdc39J43esrXJSxmbWDDiG2B/4WmCqmQ10zj2a4jyXmNkZwPfAgIT1\nkxMuK3sDR5rZ5cHrBkAbYmU8EsA5N8/MFvqdnXNnVhJjPeAW4JQUMZW6nJSxc67MzG4FXgTWAfOA\nX9M4T9F/jjOpNH9IWN5I7HLJS7z0MaCrc+7nNI+7HFjmCzK4lEqngXaSc66y7RLjNGCwc+6lxA3M\nLO3LhgQriH1z+glrWlP9N3UpylUZ9waW+LYqM3uCWFaQqtK82Tk3KkWcBvR1zi1N3CDhw56ubYhl\nLP8K9m0BPGtmRzvn5kU9WBHLVRnjnLsHuAfAzEYCn6SxW74/x5FlpctRULP/18x2Cb6hE4N/ERji\nX1R2GVTuWMuBVUF6DtATWBTse6GZnZtBqDOAwWZWPzjermbWkFh7y4CgTaQVcFAax5oOnBYcpwew\nyjn3dQaxFbVsljGwDPi9mTW0WI3UE1gc7DvSt0PW0AxiVyk+Fn8ZPgs4OVi3J/Db6g7inPvWOdfc\nOdfOOdeO2I2go2pZhZkky2WMmW0X/GwHHEfwpVhkn+PIstlP8zJiv8wbxLJFbwjQPWiwXQScDdW2\nhUDsj/4xM1tA7I/7xmB9R+CbDGK8G1gCzDezMuAuYtn2FGIf5EXAeOBNv0M1bSFPASvMbGlwnCGV\nbFPbZKWMnXOziX3pzCPW9rUBuD94ew9gZQYxXgM0tli3pIXA8GD9nUAzM1sMXBWcmyDOSts066hs\nfo6nBdtOA851zq0N1hfN59jMWpvZcmAoMNzMlqe6h1JSj1Ga2TNAn2x3K5HiEGSdzznnjih0LJI7\npf45LqlKU0Sk0PQYpYhIBKo0RUQiUKUpIhKBKk0RkQgymo2yefPmrl27dlkKpTTMnTt3dV0a1Vtl\nXPupjKPJqNJs164dc+aU7OAvNWJmdWpaAJVx7acyjkaX5yIiEWSUaYpkW1lZGQBDhoQPWPXp0weA\niy66qCAxiSRSpikiEoEqTRGRCHR5LkXlvPPOA+D111+Pr5s1axYQu2EBcPzxx+c9Lsmtjz/+GIA/\n/vGPAJx88skAnH322QWLqSrKNEVEIlCmKUVlwIDYgOyzZ8+Or/ODyowaFRt/WJlm7eCzS4Cjj45N\nLvrpp7HZLT777DNAmaaISMlTpilF5fzzzweSuxf98ssvhQpHcuD2228HwisHgGXLliVt07Zt27zG\nFIUyTRGRCIoy01yzJjb18pIlSwB4+OGHk95P/IaqasKsFi1aAPDmm/ER74v620tiXn45NpPvhg0l\nOai3VMOX6aJFiwD4/PPwSUb/Oe7QoQMAEydOzHN06VOmKSISQVFlmv7bZcSIEQB89NFHlW6XmF3u\nueeeQNjutXjxYgBWrVoFwMqV4RxdyjSLn2/b0jQstc+4cbH51+67774qt2nevDkArVu3zktMNaFM\nU0QkgoJnmontlf5pkP/9738ANG3aFAj75fms8sADD4zv47NH316y4447AvDjjz9WOH63bt2y/wtI\nVvn+eYnq14/9md5yyy15jkay4YsvvgDg/vtjszT7q4jKriZuvvnm/AVWQ8o0RUQiUKUpIhJBwS7P\n/SV4YqPwPvvsA8CVV14JQPfu3QFo2LBhyuP5y/HyXZD69++febCSNy+99FKFdU2aNAFg3333zXc4\nkgW+a9GCBQuAyrsJHnfccQDsvffe+QushpRpiohEULBMs1GjRkDYmTlT/iaBz2B32WUXADp27JiV\n40vhXHHFFYUOQTKw5ZZbAmF3otWrV1fYxj+E4gfx6NSpU56ii06ZpohIBAXvcpSpd999F4Cbbrop\nab3vvtSsWbO8xyTR+e4ofvDhxHbsI488siAxSXb4rLFv375A5Z3bffY5duzYpJ/FSJmmiEgEJZlp\nbty4Mb48Y8YMIGzL3HrrrQE45JBD8h+Y1NjUqVOTXp9wwgnx5d122y3f4UgO+F4x1T1G+dRTTwHh\ntBf+gZZiokxTRCSCksw0ffsXwNVXX5303o033gjAHnvskdeYpGbeeecdoGL/TE1pUfv4R5wvvPBC\nAG677bYK26xYsQII+20mDh9XLJRpiohEUJKZ5tNPP11hXZs2bQA4/fTT8x2O1MDatWsBuOCCCwD4\n+eefAejXrx8Q3mmV2sdfHXbp0iW+zrdh+nsTfkjHoUOHAnDWWWfFt+3cuXNe4qyKMk0RkQhUaYqI\nRFBSl+fz5s0Dwm4JED78f8kllwCw+eab5z8wiczPSOhvBHn+8ryquZ+k9PlugSeffHJ83eTJkwF4\n9dVXgbD5ZsyYMQBMmTIlvu37778PwLbbbpvzWCujTFNEJIKSyDR/+OEHAIYPHw4kj/jcs2dPAAYP\nHpz3uCQ6f5Xgy9LzN/ISR+WXuuOJJ54A4O677wYqfp4T5/ryNw0LRZmmiEgEJZFpjh8/Hgi7GiUO\n5nDmmWcWJCZJnx98FuCcc84Bwkdh/WAcvXr1AuC9995L+lmdrl27AtCqVavsBSsFVQoPpSjTFBGJ\noKgzzSVLlgDwl7/8JWm9v1MOyXfgpDj4LHL27NkAnHTSSfH3EtumAJ577rmkn1H4wW3POOOM+Dp/\n9923jW6yySaRjyvRvfbaaxXWHXTQQWnvf++99wIwYsQIoOJMlZXNXFkoyjRFRCIoykzTf6vccMMN\nAKxbty7p/WOOOSbvMUlqfiDZs88+G4Bp06ZFPkb79u0B+Oabb+Lr/NQl9erFvuPXr18PhP1277jj\njvi2ftm3lf75z38GoEePHpFjkdT8nOZ9+vSJr/NZ/ldffVXpPtOnTweSs9NVq1YBsGHDBiDsp+sf\nmfT7ALRo0SIrsdeUMk0RkQhUaYqIRFCUl+d+FO8HHnggab1v8Nf818Xp1ltvBSpeltevH/6Z7b77\n7kB4A2+vvfYCoFu3bkDYnSxxdP7E/SG8PPc3mhK7J/kmHX9jaebMmQBcfvnlQDiaTuvWraP+elKJ\nX3/9FYDvv/8+vs53DXzmmWeq3Tfx5o6/HN9qq62AcM6vY489FoCWLVtmKeLMKdMUEYmgKDNNP/dx\neX6Okeo89thjAAwYMCCrMUlqgwYNAmDcuHFA2IifWBY+08uEzzx79+6d9BPCLkc+w/RdWPzAD/6m\n0mmnnZZxHBJ26fIZIsB3332X1r5+JHcIrzj8qO7FPMeXMk0RkQiKMtOcM2dO0uurrroKCAd18G1a\nAI8//jgA1157LZDc/UTya6eddgLg22+/LVgMu+66a9LP888/v2Cx1AU77LADkNyO7buCeaNHjwbg\n4IMPBsJHJYcNG5aHCLNPmaaISARFmWm++eabSa995rJo0SIATjnllPh7frY6/6hllEe3RCQ7Ej93\n5T+DpZpRVkWZpohIBEWZafo7oH5AUn/n0/9M7N/lhxq79NJL8xmiiNRRyjRFRCIoykzzmmuuAcIn\nPsrKyoCw319if83DDz88z9GJSF2mTFNEJAJVmiIiERTl5bmfz9jPbywiUiyUaYqIRKBKU0QkAlWa\nIiIRWCazvJnZ18Dn2QunJLR1zm1b6CDyRWVc+6mMo8mo0hQRqWt0eS4iEoEqTRGRCFRpiohEUG2l\naWbNzGx+8G+lma1IeL1ZLgIys7Zm9qqZLTKzhWaWcuhtMxtkZl8HcS02s7MyjGGimfVNsc3lCf8X\nC81sg5ltncl5C0FlXO02vzWzN81svZmV7KCQhSjj4LzLzeyD4Dxvp7F9Icr4eDNbEJzzXTPbP+WB\nnXNp/QOGAxdXst6AeukeJ43z7AB0Dpa3ApYCHVLsMwgYFSy3AFYDzcttUz9CDBOBvhG27we8kK3/\ng0L9UxlX2GZ7oAtwIzCs0OVTSmUcHHM5sE2E7QtRxlsQ3hDfGyhLddwaXZ6bWfsgS5gELAR2NLM1\nCe8PNLP7guXtzexxM5tjZu+Y2X7VHds594Vzbn6wvBb4EGiVbmzOuZXAZ0AbM7vOzB40s9nABDOr\nb2a3BnEsMLNBQYz1zGysmX1oZjOB5pH+Q+D/AY9E3KeoqYzBObfKOTcH2JBubKUkl2WcqTyW8ToX\n1JhAYyBld6JMnj3fDTjNOTfHzKo7zmhgpHPuLTNrBzwNdDKzbsCZzrlzq9rRzHYCOgHvphuUmbUH\n2gKfJsR5oHPuJzMbDHzlnOtqZpsDb5nZC8B+wG+A3YllQYuAccHxrgdmO+eereJ8WwC9gLPTjbGE\nqIxrv1yWsQNeNjMHjHXO3Z9uUPksYzM7EbieWCV7VKrYMqk0lwbfwqn0AnY1M/+6iZk1dM69DVTZ\nzmFmWwFTgQucc+vSOM8pZnYwsB4Y5JxbE5zzSefcT8E2vYGOZjYweL01sAtwIPCIc24jsNzMXvUH\ndc79JcV5+wCvOefSm+y5tKiMa79clvF+zrkVZtYCmGlmi51zb6Q4T97L2Dk3BZhiZocA1wbHr1Im\nleYPCcsbibWJeA0Slg3o6pz7Od0DW6xx+nFgvHNuepq7TXLOVdZYnxinAYOdcy+VO1+/dGOrxEDg\noQz2L2Yq49ovZ2XsnFsR/FxpZk8CXYFUlWbBytg594qZPWBm2zjn1lS1XVa6HAU1+3/NbBczq0fs\nxoj3IjDEvzCzztUdy2JfKxOA+c650eXeu9DMqrzUS8MMYLC/DDGzXc2sITALGBC0ibQC0prS0sya\nAPsDT2UQU0moq2Vcl2S5jLcImq4ws8bAYUBZ8Lpoyjho17VguQuxm0JVVpiQ3X6alxH7Zd4gdtfM\nGwJ0DxpsFxG0/ZlZNzMbV8lxDiJ2Y+UwC7tF+DktOgLfZBDj3cASYL6ZlQF3Ecu2pwDLiLWBjAfi\ncwib2fVmVlU7xwnAc865HzOIqZTUqTI2s9ZmthwYCgy3WBeaRhnEVgqyVcYtgdlm9j7wDvCEc+7F\n4L2iKWPgJKDMzOYTa7cdkOrkJfXsuZk9A/RxztXKu5miMq4LSr2MS6rSFBEpND1GKSISgSpNEZEI\nVGmKiESQ0WyUzZs3d+3atctSKKVh7ty5q10dGtVbZVz7qYyjyajSbNeuHXPmpPMwQe1hZnVqWgCV\nce2nMo5Gl+ciIhGo0hQRiUCVpohIBKo0RUQiUKUpIhJBRnfPRUSyYe7cuQD06tULgG222QaAGTNm\nANChQ4fCBFYJZZoiIhEo0xSRvPrf//4HwB//+Mf4uqeffhqAtWvXJv3s378/AO+//34+Q6yWMk0R\nkQhKKtP88cfYWL//+c9/4ut23nlnAH7+OTYK/1lnxaZKfvTRRwFo3bp1fNt33nkHgJYtW+Y+WEnL\n6NGxgduHDh1a4Egk1z744AMg/Iy+99578ff8EJUJcxABcPDBB+cnuAiUaYqIRFASmeYvv/wCwHnn\nnQfAgw8+GH9v/PjxACxcuBCAxx57DICGDRsCsOOOO8a3/eab2Aj7yjQL54cfYvNjXX755QD8+9//\nBpRp1mZffPEFAKNGjQKSM8xU/Od73333ja879dRTsxhddMo0RUQiKMpMc8OG2NQhkydPBuBvf/sb\nAB999BEAm2yySXzbZs2aAbBs2bKkYzRv3hyAl14KZ/mcOHEiAJ06dcpF2JIGn1mOGTMGCNuZpfa6\n8cYbAZgwYULkff2Vyemnnx5f5zPVzp1jE2KedtppGUYYjTJNEZEIiirT9G0fPXv2BCpmln79n//8\n5/g+hxxyCBBmkeX5tk2An376KcsRS1QXXnghAL/73e8AaNCgQSHDkRzyT/k89NBDQHiHvDJRJni8\n7bbbABgwIDbbrjJNEZEipkpTRCSCgl+e/+tf/4ov+06vS5cuTdrmmmuuAZIvy73169cDsGLFiqT1\n/fr1A+Drr7+Or9t22zoz7UtRmTlzZnz5119/BWr2WJz/u1izZg0A++yzDwCvvPJKfJvZs2dXuu+e\ne+4JwLHHHhv5vFIzd955JxA+Elm+47q/kQMwffp0IPy78N2TEm/klvfMM88AYbekM888Mxthp6RM\nU0QkgoJlmv6mzwUXXBBf5zMJf+Pn8ccfB+Coo46q8jjfffcdUDHD2GqrrYDkzMLfhJD8ev755+PL\n9epV/z3t/y769u1b4T2fsfiri1atWgGwevXq+DYff/xxpcf1Vxlt27YF1NUpHx544AGgYobZtWtX\nIPx8Q/jAiS/TJk2aANVnmo0aNQJg++23z1LE6VGmKSISQcEyzWuvvRaABQsWxNf5DNO/l077U1Xf\nRE888QSQPLiH/4aT/PBZY2IZ33fffQDxKWPbtGkDwHbbbQeE7do+q4SwO8qSJUuSju87PPt2UoDr\nr7++0lh827b+Bgpv0KBBQHJ3wO+//x4I26v930l1Dj30UKD6K9FcUKYpIhJB3jNNP/jGtGnTKrzn\nOzr7wRzS4e+6lffZZ58BcNlll8XX+WHkJD/8wAqvvvpqfJ0feNY/9jpp0iQgzDQbN24MhI/QQphp\nrlq1Kun4Bx54IJB8NTFlyhQgfFzTDxl4xBFHAPCPf/wjo99JMnfxxRcDMG7cuPi6LbbYAoBZs2al\nfZzjjjsuu4GlSZmmiEgEec80fdbg2y4S+Sz0jTfeAML2p/r1k8P02QMk35lN1Lt3bwCuuuqqDCOW\nqN5++20gbLfca6+94u/5wRtuueUWAJo2bZq079SpUyOfr3379vFlf1d8yJAhQPh4rb87q766uXfR\nRRcBVT8a6Xu8+McsE7ctf6e9Mv5+RZ8+fTKKs6aUaYqIRJD3THOzzTYD4IwzzgDg7rvvjr/nM8ge\nPXoAcPjhhwMVswPfXgXht5bnt7366quzGLVE4ct03bp1QPKgsf4pnocffjgn5/btnlUN4CK54QcI\nB3jkkUe+xuleAAAJhElEQVSAMGtMJ3v00tm2UBmmp0xTRCQCVZoiIhEUrHO77+D64Ycfxte99tpr\nSdvMmDEj8nGPOeYYIBygQfLHj7DvL40POOAAIPlR2VwYPnx4fPmmm24CYNiwYQCMGDECSB7tX7LH\nzzCZeAPPd1Qvzz/a7MvIdwuE8AZhOq644gogHMjHN/nlizJNEZEICpZp+hsCL7zwQnydv6njO736\nby8/o6TvgvTWW2/F93n22WeTjuvnD/E3lfL9LVSX+ZtvvjHfD85RvstYtvihAhOHnvMPM/ibiBoZ\nPrfuuusuIJzptTL7778/EF6J+NkWEvmbhv4mop8nrDI+U91yyy2ByoeMzCVlmiIiERR8EOJNN900\nvuxnkLzyyiuTfpa33377VXk8P4ip/6ZSplk4ftCNlStXxte1aNGixsfzneV9duPnnkmcx97PF7PT\nTjvV+DyS2vz58wF46qmnUm47ePBgoPIM0xs9ejQATz75JADLly9PeVzfnppvyjRFRCIoeKaZbb59\nQ21ZhTdv3jwgec5q3/G5/OOT5SUOJ+cH7xg5ciQQDgXm75r7gTtAGWa++MGey08zk8g/Plt+6DZ/\ndz1x+Df/6GU6ndv9I5e+d0a+KdMUEYmgpDLNd999F0h+0L+8E088EQi/jTSxWv74gTP8/7lv00y8\nuz1w4EAAxo4dC8All1wCwCeffJJ0rMRBiIcOHQqE5b7DDjsAqbNVyZ10HpH0ZeoHzfHtlX7Q6C+/\n/DLS8Tw/FGAm7eOZUKYpIhKBKk0RkQhK6vJ848aNQPKcMFXxI8P7y0CofmY7yZyfw8fP8+NH4058\nGOHFF18EoEOHDpUew3dBS5w5tGfPngB06tQpyxFLLvkbPmPGjKnxMTbffHMg+VFZP2NloSjTFBGJ\noKQyzXT0798fgK+++goIR3mW/PFzPPnuP34+IAjndSk/34/nB2HwHaKlOP3+978HwiuGquabT5d/\nVNo/euudc845AFx66aUZHT+blGmKiERQ6zLNpUuXAuE3lJ8bWfKnfHtlYofzsrKyfIcjOdC6dWsg\nnIEhyqAZ/mGHPfbYI77OD+VXCpRpiohEUOsyzcMOOwzI7I6diKTHD8Xnf9YFyjRFRCIoqUzTT2GR\nOLuhn1rBP1rn59PW9AYikgvKNEVEIiipTNMP9/bggw/G1yUui4jkmjJNEZEIVGmKiESgSlNEJAJV\nmiIiEajSFBGJQJWmiEgE5ufSqdHOZl8Dn2cvnJLQ1jlXZyYbUhnXfirjaDKqNEVE6hpdnouIRKBK\nU0QkgmorTTNrZmbzg38rzWxFwuvNchWUmV1kZguDfxeksf0gM/s6iGuxmZ2V4fknmlnfNLf9vZn9\nmu72xaaAZdzUzB43sw+DMuuaYvu8l7HFjDWzT8xsgZl1zuSchVLAMr44+AyXmdkkM9s8xfbXJcT2\ngZkdneH5X09VZmbWwMymBGX8ppm1SXXcap89d859A3QODj4cWOec+3u5kxqxttGNqU6WjuCXPB3o\nAmwAXjCzp51z/06x6yTn3DAzawGUmdl059zqhOPWd85tyEaMiccERgAzs3ncfCpEGQfuAKY7544P\nPrgN09gn32V8LLCjc669mfUAxgDds3j8vCjQ57gtcC7QCVgPTAH6AxNT7Hqzc26UmXUCXjGz7VzC\njZcclPE5wMqgjE8FbgBOqW6HGl2em1l7M1tkZpOAhcCOZrYm4f2BZnZfsLx9kFHMMbN3zGy/FIfv\nCLzlnPvROfcLMAvol25szrmVwGdAm+Cb60Ezmw1MMLP6ZnZrEMcCMxsUxFgvyCg+NLOZQPM0TzcM\neBRYnWrDUpPLMjazpkA359wEAOfcz86579KNLY9l3Ad4MDjn60ALM6s1d9Vz/DkG2BRoQCw5awR8\nkW5szrkywIAmwVXBXWb2DjDCzLYwswlBHPPM7NggxkZmNtliVyJTg3On0gd4IFj+J3B4qh0yadPc\nDbjNObc7sKKa7UYDI51zXYCTAF8I3cxsXCXbfwAcZLHLt8bAkcCO6QZlZu2BtsCnCXH2dM6dSuxb\n5SvnXFdgX2BIkI6fCPwG2B04E9g/4XjXm9lRlZynDXA0cG+6sZWgXJXxTsDXQWU3z8zuMbNG6QaV\nrzIGWgH/SXi9PFhXm+SkjJ1znwO3E/v/+5JYmbycblBmtj/wk3Pu22BVS2A/59ylwF+B54MyPhS4\nxcwaAOcD/3XOdQSuA/ZKON74Ki7V42XsnPsZ+MHMtqkutkyGhlvqnJuTxna9gF1j2T8Q++Zo6Jx7\nG3i7/MbOuTIzuxV4EVgHzAN+TeM8p5jZwcQuBQY559YE53zSOfdTsE1voKOZDQxebw3sAhwIPBJc\nmiw3s1cT4vlLFecbBVzqnNuY8LvVNjkpY2J/d12AC4C5xC7VLwGuSXGefJdxXZCTMjazZsAxxL6o\n1gJTzWygc+7RFOe5xMzOAL4HBiSsn5zQdNAbONLMLg9eNwDaECvjkQDOuXlmttDv7Jw7M43fMS2Z\nVJo/JCxvJJZKe4lpsQFdg1o8Lc65e4B7AMxsJPBJGrtNcs5VNqVdYpwGDHbOvZS4gZmlffmfoAsw\nOfgjag70NrNfnXNP1eBYxSpXZbwcWOY/rMGlVDrTEea7jFcQu8p5K3jdmuqzsVKUqzLuDSzxbc5m\n9gSx7D5VpXmzc25UijgN6OucW5q4QQ2TF1/GKy3Wtt7YObemuh2y0uUo+Ab4r5ntYmb1SG6DfBEY\n4l9UkSInMbPtgp/tgOMI/qPN7EIzOzeDUGcAgy12Awcz29XMGhJrNx0QtHu1Ag5KdSDnXBvnXDvn\nXDtgGnBOLaswk2SzjJ1zy4FVwWU2QE9gUbBv0ZQxMB04LThOD2CVc+7rDGIraln+HC8Dfm9mDS1W\nm/UEFgf7jvTtkDU0g9hVio/FX4bPAk4O1u0J/DaNY00nduMZYs0OL6TaIZv9NC8j9su8QSyT8IYA\n3YNG+UXA2VBtexfAtGDbacC5zrm1wfqOwDcZxHg3sASYb2ZlwF3Esu0pxAp5ETAeeNPvUE17V12U\nzTK+AHjMzBYQ++O+MVhfTGX8FLDCzJYGxxlSyTa1TVbK2Dk3m1iFNI/YfYoNwP3B23sAKzOI8Rqg\nscW6JS0Ehgfr7wSamdli4Krg3ARxVtWmeQ/Q0sw+IdYmmnIC95J6jNLMngH6ZLvrkBQPlXHtFmSd\nzznnjih0LDVVUpWmiEih6TFKEZEIVGmKiESgSlNEJAJVmiIiEajSFBGJQJWmiEgEqjRFRCL4/xlh\nAaR5sjHzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc30d870d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functional Model\n",
    "\n",
    "The Keras API can also be used to construct more complicated networks using the Functional Model. This may look a little confusing at first, because each call to the Keras API will create and return an instance that is itself callable. It is not clear whether it is a function or an object - but we can call it as if it is a function. This allows us to build computational graphs that are more complex than the Sequential Model allows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an input layer which is similar to a feed_dict in TensorFlow.\n",
    "# Note that the input-shape must be a tuple containing the image-size.\n",
    "inputs = Input(shape=(img_size_flat,))\n",
    "\n",
    "# Variable used for building the Neural Network.\n",
    "net = inputs\n",
    "\n",
    "# The input is an image as a flattened array with 784 elements.\n",
    "# But the convolutional layers expect images with shape (28, 28, 1)\n",
    "net = Reshape(img_shape_full)(net)\n",
    "\n",
    "# First convolutional layer with ReLU-activation and max-pooling.\n",
    "net = Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "             activation='relu', name='layer_conv1')(net)\n",
    "net = MaxPooling2D(pool_size=2, strides=2)(net)\n",
    "\n",
    "# Second convolutional layer with ReLU-activation and max-pooling.\n",
    "net = Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "             activation='relu', name='layer_conv2')(net)\n",
    "net = MaxPooling2D(pool_size=2, strides=2)(net)\n",
    "\n",
    "# Flatten the output of the conv-layer from 4-dim to 2-dim.\n",
    "net = Flatten()(net)\n",
    "\n",
    "# First fully-connected / dense layer with ReLU-activation.\n",
    "net = Dense(128, activation='relu')(net)\n",
    "\n",
    "# Last fully-connected / dense layer with softmax-activation\n",
    "# so it can be used for classification.\n",
    "net = Dense(num_classes, activation='softmax')(net)\n",
    "\n",
    "# Output of the Neural Network.\n",
    "outputs = net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Compilation\n",
    "\n",
    "We have now defined the architecture of the model with its input and output. We now have to create a Keras model and compile it with a loss-function and optimizer, so it is ready for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new instance of the Keras Functional Model. We give it the inputs and outputs of the Convolutional Neural Network that we constructed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2 = Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the Keras model using the `rmsprop` optimizer and with a loss-function for multiple categories. The only performance metric we are interested in is the classification accuracy, but you could use a list of metrics here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model2.compile(optimizer='rmsprop',\n",
    "               loss='categorical_crossentropy',\n",
    "               metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "The model has now been defined and compiled so it can be trained using the same `fit()` function as used in the Sequential Model above. This also takes numpy-arrays as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "55000/55000 [==============================] - 2s - loss: 0.1924 - acc: 0.9409       \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.callbacks.History at 0x7fc30c02ceb8>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit(x=data.train.images,\n",
    "           y=data.train.labels,\n",
    "           epochs=1, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Once the model has been trained we can evaluate its performance on the test-set. This is the same syntax as for the Sequential Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 8992/10000 [=========================>....] - ETA: 0s\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b"
     ]
    }
   ],
   "source": [
    "result = model2.evaluate(x=data.test.images,\n",
    "                         y=data.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a list of values, containing the loss-value and all the metrics we defined when we compiled the model. Note that 'accuracy' is now called 'acc' which is a small inconsistency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.0654281976447\n",
      "acc 0.9786\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model2.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print the classification accuracy as a percentage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 97.86%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model2.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model2.predict(x=data.test.images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUFNX5//H3g6iACwqo4AJoIAjRiBHBsGgURY2igBsJ\nRlEIB8U9ijGuxDW4oYgokYCKWwAlaiS4APqTuACCsolovoqgKKCAuCDI/f1Rdbtrhpnprul9+LzO\nmTPV3bU8M7f79lO3bt1rzjlERCQ9tQodgIhIKVGlKSISgypNEZEYVGmKiMSgSlNEJAZVmiIiMajS\nFBGJQZWmiEgMqjRFRGKoncnGjRo1cs2bN89SKKVh9uzZq5xzuxU6jnxRGdd8KuN4Mqo0mzdvzqxZ\nszLZRckxs08KHUM+qYxrPpVxPDo9FxGJQZWmiEgMqjRFRGJQpSkiEoMqTRGRGFRpiojEoEpTRCSG\njPppFoMHHngAgPPOOw+AiRMnAtCrV6+CxSTpW79+PQATJkyodJ0ZM2YAMHr0aAB69OgBwIABAwA4\n7rjjchmiFIFNmzYlln2f0jlz5pR5vHjxYgBatWoFwAUXXJDY5uCDD85aLMo0RURiKMlM8/77708s\nR79NAHbaaad8hyPV4DPMyy67DIBXXnkFgNatWwPQsGHDLbbxGeWrr75a5vc999yTWOfMM8/MUcSS\nTxs3bgRg5syZANxxxx2J15555pkqt/VnJu+8807iOZ+VZoMyTRGRGEoq0/zvf/8LwMUXX5x4bvvt\ntwdg3LhxABxzzDH5D0xi+/DDDwE44IADABg1alTa2w4ZMgSAm266CYBzzjkn8ZoyzdLm2yX9Z3zK\nlCkpt2nUqBEABx54YJnn77vvvixHF1CmKSISQ0lkmgsXLgSgd+/eW7w2dOhQAE455ZS8xiSZadu2\nbZnfcfgs5B//+AcAn332WeK1999/H4D9998/0xAlx6JXxK+99loARowYAcA333xTZt369esnln1P\nmd/97ncA7L777gA0btw4d8FGKNMUEYmhqDPNjz/+GIBjjz0WSGYUd999d2KdCy+8MO9xSWFNnz4d\ngLVr127x2qpVq/IcjVTXVVddlViOXh2P8p/96Ou+HbxQlGmKiMSgSlNEJIaiPD33DcS+4/Py5csB\nuPTSSwG46KKLKt32p59+AqBWreD7wMxyFqcUxty5cwFYt24dAF26dEm81rlz54LEJKn5z/XVV18N\nVHxKvu222wLJm1ZuvvlmAOrWrZuPENOiTFNEJIaizDSHDRsGJG+X8l2N7rzzzkq32bx5c5l1fQNy\n//79cxanpOeHH35ILPvM4fXXXwe2PBO4/fbbgeSgC1G+O9EjjzwCwDbbbANAv379shyx5ILPMH03\nwahmzZoBcP311wNlb1goNso0RURiKKpM85NPglk17733XiB5W5T/9qnKsmXLgOQQY4sWLQLK3lZX\np06d7AUraVuxYkVi2bdZHXnkkQA8/PDDACxduhSAww47DIDu3btvsZ/nnnuuzGM/V/dBBx2U3YAl\nK3wbpu9aVL4Nc7vttkssP/nkk0Cy/IuZMk0RkRiKKtO87bbbAPj0008B+NOf/gRUfUucH0LKt5d4\ne+yxB6Dsshj4jBDguuuuK/OaHyza3xq5Zs0aoGxW6XtE+DZM76mnngKUaRYr3/ZcWcf1l19+ObFc\nChmmp0xTRCSGgmeafogwgDFjxgBwwgknAFX3x/R8VuqHhvP8lAhS3PwtcX4Q4rPOOguA9957L+W2\n//rXvwD41a9+laPopDp8zwjfz9rz7dkjR44ESrdPrTJNEZEYCp5pRtuuNmzYACT7XKZj/PjxFT5/\n2mmnZRaY5IUflGW//fardJ2zzz4bgG+//RZI9pC48cYbt9jWryv55ZxLLPvBf8sPqLLDDjsAyc/5\nd999l3jN3/Hj7+QrZsUfoYhIEVGlKSISQ8FPz/3tjgC1awfhTJ48GUiOxv6Xv/wFgHbt2gGwZMmS\nxDZ+nhjP3zbpR3OW4uQv3PmZRX13oq5duwLJMgc4/PDDgeTtmDvuuCMAjz76KKBBWYpB9FTbdwUr\nz3cnGzRoUJnfkBwX13eEb9KkSU7izAZlmiIiMRQ802zTpk1i2d8u6ecL8QN2TJs2DYD27dsDyVkp\nITl/tm9A9hcHSqFBeWvmuxb5DNNnkz5Lic4J4/kbFfzgDt6kSZO22K/kl//MVtfw4cOBZId3/5n3\nN6kUE9UsIiIxFDzTjLrmmmsAaNGiBQCDBw8Gkh3YX3zxxUq39ZlKvmakk+q54ooryjzee++9AXj6\n6aeBijPMVKKDEEthvPTSS1s8t9NOOwEwevToCreZOnVqYvmBBx4AkgPt+Fswy79fioEyTRGRGIoq\n0/T8QMJ+MAc/YMOCBQsAOPTQQxPr+g6zY8eOzWOEUl2rV68u83jAgAFAehmmn2nS3z7pNWzYMEvR\nSTb17dsXqPxGk+hNCT7T9P7v//4vZ3FlSpmmiEgMRZlpetFBSiF5pTzK9+cqf0VVikd0EGI/QIu/\n7a5jx44pt/fb+AzTT6zm26+rugVTCsefBVbGT31SapRpiojEoEpTRCSGoj49L6+iWeyit2FKcYqO\nnu9vgfS3Ppa/MOQv9kT5+YL8abq/8ONvxSzVcRm3Nn6WhSuvvBJI3rwS5Zvb/DrFSJmmiEgMJZFp\n+ob/6JwiUjp22WWXxHL5mw/++Mc/AjBv3jxgy+5EAIsXLwaSt1z6wR2OOuqo7Acr1dKtW7fE8vz5\n8wF4/PHHAXjzzTeB5IAr/nFFbr31VqC4L+wq0xQRiaEkMk3f1ci3ifh2MYA+ffoUJCapnlatWpV5\nvG7dOmDLIf6ifHbq50j3A7dI8fAzyQJMnz4dgHfeeQdIzmlfXvSsw2eYpTDgijJNEZEYSiLT9AMK\n+3lEDjnkkMRrv/71rwsSk1SPvyrqswx/Fd0P7+YHmI4O1OA7r+sqefHyM00CDBw4EIBRo0YBMGvW\nLCA5a6j/HR1OrmnTpnmJMxuUaYqIxGDRWeTiateunfPfIlsLM5vtnGtX6DjyRWVc86mM41GmKSIS\ngypNEZEYVGmKiMSgSlNEJAZVmiIiMajSFBGJQZWmiEgMqjRFRGLIqHO7ma0EPsleOCWhmXNut0IH\nkS8q45pPZRxPRpWmiMjWRqfnIiIxqNIUEYmhykrTzBqa2dzwZ4WZLY883q6qbTNlZrXN7D0zm5TG\nujdFYptnZidkeOzXzaxtinWam9nUMMZpZrZnJscslEKVsZldbmYLzGy+mT1mZtunWL8QZXxBWL5z\nzez/mdn+mRyzUApYxg3M7Gkze9/MFplZlaNHm1l/M1sZxrXIzM7N8PjjzKxHinWONrO1kf/H1Sl3\n7JxL6we4Abi8gucNqJXufmIcbzDwODApjXVvAi4Jlw8AVhK210bWqR3j2K8DbVOs8wzQJ1zuBozJ\n9v8g3z/5KmOgGfAhUCfc90TgzCIs450jy72A5wtdRqVSxuE+HwP6hsvbAfVTrN8fGBYuNwZWAY0y\nKONxQI8U6xydTh0T/anW6bmZtTCzhWb2GLAA2MfM1kRe721mD4XLe4TfNrPM7G0zOyyN/TcDjgHG\nxI3NOTef4A2wa/hNM9LM3gZuMbMdzWxsGMccM+seHq+emY0Pv90mEnyYU2kDTA2XXyH4UNUYuS5j\nYFuC/3NtoB7wWbqx5auMnXPrIg93AGrUVdNclrGZNQA6OOfGAjjnfnTOrU03NufcCuBjoGl4lvGI\nmc0AxoZnoXeFcbxnZv3DY9Yys/vDzPYloFGsf0iaMmnT3B+42znXBlhexXr3AkNdMHbd6YAvhA5m\n9kAl2wwDrqAab1Iz6wj84Jz7KnyqCXCYc24wcB3wH+dce+Ao4E4zqwNcAHztnGtNkNEcHNnfmEpO\n494lWVGeAuxsZvXjxlvkclLGzrlPgHuAT4HPgS+dc1PLr1eZPJYxZnaRmX0E3Axckm6MJSRXn+P9\ngJVhZTfHzEaZWb10gzKzFgRnJP+LxNnVOXcmMIDgPdMeOBQYZGZNgVOBfQkSmnOAjpH93Wxmv63k\ncJ3N7F0ze8HM2qSKLZPpLj5yzqUzcunRQCsLpzUgyA7qOufeAt4qv3LYBvGpc26umR0dI54rzKwv\n8A1wRuT58c65zeFyN+B4M/tz+LgO0BQ4HBgK4JybY2YL/MbOuXMqOd6lwH1m1g94FVgB/BQj3lKQ\nqzJuCJxI8AZfB0w0s97OuSdTHCffZYxz7l7gXjM7C/gL0C9FjKUmJ2VMULe0Ay4EZgPDCRKhISmO\n08fMfgNsAPo759aEx/yXc+6HcJ1uQGsz6x0+rg+0JCjjJ8L3wjIzm+536pyrrK1yJtDcObc+PCt5\nmqCCrlQmlea3keXNBKdLXvTUx4D2zrkf09xvR6CXmZ0U7mdnM3vYOXd2iu1ud84NSxGnEbRxfBRd\nIfJGSJtzbjnQM9x+Z+AU59z62Dsqbrkq427AEufcKgAze4ag3FNVmnkt43IeJ8iOa1qlmasyXgYs\n9RVy2CSSTqb+mHOuovXKl/H5zrlXoiuYWc80Y0uINhk4554Lm3p2cc6tqWybrHQ5Cmv2r82spZnV\nIqxMQi8Dg/yDyk6DIvsa7Jzb2znXHDgTeNFXmGY21LdRVdMUgm8+H4s/RXsN+H343EHAL1LtyMwa\nWfKT+BfC05WaKptlDCwFfm1mdcP/YVdgUbhtMZVxy8jD7sDiDOIqeln+HC8DvghPsyEo44Xhtheb\n2cAMQp0CnG9mtcP9tTKzugRlfEbYtrkXcESqHZlZ48jyYcCmqipMyG4/zSsJ/pj/EnzLeIOATmGD\n7ULgj2GAVbVpVuaXBKfB1TUE2MGCLisLCK4kAtwHNDSzRcC1wBy/QRXtXV2BxWb2AdAAuK2CdWqa\nrJSxc24G8CzB/3kesAkYHb5cTGV8iQXdouYStIlWehpfg2Tzc3wh8JSZvUfwJeU/I62B1RnE+CCw\nBJhrZvOBkQRnzRMIvpAXElxEfsNvUEWbZu9IGd9N2WafCpXMbZRhRjLZOXdcoWOR3FAZbx3M7N/A\nyc65TYWOpTpKptIUESkGuo1SRCQGVZoiIjGo0hQRiUGVpohIDJl0bqdRo0auefPmWQqlNMyePXuV\n24pG9VYZ13wq43gyqjSbN2/OrFnp3IFVc5jZVjUtgMq45lMZx6PTcxGRGDLKNIvBd999B0Dv3sG9\n+/vttx8Aw4ZVdIuyiEhmlGmKiMSgSlNEJIaSPz1ftiwYU+C5554DoG7dugBcf/31AOy6666FCUzS\nMm/ePACOOuooAFatWgXAzJkzE+u0a9cu/4GJVEKZpohIDCWfaZa3xx57ALDddjmdLFMy1K9fMJbv\no48+CsCmTcGANz//+c8BaNy4ccUbihSYMk0RkRhqXKZ5/PHHA7DDDjsUOBKpypQpU4AtM8z//Oc/\nAOy9996FCUyy7sADDwRg/vz5ABx55JEATJ2a9lx6RUWZpohIDCWfaY4cORKA7bffHoBLLqmJs6zW\nHOeffz4AX3zxBQCtWrUCYPLkyUBwS5/UDP6zuGDBgjLPd+nSpRDhZI0yTRGRGEoy01y6dGlieezY\nsQDUqxfMQ+/bxqQ4jR8/HoCffgqmiP/nP/8JKMOsSS6++GIA7rvvPgD8lDrHHHMMANddd12l2/79\n738H4E9/+lOl65x22mkAjB49utJ1ckmZpohIDCWZab788suJ5TVrgimKb7tta5hBtzT94x//SCz7\n8jrjjGCm1NatW1e4zWeffZZYLn+V1d89tOeee2Y1Tqm+RYsWJZbHjRsHwObNm4HkWcRJJ50EwDbb\nbLPF9g8++CAAF110EQA//vhjmdf9FXeA008/PUtRV48yTRGRGFRpiojEUFKn519++SUAQ4cOTTzn\nb7fr27dvIUKSNKxbty6x7C8AdejQAYDatYO3oO9y9Le//Q2A//3vf4lt/KAsnu/47m9gaNSoUeI1\nf3rnB/nYd999s/RXSFV69OiRWP7qq6+A5P/+3//+N7BlU8wDDzyQWL700kuB5Gn5qFGjADjxxBMB\nqF+/fmJdf9G3UJRpiojEUFKZps9GFi9enHjOdz/wA3V8//33QPL2vJ122imfIUoFRowYscVz/kLQ\n888/DyTLccOGDSn3Vz7zjL4fZsyYAUCbNm3K7F9dmnLjgw8+AJJngVH9+/cHKr/Y57ubAfzwww8A\n7LLLLgAcdNBBADRp0iR7wWaJMk0RkRhKItP89ttvAXjkkUe2eG3w4MFAMrP0cwX52/ReeOGFxLoN\nGjTIaZxS1sMPPwzAxx9/vMVrd9xxBwDPPPMMkMwwO3fuDMDll1+eWHevvfaq8jhPPfVUYvnxxx8H\nYOHChQAMHz4cgDvvvDN2/JKan4vLdyUDOPnkk4GyZRj10EMPAfDmm29u8dpdd90FQPv27bMaZzYp\n0xQRiaEkMs27774bSHZyjnZ09VdJX3zxRQCeffbZMtt++umniWVlmvnls31/xTzKl6nn26T9bbF+\nVtF0HHLIIYnlE044AUi+R/ytfIcddhiQbDuVzHz00UdAsiN7lL+O4AcCnzZtGgCzZ88G4IYbbgCS\n1x8g2SOiU6dOuQk4i5RpiojEUNSZph+01PfZ8s4999zEsp+I68ILLyyzjr/qpmkTitvuu+8OwJNP\nPgnEyzAr0rJlSyCZufps17efKdPMDj+R4TfffLPFa8uXLwfguOOOA+C1114DymaW5fkeEf5Mwffh\n7Nq1a5Yizh5lmiIiMRRVprlx40YgOeXBeeedByS/ubxevXollv20Cb6/mOfvNPFX1SF5hdYPWCyF\ns9tuuwEwcOBAAI444ois7NcP4uEzV9+26YcR01X03PNtmNXx4YcfAjBo0CAgOZCxf58UA2WaIiIx\nqNIUEYmh4Kfna9euTSz37NkTSJ3epzPTpO9qFJ3VsGnTpkCyc60fSVryr/zpc7a1aNEiJ/uV9P3q\nV78CkgNsvPvuu0Dy4pGfpRKSY2zOnTsXSN4a+9e//hVI1g2QvMhXKMo0RURiKFim6TPM6K1W5TPM\nHXfcscw6O++8MwBPPPFEYp2ZM2emfUx/ceidd94BlGkW0v7775/T/fuuaJIbPvPzg3FEZ1M4+uij\ngeQtsf7M8NBDDwVg1qxZANxzzz2JbXxWOmnSJCA5VNznn38OJOcOArjmmmuy+afEpkxTRCSGvGea\nvluRzx59+2JFhgwZAsBll10GJIePuvHGG7dY18yA5JBSfh6Z7t27J9bx32Y+Y5Wax7+/brnlljLP\nn3LKKYUIp8Zq1qxZmd/HHntsym2qaov0gwyfffbZQHJIvwkTJgBlByxWpikiUkLynmkuWbIEqDrD\n/MMf/gAk50/2/BBgX3/99Rbb+Fu2okPBSWEdf/zxAFx//fWJ5/wNBo899hgA/fr1A2DXXXfNyjH9\nkHB+fnU/eERV82hLflxwwQVAcvqLqvghHn2muXr16sRrfnCebt26ZTvEtCjTFBGJIe+ZZnRStPL8\nREy+zbL8/MgrV67cYpuzzjoLgDFjxmQrRMkS3w8v2p7oBwn2g0f7Mw6fhfjb5XxPh6r4W2T9rXeQ\nnEbDO/XUU4Hk9BdSPKIDF5fXtm1bINmDZv369YnXopPuFYIyTRGRGFRpiojEkLfTc9+QW74De3TE\nId9p3XdjKM+PdlSnTp3Ec/50rFYt1f/FKnoRxncNmzhxIpAcncrPV/7WW28BZecyL893J/OzGfqL\nSgANGzYEknPX5Oo2TYnvgAMOAJK3uPpRzCA5du61114LwM9+9jMgOd9QtIwLTTWNiEgMecs0fadj\n30Hdi3Y/6NChQ5X78BcP/MUfgIMPPjhbIUqORMvo0UcfBZJl6S8MVpQ1ViZ6+x2UHZ3fzz8zYMCA\n6gcsOeEHz/EZ5lVXXZV4zc/l5M8M/I0ovotaMVGmKSISQ94yTZ8N+DlbqsPP++N/S+ny3ZF85umz\njltvvRWoOuP0GYvPJqNdmvwAElK8/G3RL730UuI5P1tDly5dgOTsoRXNjV5oyjRFRGIo+CDEIpDs\nfO4zT/9baq7ordS+bfvee+8FijPD9JRpiojEoExTRApir732SixfeeWVAGzevBlIXk33otNd+MGM\nC0WZpohIDMo0RaTg/Hz1w4cPL/O7GCnTFBGJQZWmiEgMqjRFRGJQpSkiEoMqTRGRGFRpiojEYM65\n6m9sthL4JHvhlIRmzrndCh1EvqiMaz6VcTwZVZoiIlsbnZ6LiMSgSlNEJAZVmiIiMVRZaZpZQzOb\nG/6sMLPlkcfb5TIwM6ttZu+Z2aQ01r0pEts8Mzshw2O/bmZtU6wz2MwWmdm7ZvaSme2TyTELpVBl\nbGYNzOxpM3s//D+2T7F+fzNbGca1yMzOzfD448ysR4p1zMzuN7MPw/dile+JYlXAMr7czBaY2Xwz\ne8zMtk+xfiE+x9H31VwzOyfVfqscsMM5txpoG+78BmC9c+6Ocgc1ggtKm1MdLKbLgPlAvTTXv905\nN8zMDgCmmdnuLnKVy8xqO+c2ZTG+WcBw59z3ZnYhcBvQJ4v7z4sClvFw4FnnXK/wg1s3jW0ec85d\nYmaNgflm9qxzblUkzmyXcXdgH+dcCzPrDIwAOmVx/3lRiDI2s2bAQOAAYAMwATgNGJdi03x/jiF8\nX6W7crVOz82shZktNLPHgAXAPma2JvJ6bzN7KFzeI8woZpnZ22Z2WBr7bwYcA4yJG5tzbj5gwK5h\nNjHSzN4GbjGzHc1sbBjHHDPrHh6vnpmNDzOYiUCdqo4RHmeqc+778OGbwN5xYy1muSxjM2sAdHDO\njQVwzv3onFubbmzOuRXAx0DTMDt5xMxmAGMtOEO5K4zjPTPrHx6zVpg1vm9mLwGVT6yedDLwSHjM\n14HGZlZjuiLl+nMMbEvwWapNkPx8lm5s+focV0cmbZr7A3c759oAy6tY715gqHOuHXA64Auhg5k9\nUMk2w4ArgNj9ocysI/CDc+6r8KkmwGHOucHAdcB/nHPtgaOAO82sDnAB8LVzrjVwE3BwZH9j0jgt\n6wdMjhtrCchVGe8HrAwruzlmNsrM0j2jwMxaAM2A/0Xi7OqcOxMYAHwZlvGhwCAzawqcCuwLtAHO\nATpG9nezmf22gkPtBXwaebwsfK4myUkZO+c+Ae4h+P99TlAmU9MNKs+f49PDL9h/mlnK8s1kPM2P\nnHOz0ljvaKBVkP0DwTdHXefcW8Bb5Ve2oJ3pU+fcXDM7OkY8V5hZX+Ab4IzI8+MjpxzdgOPN7M/h\n4zpAU+BwYCiAc26OmS3wGzvnqmzjCI95IHBRjFhLRU7KmOB91w64EJhNcKp+BTAkxXH6mNlvCE73\n+jvn1oTH/Jdz7odwnW5AazPrHT6uD7QkKOMnwvfCMjOb7nfqnLs6jb+xpsrV57ghcCLBF9U6YKKZ\n9XbOPZniOPn+HE8CHnXObTCzQQRnt92qCjCTSvPbyPJmglTai6bFBrR3zv2Y5n47Ar3M7KRwPzub\n2cPOubNTbHe7c25YijgN6OGc+yi6QuSNEIuZHUfwYT8ixt9XSnJVxsuApf7DGp5KpdOmVFnbU/ky\nPt8590p0BTPrSXzLgX0Iml8gaIKpKhsrRbkq427AEt/mbGbPEHy2U1Waef0cR9vEgVEEGWqVstLl\nKPwG+NrMWppZLSD6Bn0ZGOQfpDrVdc4Nds7t7ZxrDpwJvOgrTDMb6tsvqmkKQXbjY/Hp+2vA78Pn\nDgJ+kWpHZtaO4MLASeX+8TVSlst4GfBFeJoN0BVYGG57sZkNzCDUKcD5ZlY73F8rM6tLUMZnhG2b\newFHpLGvZ4Gzwv10Br5wzq3MILails0yBpYCvzazuhbUZl2BReG2xfQ5bhJ52IOgbbdK2eyneSXB\nH/NfgkzCGwR0CtsMFgJ/hJRtmpX5JbAigxiHADtY0J1hAXBD+Px9QEMzWwRcC8zxG1TRFnIHsAPB\nacfc8Ju0pstmGV8IPGVm7xG8uW8Ln28NrM4gxgeBJcBcM5sPjCQ4o5pA8EFeSHAK9obfoIo2zeeA\n5Wb2UbifQRWsU9NkpYydczMIvnTmAPOATcDo8OVi+hxfZkGXqHcJrvb3S3Xwkrn3PPy2muycO67Q\nsUjumNm/gZNz0K1EikBN+ByXTKUpIlIMdBuliEgMqjRFRGJQpSkiEkMm/TRp1KiRa968eZZCKQ2z\nZ89etTWN6q0yrvlUxvFkVGk2b96cWbPSuZmg5jCzrWpaAJVxzacyjken5yIiMajSFBGJQZWmiEgM\nqjRFRGJQpSkiEoMqTRGRGFRpiojEoEpTRCQGVZoiIjGo0hQRiUGVpohIDBnde55rM2fOBGDUqFEA\nfPDBBwD87Gc/S6zTq1cvADp06ADAbrttNeMslLQ1a4LptZcsWQLA448/Xub1YcOSc2tVNmFW48aN\nAXjjjcTMFTRr1iyrcUr2TJ8+HYADDzwQgIYNGyZemzdvHgDdugUTQf7hD38AYOjQoXmMMD3KNEVE\nYijKTPPzzz8H4NRTTwVg6dKlANSuHYT72muvJdYdM2YMAO3atQPgrrvuAqBLly75CVZiGTduHAC3\n3HILAIsXL65wvWh2edBBBwGwceNGABYtWgTAF198AcCKFck5upRpFh9fPpdeeikAxx57LAC33XZb\nYp2///3vZdYdP348oExTRKTkFWWmWatWUJd/8803AOyyyy4APPHEE0CyPQzgqquuAkiMB/jss88C\nyjSLSbS98rzzzgPgu+++A6BBgwZAsm3aZ5WHH354YhufPW7aFExQuc8++wDw/fffb7F/37YtxePt\nt98GYO7cuQAMHJjJtPaFp0xTRCQGVZoiIjEU5en5HnvsASRPsf0pt78Q1Lt378S6nTt3BuD+++8H\nYOTIkQB07NgRgJ49e+YhYqmIPwV/6KGHEs8dcsghAFxzzTUAdOrUCYC6deum3J8/HS/fBem0007L\nPFjJuh9//BGA22+/vczz/nNcqpRpiojEUNRVfp8+fYBkptmvXz8AbrjhhsQ6/rV33nkHgG+//bbM\nbymcevXqATB16tSs7O/OO+8Ekhlsy5YtAWjdunVW9i/ZNWLECABef/11ALbddlsg+bkuVco0RURi\nKOpM02fwxqjAAAAJdUlEQVQqvg3Ld3I/99xzt1i3Tp06QLL97Mwzz8xHiJIH/nbav/3tb2We992X\norfjSfHwt8h6vuuY/6yWKmWaIiIxFHWmeeKJJwIwYcIEINk59qabbkqs45wD4NBDDwXgrLPOymeI\nkiObN29OLE+ZMgVItmXWr18fgCOPPDL/gUlK/lZmf2ukv1p+7bXXFiymbFKmKSISQ1Fnmp6/xc7/\njt7o7wdxUIZZs4wePTqxfP3115d5zZf/L3/5y7zGJOl59dVXgeRtr23btgWSfadLnTJNEZEYSiLT\nTIfvAyY1w/PPP7/Fc02bNgXg7LPPznc4ksKXX36ZWJ42bVqZ1/y1iPbt2wPJO/6i/DB/3tq1awF4\n7rnnAOjevXv2gs2QMk0RkRhUaYqIxFBSp+erV68Gkt2MotTBuWaYM2cOkDwtg+TNDVdccQUA22+/\nff4DkypFB1HZZpttANh9992B5Kn7119/XeZ3Vfw6/oKgTs9FREpUSWWaPvvwXRkgmXX4jvBSmvwA\nK34wlujZRNeuXQE4//zz8x6XpCc6C6y/7dVfnF25cmWZdcs/Bnj44YcBeOqppwBo0qQJAHfccUf2\ng82QMk0RkRhKKtOMzjroVTR4h5QeP6uo72oUHZT4nHPOKUhMUj0tWrQo8zidGUInT55c5rE/gyy/\nr2KgTFNEJIaSyDT9VfMHH3xwi9c01UFp88OHXX311WWe91fKAX7/+9/nNSYpvGLMMD1lmiIiMZRE\npun7eX388cdbvOaHCZPS4q+O33rrrQCsX7++zOvqDbF1O/nkkwsdQqWUaYqIxKBKU0QkhpI4PZea\nZ+LEiUCyU7PXt29fIDkSv2ydunTpUugQKqVMU0QkBmWaUhAffPBBhc9fc801Kbf1t9qdccYZWY1J\nCmfWrFllHvvBPoqRMk0RkRiUaUpBlM8s/EyFfnT2DRs2JF57+umnAbjxxhsBGD58eD5ClDwq/36o\nV69egSJJTZmmiEgMyjSlIN54440yj7/66isAFi5cCECfPn0Sr33yySdA8lbLI444Ih8hSgH4OdKL\nmTJNEZEYir9aJzl7Xbt27YCy7R+dOnUCoHPnzgC89NJLeY5OqqNnz55AchCWESNGlPkdHYR4wIAB\nAAwePDifIUoB+AF4ivn2aGWaIiIxlESm2aBBAwBeeOEFAPbcc8/Ea/4q6wUXXJD/wKTahgwZAsCM\nGTMAmD9/PgBt27YFyvbXPPbYY/McneSbP1O86KKLChxJaso0RURiUKUpIhJDSZyee37Gu40bNxY4\nEsmUL8t33323wJFIMZg6dWqhQ0ibMk0RkRhUaYqIxKBKU0QkBot2Io69sdlK4JPshVMSmjnndit0\nEPmiMq75VMbxZFRpiohsbXR6LiISgypNEZEYqqw0zayhmc0Nf1aY2fLI4+1yFZSZXWZmC8KfC9NY\nv7+ZrQzjWmRm52Z4/HFm1iPFOr8wszfMbIOZXZLJ8QqpEGVsZs3MbLqZLQzLOOU9sAUq4z9H/hcL\nzGyTmRXvSBKVKNTnODx2bTN7z8wmpbHuTZHY5pnZCRke+3Uza5tineZmNjWMcZqZ7VnV+kAwmkw6\nP8ANwOUVPG9ArXT3k8Zx2gLvAnWBbYFpwL4ptukPDAuXGwOrgEbl1qkdI4ZxQI8U6+wBtANuAy7J\n1t9fyJ88lvGeQNtweWfgI+DnxVbG5dbvCbxY6DIqlTKO7Hcw8DgwKY11b/KfJeAAYCXhdZdqlvHr\n/n1WxTrPAH3C5W7AmFT7rdbpuZm1CLOEx4AFwD5mtibyem8zeyhc3sPMnjazWWb2tpkdlmL3rYE3\nnXPfO+c2Aq+Fb9i0OOdWAB8DTcNvrkfMbAYwNvzWuyuM4z0z6x/GWMvM7jez983sJaBRGsf5wjk3\nC9iUbmylJJdl7Jz7zDk3N1xeB7wP7JVubPkq43J+BzwRc5uiluPPMWbWDDgGGBM3NufcfIKKfNfw\nrGCkmb0N3GJmO5rZ2DCOOWbWPTxePTMbH56JTATqpHGoNoC/HekVoFeqDTJp09wfuNs51wZYXsV6\n9wJDnXPtgNMBXwgdzOyBCtafBxxhZg3MbAfgeGCfdIMysxZAM+B/kTi7OufOBAYAXzrn2gOHAoPM\nrClwKrAvwT/wHKBjZH83m9lv0z1+DZOrMk4ws/0IsoqZ6QaV7zI2sx2Bo4Gn042xhOSyjIcBVwCx\nu+iYWUfgB+fcV+FTTYDDnHODgeuA/4RlfBRwp5nVAS4AvnbOtSbIWg+O7G9MJafq75KsKE8Bdk7V\nBJPJvecfhZlWKkcDrczMP97VzOo6594C3iq/snNuvpndBbwMrAfmAD+lcZw+ZvYbYAPQ3zm3Jjzm\nv5xzP4TrdANam1nv8HF9oCVwOPCEc24zsMzMpkfiuTqNY9dUOSljz8x2BiYCFzrn1qdxnEKV8cnA\nq865tWnEWGpyUsYWtBd/6pyba2ZHx4jnCjPrC3wDROdoHh+WHQRlfLyZ/Tl8XAdoSlDGQwGcc3PM\nbIHf2Dl3TiXHuxS4z8z6Aa8CK0hR32RSaX4bWd5MkEp70bTYgPbOuR/T3bFzbhQwCsDMhgIfprHZ\nY865ii7IROM04Hzn3CvRFcws7dP/rUzOytiCCxBPE7QhPZvmZoUq497AoxlsX8xyVcYdgV5mdlK4\nn53N7GHn3NkptrvdOTcsRZxG0B79UXSFSIWeNufccsLmv/BL/JRUX+BZ6XIUfgN8bWYtzawWZdsg\nXwYG+QeVpMhlmNnu4e/mwEnAk+Hji81sYAahTgHON7Pa4f5amVldgnbTM8J2r70AzdxVTjbL2IJ3\n91hgrnPu3nKvFVUZm9muBBXAcxnEVBKyWcbOucHOub2dc82BMwkuop0dbjvUt0NW0xQg0avGzPxp\n+GvA78PnDgJ+kWpHZtbIkrXtXwibHaqSzX6aVxL8Mf8FlkWeHwR0ChvlFwJ/DIOtqi1kUrjuJGBg\neLEAgotEqzOI8UFgCTDXzOYDIwmy7QnAUmAhQaN1YqrEytq7zGxvM1sGXATcYGbLzKx4J2vOjmyV\n8REEF1aOsWTXFz88e9GUcegUYLJz7vsMYiol2fwcV+aXBKfB1TUE2MGCbkkLCHoEANwHNDSzRcC1\nBE17hHFW1qbZFVhsZh8ADQh6w1SppG6jNLN/Ayc752rkFWtRGdd0YVY32Tl3XKFjqa6SqjRFRApN\nt1GKiMSgSlNEJAZVmiIiMajSFBGJQZWmiEgMqjRFRGJQpSkiEsP/Bzhh8lgFzPWKAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc310343470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save & Load Model\n",
    "\n",
    "NOTE: You need to install `h5py` for this to work!\n",
    "\n",
    "Tutorial #04 was about saving and restoring the weights of a model using native TensorFlow code. It was an absolutely horrible API! Fortunately, Keras makes this very easy.\n",
    "\n",
    "This is the file-path where we want to save the Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_model = 'model.keras'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving a Keras model with the trained weights is then just a single function call, as it should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model2.save(path_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the model from memory so we are sure it is no longer used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "del model2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import this Keras function for loading the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.models import load_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the model is then just a single function-call, as it should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model3 = load_model(path_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use the model again e.g. to make predictions. We get the first 9 images from the test-set and their true class-numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "images = data.test.images[0:9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_true = data.test.cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use the restored model to predict the class-numbers for those images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model3.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the class-numbers as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the images with their true and predicted class-numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViI\nsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL\n19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpT\nRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fn\nc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF98\n8QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/Pe\nTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JL\naDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnD\nqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqo\nSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOje\nvXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfA\nKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U\n78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih5\n8/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5Xhrd\nunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzm\nGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH75\n5QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlW\nAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8\neDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcA\nmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oS\nUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2m\nh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQp\nsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtX\nL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV\n6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwx\nScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qE\nMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJ\njzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueee\ne/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/J\nfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv\n1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGCl\nYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim2\n0L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXM\nP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f\n//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUp\nIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/v\nYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHK\nLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591\nE1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk5\n7l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTu\ny3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRi\nlrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tn\ncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj7\n7mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHT\ngEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA\n6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTc\nD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7\nyhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7\n+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRD\nzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+Y\nqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHj\nGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0ef\nVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD9\n3T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId\n5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGR\nBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZR\npikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7X\nY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0\nqMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrr\nr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LL\nwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBF\nZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpv\nL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377\nbZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/nj\nQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWent\nvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCN\nCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdm\nzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4p\nrngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+F\nR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzV\nq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD\n8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDt\nJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQ\ntuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZg\nn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOg\niyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIi\nCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRT\nT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh\n9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMA\neP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxT\nRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCO\ncPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1\nukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcX\nL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqaw\nOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1\nzitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSki\nkoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0R\nkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvV\nzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m\n9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l\n73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OA\nvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7p\nFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7\nv+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+\n4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9a\nKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHs\nYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7Hxj\nG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7\n+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX\n87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3ef\nZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0u\nLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4\nDrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd\n5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eyg\np5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3\ndvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XG\nx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B31\n5roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYe\neI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqj\nap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSp\nk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajS\nFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801\nAOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZ\nZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu\n0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73\nvwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBM\nU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDu\nXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCC\nXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJE\naYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrr\nA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADA\nhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70A\nmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuX\np9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5\n+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+\nqvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcD\nmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFD\nBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaj\nu+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcf\nbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4\nYxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7Dzzjtn\nvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3N\nkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz2\n2GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa8\n4qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1w\ne77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOB\nzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccd\nAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7+\n+uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRh\nFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosy\nTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM44\n4wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAA\ntttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZ\nuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoK\nQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6c\nCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDp\np58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJ\noKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJW\nE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8\nF8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZP\nZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWz\nw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/D\nF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X\n1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCP\nmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY\n4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9\nPmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j7\n18CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2\nZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72B\nL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXx\nq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKm\nKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zu\nxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YL\nwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEe\nZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1\nSl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeF\nu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+g\niTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYP\nW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+\nFr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2br\nkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerG\nVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bp\nyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Ch\nmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYp\nHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoi\nIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc310396ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            cls_pred=cls_pred,\n",
    "            cls_true=cls_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of Layer Weights and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting convolutional weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_conv_weights(weights, input_channel=0):\n",
    "    # Get the lowest and highest values for the weights.\n",
    "    # This is used to correct the colour intensity across\n",
    "    # the images so they can be compared with each other.\n",
    "    w_min = np.min(weights)\n",
    "    w_max = np.max(weights)\n",
    "\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = weights.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot all the filter-weights.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the valid filter-weights.\n",
    "        if i<num_filters:\n",
    "            # Get the weights for the i'th filter of the input channel.\n",
    "            # See new_conv_layer() for details on the format\n",
    "            # of this 4-dim tensor.\n",
    "            img = weights[:, :, input_channel, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, vmin=w_min, vmax=w_max,\n",
    "                      interpolation='nearest', cmap='seismic')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Layers\n",
    "\n",
    "Keras has a simple way of listing the layers in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "reshape_2 (Reshape)          (None, 28, 28, 1)         0         \n",
      "_________________________________________________________________\n",
      "layer_conv1 (Conv2D)         (None, 28, 28, 16)        416       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 14, 14, 16)        0         \n",
      "_________________________________________________________________\n",
      "layer_conv2 (Conv2D)         (None, 14, 14, 36)        14436     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 7, 7, 36)          0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 1764)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               225920    \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 242,062\n",
      "Trainable params: 242,062\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We count the indices to get the layers we want.\n",
    "\n",
    "The input-layer has index 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_input = model3.layers[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first convolutional layer has index 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.layers.convolutional.Conv2D at 0x7fc30c26cb70>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_conv1 = model3.layers[2]\n",
    "layer_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second convolutional layer has index 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_conv2 = model3.layers[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Weights\n",
    "\n",
    "Now that we have the layers we can easily get their weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_conv1 = layer_conv1.get_weights()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a 4-rank tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 5, 1, 16)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_conv1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the weights using the helper-function from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEJVJREFUeJzt3X10lPWZxvH74S1DMhsSMkAIIQxoAUERJXVZRVarRndL\nUYQqLNhTdKtlZfdUagHFXcR1rSge2q0LvhTqtlQQlWhLV0G0HuAAaxMsVCggLUEg8hII5KWEF/Pb\nP7rb/9jrnlPjMxy/n7+vc9/jw+Ry5pzf80wUQjAAwLm1i/sFAEC2oygBQKAoAUCgKAFAoCgBQKAo\nAUCgKAFAoCgBQKAoAUDokEm4sDAVSkrSMte5wxmZqd561rWzoKCzzBw/ftIxqdZCqI9cS2OQSKRC\nMpmWudOn9az+X/DdbfX7PfpylNRXy8wBM6sPIWuvbUEUhWJHznPVOjp35jgyUbHnVZlVHzxYF0Lo\n5lz9mUvl5IR0Xp4OJhI6c+yYb2lRkYzU1dbKzFEza3S8dzMqypKStC1dWiVzQ1L6BUa9jrh2Xnvt\npTJTWfkbx6Txrn1xSSbTNnq0vrb79+tZq1c62tTMbpvUSWYefVn3362ubfEpNrPnHLlWR6a7c2c/\nRyYxebJrVvTd7+51ro1FOi/Pqq6/XgcHDtSZZct8SydNkpHFs2fLzCO+bXz1BgCFogQAgaIEAIGi\nBACBogQAgaIEAIGiBAAho3OUe/eaffObOvf1r5fIzJgxOmNmVln5rMx89av3yMyaNfrgepxOnTLb\nvVvn1j69VYf2J107H3pIn/br/9AWmUlMmODaF5dPzKzBkRu1cqUOPfCAa+fWJfrfacgmz+nO7PdJ\nfb01vPyyzL37uj7SP7qlxbd06VIZubOgQGYWNDa61vGJEgAEihIABIoSAASKEgAEihIABIoSAASK\nEgAEihIAhIwOnPdsrrYHN+oHuf7tRj1r5Vd8T+EOU/SBZ3ujr4yUN+uHCcepqanZ1q3bJHNDJg2X\nGeeDs103D+waqx+cXO9bF5su3bvbqIkTZa7/fV+WmeJinTEzs6k6sm7dDb5ZWe6MmXn+uvrd7HgI\nfjrt2vn7t/fofcV/0INGjHDt4xMlAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBnd\nmdPl0ktt1DvvyFxTUZHMeJ/4PrfPApkZW6NP/J/yrYtRBzNLydTTT+tJDz/s25gcq6/bBseck751\nsak+nGvRfH2H0ZgxelZlpedHJczM9B1lvXtf7Zq0b59zZUy22WC7yJbL3LJlg2TmP9717XypXGeO\n/f3DOlTru2OPT5QAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBkdOD8SH0He+6VrjI30DFr\ntVX4lj7+Kxn5d8eYE75tMWpvZvkyNXLgYZl57Zc9XBv1EWEz/a+d4ZsoFqfNbL9MNTXpST176n8j\nM7MXXtCHyU/f6PhpBDP7iisVpzoz+0+ZGj++k2OW86c2rFQmrlz/hMzsaNI30JjxiRIAJIoSAASK\nEgAEihIABIoSAASKEgAEihIABIoSAASKEgCEKITgD0fRETPb23Yvp031CSF0i/tFnAvXtu2c59fW\njOvbllzXNqOiBIDPI756A4BAUQKAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKAkNHPnaRSqVBWlpa5\ndoc+lpmGvJ6unfm1O2SmurmvY9IBC+GY70dKYtC5cyrk56dlruBwtcwkLx/m2hk261nN/fWsgwdr\n7MSJuqy9toWFqdCrV1rmfvtbPau1tcW1s2/fhMzs2XPaNcvsN3XZfGdObm4qdOmSlrleecf1sPbt\nXTsbdu2SmfyLLpKZmtpaqzt+XL53MyrKsrK0rV9fJXO58x6RmdXD/8W1s2L2X8lMtOlFx6TRrn1x\nyc9P28SJ+tqOnq/76MqNeo6Z2ekcPeu9hXrWlCnlrn1x6dUrbcuX6/+O4cP1rMbG7a6djz46SGYm\nTvTe9ZfO6tsDu3RJ2+TJ+vo+Nvxnelgy6dq5+rrrZKZiyRKZKZ80ybWPr94AIFCUACBQlAAgUJQA\nIFCUACBQlAAgUJQAIGR0jvL4+9W2Ik+fvSt1zCo/6jtHGW36L0dqmiOjD8HHqbjY7P77da7kjYEy\n07XYt7PeVunQdVsck076FsYk0VRngzYtlrklS+6UmZtv9lwPs3Xr9DnK3r37uGbt2+eKxaZX3nF7\nrHyFzN36wq0yM63Sd99CRTotM5uH6Zsl/uDaxidKAJAoSgAQKEoAEChKABAoSgAQKEoAEChKABAo\nSgAQohCCOzwgisJCR+5LjkzFDb69F1ygM8884zlMfpOFsCVrn8JdnpMTqkpKdHDnThnZnpPj2tnP\nkels2xyp2yyED7L22g6JorDSketwQL8nSzbpg9VmZrvGjpWZlU/5/ga+/e2oOoSQtU9HHhpFYY0j\nl1q0SIcKClw7NxTrw+sJ/ZB5mzSp3LZvr5LvXT5RAoBAUQKAQFECgEBRAoBAUQKAQFECgEBRAoBA\nUQKAQFECgJDRT0G0mNluR+60I7N65juZrP5/LZxzscyUV2TtjSN/VFJiNmeOjEU5e2QmvPqqb+cH\nH+hZBfqei/J5Db59MTlrZsccuaHHt+vQbs9fgFnScZdPQ68sf086bbFC62YVMjdhjf6pjaVLm107\nwxh9Z87ab+m7qFpbXev4RAkACkUJAAJFCQACRQkAAkUJAAJFCQACRQkAAkUJAEJGB87zzfczD2WO\nzOIazySzK+7Sh3IvnjBBD9q3z7UvNrm5ZkOHOoJ7ZeKlM/owrpnZ7Utn6dDBgzrT2OjaF5fczp1t\n6IABMvfrwYNlJt+584OB02Xm4aNHXbPmFBU5t2a3ykqdCRO+4ZrVslQPGznzPZlJRr4D7nyiBACB\nogQAgaIEAIGiBACBogQAgaIEAIGiBACBogQAgaIEACEKQT+y/k/hKDpinltDslOfEEK3uF/EuXBt\n2855fm3NuL5tyXVtMypKAPg84qs3AAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgZ/RREKpEI6WRS\n5qqP6rHJZKlrZ1PTKZmJohyZCaHGQqjTvysRk/woCt0dOc+p179w7uzg+Lc800//hML+/TV29Gj2\nXtsoSoUoSsvc5e236GE9e/7Zr+f/VO/LcyZ31GXzgfNOnVIhkUjLXP++Z/SwujrXzupa/TdfVtZV\nZo4erbGmJv3ezago08mkVY0eLXPRj/QLvOyyea6d69btlpmcnAtl5tSpcte+uHQ3M88VOevIXOPc\nmbrsMpk59PJamamoyO5rG0Vpy8mpkrmqAkcJzpjx57+g/xVNvcKZvCKr73pJJNI2fLi+vqtfqNXD\nfvhD185odj+ZmTFjkszMnet77/LVGwAEihIABIoSAASKEgAEihIABIoSAASKEgAEihIAhIwOnFcf\n7W3Rj74nc+G/d+hhrz3oWzrvFhnZmtBjJkzwrYtLQV6e3XLJJTroOCT+xMKFrp3TCwpkpscdFTLT\ncc8u1764hBCspUXfFfLcnI9l5u7d031LHXeYrFp1r2vUjTf6Vsalf2O1rX5L35h15Th9X9nGjcNc\nO5999suu3KeFT5QAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBkdOB8sP3allsXmTv2l3pW\n1+9/37f0rH6m95CpI2Wm80c7ffviUlpqNneuziX06frpa9a4Vr7485/LTGqVPiTcMDW7n3A+rMcB\nq/raLJmruefJT23n1i36ut146f2f2r5YDRhgtnixjF3/ph61Y4fvIPndC/WNF8fefl9mFixwreMT\nJQAoFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgZ3ZkTmZnjVxes1TOs3Hk3x/r1OlNa\nqjPbtvn2xaR6Z6tFf90icxMm6LuQHvzwQ9fOlCNTkVgrM/ntmlz74lJ/6JC98qS+6+ZKz7C333bt\nHHJ9d0dK/8zJ+WD7R0m7fKq+epsLviQzM+t/6dq5oV5nzhbpn6fQf3F/xCdKABAoSgAQKEoAEChK\nABAoSgAQKEoAEChKABAoSgAQMjpwnlNQYP2uvVbmospH9bCrBrh2hhselpmmt96SGdch+FgdNrMf\nyNTSpWdk5jHnxopXX9Wh4mKd6ZDR2+gzF5lZJ0eu6nX98w3Hr9OHmM3MvnbggMwsW1fimjV+/POu\nXFwGnay2qvcd12XoUBn5nnPn5Y7M9qf0v2fzfN+NL3yiBACBogQAgaIEAIGiBACBogQAgaIEAIGi\nBACBogQAgaIEACEKQZ9e/1M4io6Y2d62ezltqk8IoVvcL+JcuLZt5zy/tmZc37bkurYZFSUAfB7x\n1RsABIoSAASKEgAEihIABIoSAASKEgAEihIAhIye4V9UlAqlpWmZ61jrOHt64oRvqeNnBj7sOEhm\nTp6ssdOn63zP8Y9B1ygKvR25jp5h3bv7lhYWysjhnTtl5piZNYWQtdc2ihLBLOlI6jPdZWV5rp2F\nH1XLTIfBg12zqrdtq8vmA+epVCr06ZOWuc2bTzmm5bh2JhI609LysWPScQuhWb53MyrK0tK0rV5d\nJXM9/vluPezNN31Li4pk5G+K9WvasMH32xhx6W1mbzhyjl+wsXYTJ/qWjhsnI09fdZXMPOHbFqOk\nmd3syH1DJmbNGu7aOO4e/f+NritWuGZFAwZk9V0vffqkbeNG/TeYk7PbMa2va2c63V5mduz4N8ek\nBa59fPUGAIGiBACBogQAgaIEAIGiBACBogQAgaIEACGjc5QnTpj94hc61/3552Vm1He+41vapYuM\nvLFqpMyUt+qD03FqNLO1jtx4x0nb5fPnu3beNnOmzKQcczJ6E8WgrCxtM2Yskrl/SC3XwwoafEun\nTJGRhgEDfLOyXEOD2Zo1OnfCviAz+V/8omtn9KsXZaa5eZbMjBhR6drHJ0oAEChKABAoSgAQKEoA\nEChKABAoSgAQKEoAEChKABAyOiu8d+8ndtdd9TL3kmPW2SefdO3ssG2bzEQPjXJM+jvXvrjsbTfM\n7s7TDz899riedc01vp1Rj09kJtx3n8zM++lPfQtj0tRktmmTzpX85HaZuWW478G9duCAjFzYLfhm\nHcnah8ebmVlurtmwYTr3jGPWuGXvuXY+9ZrO5E69U2bafVTj2scnSgAQKEoAEChKABAoSgAQKEoA\nEChKABAoSgAQKEoAEChKABAyujOnrKy9zZhRKHMX3utYfPXVvqWP61tRXn/9xzIzbVpn376YtLbW\nWmPjbJm7995WmZk8+V9dO++4o73MRPM9s9517YtL364n7Me3698wee0njmGXXOLaWee4FejwP/6T\na1b0A1csNh3PnrQeh7bK3IWOWbUX+O5COuvI1H1Kc8z4RAkAEkUJAAJFCQACRQkAAkUJAAJFCQAC\nRQkAAkUJAEIUgvNx9GZ2cRSFVxy5gUOH6tCiRb6lAwfKSFNensyMNLPNIWTtM/U7dSoPqZT+KYhk\nUs8aP96385HS53SooEBGyh94wKp+97usvbZDoij8zJGbNkb/Laz41lrXztYRI2WmffstrllmQ6tD\nCOXO8GeuY8fyUFio37sjRuhZK5oqfEtvuknPSk+TmenTy2337ir53uUTJQAIFCUACBQlAAgUJQAI\nFCUACBQlAAgUJQAIFCUACBQlAAgZ3ZkTRdERM9vbdi+nTfUJIXSL+0WcC9e27Zzn19aM69uWXNc2\no6IEgM8jvnoDgEBRAoBAUQKAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKA8D+JlqiNV5afRQAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc302a554a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv1, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the weights for the second convolutional layer and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights_conv2 = layer_conv2.get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAADuCAYAAACEaORrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VOW1N/A1SUiGkHsm4ZIQNnJRKSjgKHipIqWIlqJF\nFI5SDrUcBOVjqeWoVQ9SREVqeXl9Uaj6UUqRigcRkVpUjkUO9UUJCAoSEWHCJQQzuUAuJCRkv38I\nHtu11p6MrJl+fP19//ztLPazM8liMvvZz+NzXZcAAODsJPyzBwAA8P8DNFMAAANopgAABtBMAQAM\noJkCABhAMwUAMIBmCgBgAM0UAMAAmikAgIGkaL440L6962RmsrylU6F+gqovxLy8NV+t6XRiP8tC\ndXUUbmrytWGYZyXg97tOejrLW8JhtSbJJw8rXDRQP0/2KZaFDhygcGVl7K8xM9N18oXvf3W1WnMo\nyRHzhgb9PKf4JZ6u2Rp2XTfPY4gm2rULuH6/w/K6Or3monPlgztDaWpN344VLAtVVlK4ri72r2Va\nmuvk5kZX1Nwsxqc6dlFLEhN5FgqFKBwOx/waiYgC6emukyf8yFRVqTVHOvQU884JR9WaxsyOLDt8\nOETV1ZGvM6pm6mRmUvGECSyvum+eWpPzp6fE/PG6O9Wae3fcwrLgm2+2YYRnz0lPp+Kf/ITlVc8+\nq9bkJMnfxudnFqs1t405zrLgVVe1YYRnz8nPp+IFC/iBV15Ra+4JPC/m27fr56mpkfMtW3ylXuOz\n4vc7NGAAfw3++7+VLk9Exc/8Tcx7T7pSr5nxDMuCjzzShhGePSc3l4ofeCC6okOHxPj4jNlqSZrw\nf8kllwSjO+9ZcPLyqPjhh/mBFSvUmtnBNWI+M22+WrNn5N0sGz26bdeJP/MBAAygmQIAGEAzBQAw\nENVnpk15hbRvCv98NODxr9z1qfzZ6JONk/WiQIBnyueS5vLyiKZMYXHusx6fSzVniHH3OXrJwJ/z\nG3ke93JsVVURvfgiz6Xv+2nzxnwgH7ihRa15jy4T88sv9xydmYICorlzeX5ZUP/M9I318mejI0fq\n57m7hP8sH2zkn6PGwpGWPJpdzs//0EONXlVy/LB8Y4qI6Gc/a8eyUCjC4CyVlhJNncrz2lq1ZNJi\n5cCrKWrNsmU887jH9XfwzhQAwACaKQCAATRTAAADaKYAAAbQTAEADKCZAgAYiGq+UcreXXTOj87n\nB3bsUGvmzEkWc1/mIrXGvfbHPGz0muphqLJSnB9RWqo/gla0YamYP36EP3p7xr5efFfYk/fE7/E8\naarZXfSk+uX/Z5D8/T982K/WFK+MfliWUlKIevRQDiiuUx5PvO65SWpNQzl/NHjDhkijs9G5+QDN\nLL+D5TN/Jz+XTkRE06aJ8cw5fPrTGbO3j2JZsGVv5AEaORgYQHeN5Y8GP9nCr/2MLjdeKh/wmNNV\ncQOfytmiz/77O3hnCgBgAM0UAMAAmikAgAE0UwAAA2imAAAGols9JDWV6OKLeT5xolqSsVe+49dE\nW9Sa5eP5ne6qkjjd6U5NJerfn8WBbvpC2wdK+XiJiO7N4nd5z/BlbhNSfdEGS01dutO+WXwGwgMd\n9JoFC+S79l53rZcsiW5c1tpVHaWOLwqzMOrr1Zqnl6SK+R2Dn1NrpEWwtV0GzFVUEC3iM2MOeJQU\n/fGPYj7Za6XvXbt4dvPNEQZnJyeH6NZbef5O/dNqzdDiS+QDPxZmC5226IerWLblbWWV83+Ad6YA\nAAbQTAEADKCZAgAYQDMFADCAZgoAYADNFADAQFRToz6q7U5d1vMpNcrsJyLS907vsuAeteaWZdex\nbH7lZxHHZ6E5I5eOXsMXKOmYLi8OQUSUr02b6tRJrXFfeYplwXviM5/GdeXFG9LT9RptbYihoefV\nmm2D+cIURES+Dz0GZyk/X17UY9MmteSOIfJrVjXuLbVm8waeecy+MlUauIgm/4R/n0tK9JpZs+S8\nokKvGfuDzjwMh70HZ6i+nuj996MbwuYb5H3L7l9cpBfV1fGsujrC6L6Ed6YAAAbQTAEADKCZAgAY\nQDMFADCAZgoAYMDnuvIiHeIX+3wVRFQau+F46ua6bl6sT4JrjIvvwnXiGg19G64zqmYKAAAy/JkP\nAGAAzRQAwACaKQCAATRTAAADUT2bH0hLc52cHJZXJuarNbmNh+UDR4+qNY3CTbHDRFTtuvreIUYC\n7du7jvCQeksX/XnepE+FLR2IqKbge2pNVjp/Dj904ACFKytjf405Oa5TWMjywxXJak1BhrKlSlmZ\nfiJl746tJ06E43EXOJCR4Tr5ws9mVpZe9PHHct69u17Trh2LQocPU7iqKvavZUqK66QKW60k669l\nVYeuYp7j059Br2zNZlk4HKLa2nDMr5GIKDs74BYUOCz3H/JYGETT2hrVl4caGyl88mTE64yqmTo5\nOVQ8YwbLl2bdpdZMKLlfPvDEE2rNJ83NLIvXbjNOejoVjxnD8qo5+l4zOZefL+ZrHpcX+iAiGjWE\n7w8VvOqqNozw7DmFhVS8di3LvRaAeHTYO/IBbdUMIqJauQH7tm+PyxQXJz+fiufNY3nrDaPVmoQe\nStN8Xl/QRVrQJnj99RHHZ8FJTaXiq68WDjhqzfKgsC8WEd2S9LJas7SR/wY+9FCc9mUjooICh1at\n4r9PvWeMiv4fa2yM6suDmze36evwZz4AgAE0UwAAA2imAAAGovrM9FhKPr3Rk38+OmEn/1zqK9pn\nauXlakmfYv7ZiN9rBWpLfj/ReeexOC3No+Y3vxHjKVP0kmVXZLBsX2lipNGZqDiWTM+s45+PPvqq\n/NkvEVHDY/Jqw6leG8Q/9pice+3PbsnvJ+rbl8UJc2brNT/8oRivOHiZWtJZWGi7rjkl4vAslHfo\nQY8P4nu93/vpbWrNuFuVeylNTWpNyqs8S4jjW7GmJnkR+k7L1qg1fr+cJ2/QF/qmkSN5Jq2kLsA7\nUwAAA2imAAAG0EwBAAygmQIAGEAzBQAwgGYKAGAgqqlRmXSMrnP/zA9Mn67W9O4rPyP82Wf6c+uP\nPMIf3TvyVJweXWtoEKfuNKboj+Ym9+ol5tOm6Q/BSt+yK66IPDwLeWknaPLgj1g+vOtutSarn5xP\n2aCfx/mXB+QDDz7oMTpDx44RCY/N+h7S56yVl8vrTPzvn+inEZ4+pvr6iKMzkZKiPDn61Hq1JqGf\n/GKueFV/nv98YdacNvUoFlJTiS66iOfDhuk1ublynp4+XK1ZEDrJsuZr29Z78M4UAMAAmikAgAE0\nUwAAA2imAAAG0EwBAAxEdTefTp4kOiysnL9smVqyp1y50/++frdx/iaexW1RhcpKohdeYHHGK6+o\nJaOXyYsNL5mmn0ZabzZed4Dp2DGidetY/NY4fTHrCRvkhTOGrtYXBl8z7Mnox2apvJxo7lwWu99f\nrdc0yT/L7y3xWFB4yRIWLW9Rdpgwll31OY1dIfz8CbMYzpi66AIxv8lj7wNpkRGPdVHMJX20lbI7\n8Rk1H9x0k1rz/q/kxa4H7XhGP9F9vPm0O7Q/8gAJ70wBAEygmQIAGEAzBQAwgGYKAGAAzRQAwACa\nKQCAgaimRh1PyaO3nMksH17xol6kLYIyaJBacvdPf8qy5Q1tm55w1lJSiIqE/eP791dLVg2pkg90\n66HWDK2sZFl6esTRmahI7ETPZN3D8skdhUVsTtO2wXn6PH360x3BsqjHZio3l2jCBJ6//rpeM368\nnC9YoJa0CHtduZHGZqVbN6LFi1l8MktesIWIaMgQOa+o0E9z7bU8i9fPKxGR78ILKfktvnfTcb9+\nnYNqDsgHLp6k1hwYwfvbye1Y6AQAIG7QTAEADKCZAgAYQDMFADCAZgoAYMDnum2/7+jz+SqIqDR2\nw/HUzXVdj6UYbOAa4+K7cJ24RkPfhuuMqpkCAIAMf+YDABhAMwUAMIBmCgBgAM0UAMBAVM/mZ2UF\n3M6dHZZ3qP9CLzp4UM6zs/Wa48dZFGptpXBrK9+3wFhaWsDNzXVYnlfxiVeRnHvtQ+LjlxJqaqJw\nc3PMrzGQleU6XbrwA8nJelEoJMZlqT3VEu0l/uSTreF43AUO+HyuIx0YMEAvUvbiaHDbqyWpyXzh\ngtDBgxSurIz5a5mTE3ALCx2We91X/vxzOe9TVKfW7CnjP+ONjSE6eTIc82skIsrODrgFBQ7L/aRv\nJ1PX4hfz6mr9PF2TjrAsVFND4fr6iNcZVTPt3NmhJUv4PkGD3tcXu2j9xS/EPGHYMP1Eb7/NomBt\nbeQBGsjNdeiBB/g1Tl4o75tDRERXXCHnxfqeSpSYyKLgxx9HGp4Jp0sXKpb27XIcvWjiRDGe2X+N\nWjJmjJxfeKEvLlNcHCISX4FNwiZjZ0ibHRHRthb99R9YyN9MBIcP9xyblcJCh9au5VepLUxDpL8u\nxQs2qjVDZ13Jv764bQuAWCgocGjVKn6dvVv0Nznv1fQR85Ur9fPMz32EZcGnn448QMKf+QAAJtBM\nAQAMoJkCABiI6jPTigqi3/+e54NfuFqtyc6WPwnf/oR+nqKWfTy8/vpIwzORV/MZTV59HT8wSV9Q\nllbL+7Dv27JFLSn5M/++HPtFnD6DSk4mKizkeZLHj0PfvmI8e5rHzUftxly85OYSjRrF4roOHdSS\ntGPHxHzg6/oC6GsO3cqymrqofrW+saYm+WNe5aNfIiLaNuRuMV9xZL5a845zG8uCH4UiDc/Mrl0u\nnXsuv9mUlyd/Lkr05csvee01jxMV/pJnr7wSYXRfwjtTAAADaKYAAAbQTAEADKCZAgAYQDMFADCA\nZgoAYCCq+RuZmUTXXMPz5y/SH0N7q1c/MT90SD9P0XNzeHiEPzMbC7WdetE7M95gufbEKBFRaMRd\nYt57/CVqzcS5PCsvjzg8E5/sSaKBI/h+49uc0WrNxumrxLxlp36eoc/dEvXYTLW2EtXx58216U9E\nRM+8lCHmkxc+rtaM2vRjls1OP9WGAZ699NRTNDQorGURkq+DiIhKSsR4yL16SWjcCyw7GXF0di7q\nVUvFC3mfabhCf2xXuUzqPV2Y+njGuHE883qY/2vwzhQAwACaKQCAATRTAAADaKYAAAbQTAEADER1\nNz+7fDeN/Z1wh7qiQq2p+92dYt6pk8eJhgzh2V//6j04I+nJTTTU4QutfLL3HLWmT1aZfOAJfTWX\ndcKaJl4zBiydey7Rhg3CgeJpas2SJXI+a5Z+nuUjl8sH/vQnvchSUhJRIMBz8eK/tHcvXxiFiDx/\nxmndOp4Ju0XEwrYdidS+I79z73GJdHwMn61CRNSxb5Fec4wvzJN4VfwWh65pzaA1jfzO/aidH6g1\nA4VF5omIaP16taZqGf/etMzXF7//OrwzBQAwgGYKAGAAzRQAwACaKQCAATRTAAADaKYAAAai26im\noYFI2Neo7LC8zxMR0eiWA2I+e4k+DWNJzQSWHWxo2/SEs1ZWRvTggywOT1Gm+RARlWyW/6nB+sIh\nC/95a7lQ4vFqylj3Mj/w0ktqTahmqJgXTdIXmrhl/Hgx5zsmxUZNRhGtGcH3PB918Cm1Zl7S/fKB\nSy9Va97vdjPL6pPnRR6ggYG9aql40Tss3zT4B2pNxrJlYn50i/y7SkT0Q2Hantc+U9ayMlpp1LAG\nljeQvphQas+eYv6o7wG15v7pvPckHdzfhhHinSkAgAk0UwAAA2imAAAG0EwBAAygmQIAGPC5rn4n\nnn2xz1dBRKWxG46nbq7r5sX6JLjGuPguXCeu0dC34TqjaqYAACDDn/kAAAbQTAEADKCZAgAYQDMF\nADCAZgoAYCCqhU4CaWmuk5PDcjcvX62prpbzBI82nnXyC5aFqqooXFfnizjIs5SVFXC7dHFYnpqq\n13z6qZx37arXpJbuZlmoqYnCLS0xv8ZAu3auk5LCcre+Xq3xJSaK+clTp/QaJf+IKByPKTXZ2QG3\noMBhecqurWqNLztbzBs66XuApSa3sCx08CCFKytj/lp26BBwc3Iclnes+1yt2efrIebnOK36ifbs\nYVGoqYnCzc0xv0YiokD79q6TmckP+PTTf95YIOZNTfp5+qSGWBaqq6NwY2PE64yqmTo5OVQ8YwbL\nT065S61ZuVLO09L080ir+gQffzzi+Cx06eLQsmXFLO/fX6+R9v8jIlqwQK8ZOIWvdhPctSvC6Gw4\nKSlU3K8fy09ulle/IiJKTk8X81BNjVrjV/LOcZovWFDg0Msv89ey5/f034vkYcPEfNt9wipbpw0s\n5P/5B4frq2lZyslx6Je/5Nd49yZ9xbKbk1aJ+ctL+KpMX/kBX4Uq+PHHkQdoxMnMpOIJfEUnStJb\n2OiSR8U8FNLPU9z/NpYF16yJNDwiwp/5AAAm0EwBAAygmQIAGIhupX0iIuFGhLKgOhERTZok5+ed\np9dMWHkny/bXvxBpZCZSE5toYNY+fmDtTrVm8eJRYv6rX+nnGXLjBywrLwtGHJ+J9HSiq69m8SGP\nz0xTlc9Gl/5Gfxx5ZuHz8oGf/9x7fEYqKoiee47n899/Xy9SfjAH+k+qJWvW8RuwNXXR/2p9Ex3b\nH6e7+77FDzzLb3Ce8fKKj+QDOxv1Ewmfscd1qf2jR4l++1uev/uuWrKqv/I599ix+nlG/Jpn7dpF\nGNyX8M4UAMAAmikAgAE0UwAAA2imAAAG0EwBAAygmQIAGIhq/kbpiXyavINPW5o2Ta/Zvl3Oh2+f\np9ZMmXIPyz7gM4li4+hRoieeYPEnixapJX3mzBHziy9+QK3593/n2X/+Z+ThmfD7xSlAhU36NKf7\n7pPz9crjwkREdSP4o3lfis/UqK6FLs2fK0xp6jlGrWkNHRDzhEb9UcsTJ5JZFq8NLBqTM2iPwx9d\nLSwpUWs2lV8g5sqP8Zc1m/jjz667LfIAreTlEY0RXrfrr9drBgyQ86lT9RrpB/0vf/Ee22l4ZwoA\nYADNFADAAJopAIABNFMAAANopgAABqK6m18U3koLn+UL6yZvFhZBOO1KZYV2rztqAy/n5+gQeXgm\najKKaNWwp1nedzrPvhLMEOMpJfrd/ITVwgK9HgstW6pNzqV3CvlCuxN76jUVFXL+4ot6zej7eou5\nsFxFTNQc89GqtfxO++iRI9WahFkzxXzjsNlqzdj7urPst2VlbRjh2XNdokZhfZL1r+nTCZxOcr7R\nERZfPqOQ7yYQfHN/pOGZKU8uose78d/Bfi/qv5eDB8t5c7N+niGDeBYKKT3sH+CdKQCAATRTAAAD\naKYAAAbQTAEADKCZAgAYQDMFADAQ1dQoX2KivH+6tpoJEVFdnZx7bRz1T5RVvZ9Gr+ZTRBpGLNWL\nOslzTbKyPE60W9ij58SJCKOz0dxMVF7O8zv5GjZfcRw5D3ptWzV3rpzfeKNHkR2/n6hvX+HA7gK9\n6M03xfjKRr74zlf+4z94NlufSmUpIYEoLY3no/zCvlBfkX9eW5foP+MJK4X9lLz20jLm9xOdfz7P\nw2G9RvoZJyLqQ5+oNW++2YdlHjPp/g7emQIAGEAzBQAwgGYKAGAAzRQAwACaKQCAAZ8bxf4KPp+v\ngohKYzccT91c182L9UlwjXHxXbhOXKOhb8N1RtVMAQBAhj/zAQAMoJkCABhAMwUAMIBmCgBgIKpn\n8wNJSa6TzLeBoPx8jzPIp2jN0B9cTzh8kGWh48cpfOIE38/EWCAtzXVyclh+LEW/xsyje+QDPT32\nARHWLAgdPUrhY8difo25Pp9bJORl+RepNV0LWuUDJSX6iaSHxoloa0VFOB53gZOSAm5yssPy887T\naxIqjop56ERHtcap3Mq/nojCrhvz1zIzM+B27OiwPKNS31KkMoNvs0JEdPy4fp7uQkkoFKJwOBzz\nayQi6tAh4GZlOSzv3OixdUqH6Dc7Kmvhv+c1NSFqaIh8nVE1Uyc5mYqlBjF9ul4UCIhxw7BRaknq\nfXexLLhiRcTxWXBycqj43ntZ/kZ3fRWQ6xYMlw+sXq2faNMmFgWnTYs4PgtFRPRXIZ99a7FaM39O\ng3zg8sv1E116qRj7Fi2KyxSX5GSHevbk1yR867+Suni+mN+282615rkX+O/ZJZGHZ6JjR4cWLuTX\nOHzJLWrN0hHLxXz9ev08S5fw/0yDl8TrKomyshy6/XZ+nTP3euxbNUjY0CmCmUf57/lzz3mt5vM/\n8Gc+AIABNFMAAANopgAABqL6zJR69yZ65x0W+3KVVViJ6JFH+GKrREQ3X69/nvveH/hTWZWvv9eG\nARqorSX6r/9i8XU36jfMDrz9tpgXjRun1rSuXsPDjIzI4zOQeM45lPXYYyyfX/6kWnP/HP45NhFR\nSfcP1ZpXF/Hv45cWeY7PSlMTUSjE82L9o2G6Ulno/PmRwuLIX/kZj9YIr28MZNBxGk7CQtBDhqg1\nE0ZWifm4cfzG6xlHK/j7Lq/95611/mIHzVwo3ASeNEkv+v735dxjtecZB/l9i9cjDe40vDMFADCA\nZgoAYADNFADAAJopAIABNFMAAANopgAABqKaGrX7syS6ZASfPvHUU/qUinF3ylOgSv6mL0odFGYh\nfYPHbL+Riowe9PSwVSy/I0/fh7yoa1cxn9lfnx4zezU/B9XURB6ggUP12XRP8c0s93rMfs2CfWLe\n0KNH1OeP00tJA/q1UPE7wjQgv18vOu8JMX56pb42wx0PCo8bbtkSaXg29u8nGj+e5x6PJm/cKf++\ntl6lT1cc4jgsa1dWFnF4ZlJTiYLC93nwYLVk9uoLxHzm7berNRn9+rEs8W79UeKvwztTAAADaKYA\nAAbQTAEADKCZAgAYQDMFADAQ1d3883u10Afr+N3Ra2/V7+bfsXChmHstRDvTWcoyf31l5AEayOvQ\nQHcM3sYPbD+kF23eLMcT9ZLjM0az7FT6oxFGZyMQIJo4ked9fi4v5kxE1NJDvsbU117Ta66/Ptqh\n2aqp8V6gW6KsHH1HOKzXFAsLoB85Et15v6l+/cQxjxqXqpasmfKFmH/gcZoGYcUYZe+FmKgO9KIV\n//oGy8cOUxYtJ6IHlfVMzv/eA2rN7teEXTO8Zn98Dd6ZAgAYQDMFADCAZgoAYADNFADAAJopAIAB\nNFMAAAPR7QFVVUW0bBmL//K3B/WSdbXygd/o+9CTsKgCpaREGJyRAweI7uRjG935/6ol03vK+cGD\n+mleeolnVfLWPOb8R0upz4LJ/MDVV6s1SdJmSkTyJkun9e6uLGazX19Qw9KJDgH6KHgby732gLpt\nRJqYL23kC8OcMfJf+fW0RB6ejZYWImHaVs+eRWrJBcPkRVtC6friQxs28KxxfNv2k7eQXbGHxv5+\nKMvf68r3pDvjsjp5caLdU/UVfW6Zxfc6238EU6MAAOIGzRQAwACaKQCAATRTAAADaKYAAAZ8rqvf\nwWNf7PNVEFFp7IbjqZvrunmxPgmuMS6+C9eJazT0bbjOqJopAADI8Gc+AIABNFMAAANopgAABtBM\nAQAMRPVsfkpKwE1Lc1jeoO8cQHnKPbD82s/1os6dWRQqK6NwTU3MH+oOtGvnOtI2BT6PU584Ican\n+l6oluzcKXz9qRC1toZjf425ua5TxJ/drjuRqNakJTXKB+rr9RNlZorx1h07wvG4C5yREXDz8x2W\n19ToNU6RshmH143aBP6eJFRaSuFw7F/L5OSA2769w/JePfRNRUoPRv8eqltGNctCFRUUPn48Lgst\n5OQE3MJCh+W+j7eqNe2U39lWj9fy5PcuYtnhwyGqro78WkbVTNPSHLrmGr5KxIcf6jVTp8r5XRv4\nHkhfeZAvnBIcPz7S8Ew4fj8VDxjADyR5fKt27xbj4+/qK2r0FBZHqa6Oz8IRTlERFb/7Lsvf25mh\n1lwWEPbGIVL3vyIiohEjxNjXsWNcprjk5zs0bx5/Ddau1WueX6i8M2jxWLpE+M83eKm+n5al9u0d\nuuwyfo1/eUV/hzN5ur4/lOaZYS+zLPjrX0f973xThYUOrV3LrzOpm97juiiLIzU0Km8MiOjQKn6O\n0aPb9nuJP/MBAAygmQIAGEAzBQAwENVnpt0bdtHyD8/nB/7wB73oF7+Q85tu0mtmzeJZWZnn2KxU\nBs6lpZM2snzCCHmvcSIiysoS45pyveSLJ5ayLPhQZcTxmdi5U/zQtmWlxzWOHSvnp07pNSX6Irzx\nkJVFdMMNPB8yRK+5coT8eeLGK+5Xa5b3fZRlVdXxWQA7ECCaOJHny1frn4s+s6G3fGDGDP1EPYXP\nDeO1YDsRJSbKv2YZ//ZvetEf/yjGXp8Y9675gGX+Ux43Wb8G70wBAAygmQIAGEAzBQAwgGYKAGAA\nzRQAwACaKQCAgaimRpHfT3Q+nxq1J+sStWTxpfJ+8/OdVWrN1II1LDuQHJ9HLY8eJVqwgOeOI+81\nTkR05Rj5WNGUKWpN1cMPsyxee61/nnYhjb6CPza3Kuk9taZ9ifzM8Ik/8McMz/ikr7LX/GOPeQ/Q\nSEMD0fbtPB848QK1ZqM0l4qI3hjMpz+dERB+i7yePraUndlKY3/MHx2t6dBBL1Km8h24/Xa1pG4X\nf569MSH6x1K/KZ9P+Z6OGaMXlctzE5eP4/3ljFsHSVP9PL6XX4N3pgAABtBMAQAMoJkCABhAMwUA\nMIBmCgBgILp7jsnJRI7D/xGPf0W7of1Ro7449KIb+CrhW7ZEGpyNrCyikSN5PniwR9GkSXLucadx\n4vbZLAttjM+MhYICorlzeT5hzmVqTVOTnN+/XbljT0SbF0c7Mlupn35IA4cIC157LOjx1uCZYt6i\nrydMtbU881r/xVJTcwLtK+d31ccM0FeT33bFXWK+OO1JtebRLL7QkD+xuQ0jtJFwsJRSp0/mB6RV\nXk672S/ftX95tf4ze8sOYWH6f5F30vhHeGcKAGAAzRQAwACaKQCAATRTAAADaKYAAAbQTAEADEQ3\nNaqpiWjvXhYr6yYQEVGnTnIu/DP/Q1oI48gR77EZ6ZJZT7NH8n1gaOEmvUiZnjH1fyl77RDRhg08\nq2/bVjNnrbaWaP16nnu9JsV8XRQiIhpY/IxetPIJMY7P7khElJpK1L8/i9f0l6c/ERHN4TNjiIjo\ng4f+rNbF2Y8RAAAAxklEQVTc/MqPWFZdHXl4FvbuJfoRPz399Kd6TdFieQrUgXH3qDUDR85jWcln\n7SKOz0xNDdHrr7N443j952+xNjVvwzi1pizAF8FpTmofcXhEeGcKAGACzRQAwACaKQCAATRTAAAD\naKYAAAZ8rqsviMC+2OerIKLS2A3HUzfXdfNifRJcY1x8F64T12jo23CdUTVTAACQ4c98AAADaKYA\nAAbQTAEADKCZAgAYQDMFADCAZgoAYADNFADAAJopAIABNFMAAAP/D3DDnMk3DFIiAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2cc6bde10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv2, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting the output of a convolutional layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_conv_output(values):\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = values.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot the output images of all the filters.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the images for valid filters.\n",
    "        if i<num_filters:\n",
    "            # Get the output image of using the i'th filter.\n",
    "            img = values[0, :, :, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, interpolation='nearest', cmap='binary')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input Image\n",
    "\n",
    "Helper-function for plotting a single image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_image(image):\n",
    "    plt.imshow(image.reshape(img_shape),\n",
    "               interpolation='nearest',\n",
    "               cmap='binary')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot an image from the test-set which will be used as an example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADV5JREFUeJzt3X+oXPWZx/HPUzeNYKrmNtMYbextc0UJwabLEFYra1dt\nuAmB6D+SICUFaQoqrlB0xaKr+E9YbYqgVG80NC6tbTGVBAmubqhooJaMJv6Ku+uvG5twzZ0YoSkI\nadJn/5iTcqv3fGecc2bO3DzvF1xm5jznzHlyyOeemfmeO19zdwGI5wtVNwCgGoQfCIrwA0ERfiAo\nwg8ERfiBoAg/EBThB4Ii/EBQ/9DPnc2bN8+Hh4f7uUsglPHxcR0+fNg6WbdQ+M1sVNIDkk6T9Ki7\nb0itPzw8rEajUWSXABLq9XrH63b9st/MTpP0kKQVkhZLWmtmi7t9PgD9VeQ9/zJJ77j7e+5+TNKv\nJK0upy0AvVYk/OdJ+uOUxweyZX/HzNabWcPMGs1ms8DuAJSp55/2u/uYu9fdvV6r1Xq9OwAdKhL+\ng5IWTnn81WwZgBmgSPh3S7rAzL5uZl+UtEbS9nLaAtBrXQ/1uftxM7tJ0n+pNdS32d3fLK0zAD1V\naJzf3XdI2lFSLwD6iMt7gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxA\nUIQfCKrQLL1mNi7pqKQTko67e72MpgD0XqHwZ/7F3Q+X8DwA+oiX/UBQRcPvkp41s5fNbH0ZDQHo\nj6Iv+y9z94Nm9hVJz5nZ/7j7C1NXyH4prJek888/v+DuAJSl0Jnf3Q9mt5OSnpK0bJp1xty97u71\nWq1WZHcAStR1+M3sDDP70sn7kpZLeqOsxgD0VpGX/fMlPWVmJ5/nl+7+TCldAei5rsPv7u9J+maJ\nvQDoI4b6gKAIPxAU4QeCIvxAUIQfCIrwA0GV8Vd9ITz55JO5tU2bNiW3Pffcc5P1008/PVm/7rrr\nkvVzzjkntzYyMpLcFnFx5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoBjn79Ctt96aWxsfH+/pvh9+\n+OFk/cwzz8ytLV68uOx2ZoyFCxfm1m677bbktvX6qf8t9Jz5gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAoxvk79Oijj+bWXn311eS27cba9+3bl6zv2bMnWX/++edzay+99FJy23ZTqH3wwQfJehGzZs1K\n1ufNm5esT0xMJOupf3vqGgCJcX4ApzDCDwRF+IGgCD8QFOEHgiL8QFCEHwiq7Ti/mW2WtErSpLsv\nyZYNSfq1pGFJ45KudfePe9dm9a688squap0YHR0ttP3HH+cf+nbXCLQbz969e3dXPXVi9uzZyfqF\nF16YrF900UXJ+pEjR3JrixYtSm4bQSdn/p9L+vT/ztsl7XT3CyTtzB4DmEHaht/dX5D06V+hqyVt\nye5vkXR1yX0B6LFu3/PPd/eT11Z+KGl+Sf0A6JPCH/i5u0vyvLqZrTezhpk1ms1m0d0BKEm34T9k\nZgskKbudzFvR3cfcve7u9Vqt1uXuAJSt2/Bvl7Quu79O0rZy2gHQL23Db2ZPSPq9pAvN7ICZXS9p\ng6Tvmtnbkq7KHgOYQdqO87v72pxSscFtlGbu3Lm5tSuuuKLQcxe9hqGIrVu3Juup6xsk6eKLL86t\nrVmzpqueTiVc4QcERfiBoAg/EBThB4Ii/EBQhB8Iiq/uRmUmJ3MvDJUk3XDDDcl668ryfHfddVdu\nbWhoKLltBJz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAoxvlRmYceeihZb3cdwNlnn52st/vq7+g4\n8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzo6d27dqVW9uwodh0D9u2peeKWbJkSaHnP9Vx5geC\nIvxAUIQfCIrwA0ERfiAowg8ERfiBoNqO85vZZkmrJE26+5Js2d2SfiCpma12h7vv6FWTmLl27Mj/\nb3Hs2LHktldddVWyfskll3TVE1o6OfP/XNLoNMt/6u5Lsx+CD8wwbcPv7i9IOtKHXgD0UZH3/DeZ\n2WtmttnM5pbWEYC+6Db8P5O0SNJSSROSfpK3opmtN7OGmTWazWbeagD6rKvwu/shdz/h7n+VtEnS\nssS6Y+5ed/d6rVbrtk8AJesq/Ga2YMrDayS9UU47APqlk6G+JyR9R9I8Mzsg6d8lfcfMlkpySeOS\nftjDHgH0QNvwu/vaaRY/1oNeMAN98sknyfozzzyTW5s9e3Zy23vuuSdZnzVrVrKONK7wA4Ii/EBQ\nhB8IivADQRF+ICjCDwTFV3ejkPvuuy9Z37NnT25txYoVyW0vvfTSrnpCZzjzA0ERfiAowg8ERfiB\noAg/EBThB4Ii/EBQjPMj6emnn07W77333mT9rLPOyq3deeedXfWEcnDmB4Ii/EBQhB8IivADQRF+\nICjCDwRF+IGgGOcP7qOPPkrWb7755mT9+PHjyfrKlStza0yxXS3O/EBQhB8IivADQRF+ICjCDwRF\n+IGgCD8QVNtxfjNbKOlxSfMluaQxd3/AzIYk/VrSsKRxSde6+8e9axXdOHHiRLI+OjqarL///vvJ\n+sjISLLe7u/9UZ1OzvzHJf3I3RdL+idJN5rZYkm3S9rp7hdI2pk9BjBDtA2/u0+4+yvZ/aOS3pJ0\nnqTVkrZkq22RdHWvmgRQvs/1nt/MhiV9S9IfJM1394ms9KFabwsAzBAdh9/M5kjaKukWd//T1Jq7\nu1qfB0y33Xoza5hZo9lsFmoWQHk6Cr+ZzVIr+L9w999miw+Z2YKsvkDS5HTbuvuYu9fdvV6r1cro\nGUAJ2obfzEzSY5LecveNU0rbJa3L7q+TtK389gD0Sid/0vttSd+T9LqZ7c2W3SFpg6TfmNn1kvZL\nurY3LaKId999N1lvNBqFnn/jxo3J+qJFiwo9P3qnbfjdfZckyylfWW47APqFK/yAoAg/EBThB4Ii\n/EBQhB8IivADQfHV3aeA/fv359aWL19e6Lnvv//+ZH3VqlWFnh/V4cwPBEX4gaAIPxAU4QeCIvxA\nUIQfCIrwA0Exzn8KeOSRR3JrqWsAOnH55Zcn663vesFMxJkfCIrwA0ERfiAowg8ERfiBoAg/EBTh\nB4JinH8GePHFF5P1Bx98sE+d4FTCmR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmo7zm9mCyU9Lmm+\nJJc05u4PmNndkn4gqZmteoe77+hVo5Ht2rUrWT969GjXzz0yMpKsz5kzp+vnxmDr5CKf45J+5O6v\nmNmXJL1sZs9ltZ+6e3pWBwADqW343X1C0kR2/6iZvSXpvF43BqC3Ptd7fjMblvQtSX/IFt1kZq+Z\n2WYzm5uzzXoza5hZo9lsTrcKgAp0HH4zmyNpq6Rb3P1Pkn4maZGkpWq9MvjJdNu5+5i71929XqvV\nSmgZQBk6Cr+ZzVIr+L9w999KkrsfcvcT7v5XSZskLetdmwDK1jb81vp61sckveXuG6csXzBltWsk\nvVF+ewB6pZNP+78t6XuSXjezvdmyOyStNbOlag3/jUv6YU86RCFLly5N1nfu3JmsDw0NldkOBkgn\nn/bvkjTdl7Mzpg/MYFzhBwRF+IGgCD8QFOEHgiL8QFCEHwjK3L1vO6vX695oNPq2PyCaer2uRqPR\n0bzpnPmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKi+jvObWVPS/imL5kk63LcGPp9B7W1Q+5LorVtl\n9vY1d+/o+/L6Gv7P7Nys4e71yhpIGNTeBrUvid66VVVvvOwHgiL8QFBVh3+s4v2nDGpvg9qXRG/d\nqqS3St/zA6hO1Wd+ABWpJPxmNmpm/2tm75jZ7VX0kMfMxs3sdTPba2aV/v1xNg3apJm9MWXZkJk9\nZ2ZvZ7fTTpNWUW93m9nB7NjtNbOVFfW20Mx+Z2b7zOxNM/vXbHmlxy7RVyXHre8v+83sNEn/J+m7\nkg5I2i1prbvv62sjOcxsXFLd3SsfEzazf5b0Z0mPu/uSbNl/SDri7huyX5xz3f3fBqS3uyX9ueqZ\nm7MJZRZMnVla0tWSvq8Kj12ir2tVwXGr4sy/TNI77v6eux+T9CtJqyvoY+C5+wuSjnxq8WpJW7L7\nW9T6z9N3Ob0NBHefcPdXsvtHJZ2cWbrSY5foqxJVhP88SX+c8viABmvKb5f0rJm9bGbrq25mGvOz\nadMl6UNJ86tsZhptZ27up0/NLD0wx66bGa/Lxgd+n3WZu/+jpBWSbsxe3g4kb71nG6Thmo5mbu6X\naWaW/psqj123M16XrYrwH5S0cMrjr2bLBoK7H8xuJyU9pcGbffjQyUlSs9vJivv5m0GauXm6maU1\nAMdukGa8riL8uyVdYGZfN7MvSlojaXsFfXyGmZ2RfRAjMztD0nIN3uzD2yWty+6vk7Stwl7+zqDM\n3Jw3s7QqPnYDN+O1u/f9R9JKtT7xf1fSj6voIaevb0h6Nft5s+reJD2h1svAv6j12cj1kr4saaek\ntyX9t6ShAertPyW9Luk1tYK2oKLeLlPrJf1rkvZmPyurPnaJvio5blzhBwTFB35AUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBThB4L6f6yMEem39pFEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2cc21aac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "image1 = data.test.images[0]\n",
    "plot_image(image1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of Convolutional Layer - Method 1\n",
    "\n",
    "There are different ways of getting the output of a layer in a Keras model. This method uses a so-called K-function which turns a part of the Keras model into a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras import backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_conv1 = K.function(inputs=[layer_input.input],\n",
    "                          outputs=[layer_conv1.output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then call this function with the input image. Note that the image is wrapped in two lists because the function expects an array of that dimensionality. Likewise, the function returns an array with one more dimensionality than we want so we just take the first element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 28, 28, 16)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_output1 = output_conv1([[image1]])[0]\n",
    "layer_output1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then plot the output of all 16 channels of the convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXlwlOd5wH+SAEkgJAUJMIcODnGbwzZnOYxtJsYnSW1i\nx6mTSRzHk0madtqm02mnbsdtpzNtp+20M53JJPakpXXSmMQ4xsVJbDDmPgXmEiBAIIHuC91n//jy\nPN9KWu2nlfbQLs/vn4Xd1erVu9/3vM/9JPT29mIYhmEMTmK0F2AYhjHaMUFpGIbhgQlKwzAMD0xQ\nGoZheGCC0jAMwwMTlIZhGB6YoDQMw/DABKVhGIYHJigNwzA8GBPMm7Ozs3vz8/PDtJTwcuPGDaqr\nqxOivY7BsL0NH9nZ2b15eXnRXsawOXXqVHVvb+/kaK9jMGJ5f0tKSoZ07QYlKPPz8zl27NjwVxVF\nVq1aFe0lBCQ/P58TJ05EexnD4qGHHor2EgKSl5fHwYMHo72MYZOamloS7TUEIi8vj8OHD0d7GcNi\n7dq1Q3qfmd6GYRgemKA0DMPwwASlYRiGByYoDcMwPDBBaRiG4YEJSsMwDA+CSg8KxJ07dwC4ePEi\n9fX1AJSXlwMwceJE5syZA8B9990HwMyZMwFITExk3LhxAPT09IRqOXFFQ0MDANeuXaOyshKAzs5O\nwEnNkL0dP358dBYYw9TV1enj5MlOqqI8DpW2traQrytekHt7JHR0dIRgJSMjZIJS8qiOHz+uN/GR\nI0cASElJ0Q1LTU0FIDMzE4Dk5GQyMjIA6OrqGvTz5efmz5+vgmH69OmAIywSEpyc0XgUtufPnwdg\n165dXL9+HYC0tDQA1q9frweTCErZq+bmZtrb2wFobW0d9PPHjh0LwIwZM5g9ezYASUlJof4zRiUi\nKPPz84d90KSkpIx4HSZsB2c0CFszvQ3DMDwImUaZnJwMQE5ODnPnzgUckxvg2LFj3L59G3DNcV8+\n97nPAZCeng5Ae3u7akWincqp0tzcrJpndXU14JzoYtLHI01NTYCjdSxduhSA7OxsoG/FkeytuEGq\nq6t1r1paWgDo7u4eoAGJ9piSkkJBQUG4/oxRyaJFi6K9hLhGrl2xgGIV0ygNwzA8CJlGuXjxYgAe\nfPBB9R0+/vjjABw4cICKigoArly5Arj+yKNHj3L37l3ADfTU1taqxrR3717A1ZYyMzM1uCE+pe7u\nbp544gnA9bfFE7IX27Zt0yDYtGnTgL4n9YIFCwAYM8b5Wru7u9VH2d3dDThaqQQrSkqcEuKioiLA\nDbAZRqgQ33htba3ew2fPngXg9u3baomKb1waw9x///0h8f2GipAJSvlD/bF+/fqgP09u8H/9138F\n0IDFl7/8ZQ1oyMZnZGRoNHjGjBlB/67Rjhw82dnZKiD9HQgiIIWkpKQBAQpxh4Ab+CouLgZGf3OL\ncCCm4ZgxY7h48SKAHur79+/X/du0aRMACxcuBNzDa6T09vaG5HNGK1evXgXg4MGDqiTJfVtcXKwH\nuFzjIjjr6+tJTHQM3qysLMCRCeKmk2DwM888A8DWrVv9/v7GxkZg5AE3M70NwzA8CJlGGWrkZPn+\n978/4LUlS5YAcPPmTcDRBuTUiUfETZGZmanBrZG4GESLefPNNwE0tWrSpEkjWWZMIhplSUkJn332\nGQCHDh0CHC1ILJkdO3YAbhpPUlKSBhhFKxo/fjxnzpwBXA3njTfeAByXVH/OnTun36doqvGGuMmS\nk5NVO5d0wAkTJmiQ98aNG4C7l83NzX2CmOBohRLwXb58OeBqp42NjfqafOa1a9eYMGEC4LoGh4tp\nlIZhGB6MWo1yKOTm5gKOhhTPCefiD+vt7dW0Kfk7ExMTVUOUPQhEb28vb731FgC/+c1vAPjWt74V\n8jXHChIMy8vLU7+XWCzPPvssNTU1gJuKVlpaCsD169fV/1VbWws4yeuiGW7ZsgXwr0l++umngBPk\nfPjhh0P+N40mVqxYAbhBGnD3q7q6Wq1C8QtLGlt1dbUGguQ7yM3NJScnB4A1a9YArqboe9+LlllW\nVqaa50gxjdIwDMODkGmUXpqcRLCGwvvvv8/u3bsB+JM/+RMgcFR9KJpULOMvYudvP6VMS3xnnZ2d\n6ssUP8///d//qUYpp/eGDRtCv+gYQTIFMjMzVaP0nf8i2rr4EmWP29raaG5uBqCqqgpwIrVPPfWU\n5++UcSrjxo3TLIZ4RaLU8gh972Xxv/ePTtfV1Q2QKbm5uQF986KVXrt2DXB8oMHW7Q9GxEzvoZjE\nJ0+eBOC9997TzQwkIGVz09PT4z7NIhByUPSvifW9qMRk/NWvfqXpVV/72tcAtHbeGEj/vZXHtLQ0\ndYlIwMArD1Wub0nH2rZtW8hu5FhFDqr+gUTftLah1nrLASRpSFu3btXvZqSY6W0YhuHBqAjmSErA\nuXPnACex9I/+6I88f07SASD+E3eHi+zLpUuXACgsLNQg2Le//e2orSseECtpKKlabW1t7Ny5E3Cr\nqgoKCu6ZLk3DIZC7Ttwesn9Xr17VCj8JqM2cOTNkbjnTKA3DMDwYFRqllDlJsu7q1avjsmY7Gkiq\nhCRRl5aWqiYpZWPG8JCy2aEEZHbv3k1ZWRkAX/rSl4C+AQ5jIP1Lcn3pnxJ3/fp1/T4+//nPA6Ft\nZB11QVlZWcmHH34IuFHYF198MajPiMfcyVBw9+5dzdl7//33AZgyZQqvvPJKNJcVFzQ0NGgEXKrI\n/CHm4NmzZzWncuXKlUBoGv7GK14BHMl/lcBkU1MT8+fPB9Bcy0CCNljM9DYMw/AgahqlnMYHDhzQ\nkRGSxT9Uk8Q0Sf9IcOzMmTP87Gc/A1Cz76/+6q/u+ZSUkSB5f1VVVVrBEwhJWWloaND671hvYhtO\ngg1u+WqUa9euBUIzOqI/plEahmF4EHGNUpywElz44IMPtPfkF7/4xUgvJy6RRrw7d+5UH6X0U/zu\nd78btXXFMnLdSrK4V7cfqeQ5ePAgAPPmzdMxG/FeSTYcZE+GqlFeuHABcDsF5eTkhNVSMo3SMAzD\ng4hrlJIoKhplU1MT3/ve9wCGPNjKfJP+Ec1cfL579uzRuuXvfOc7UVtXPCA+del76uUHk16f0vnm\n8ccfD1k5XTwy1Ai1dBsS+SEsXLgwrCmFEROU0jLp448/BlyTZNWqVXHfairciJlXWFgIwDvvvAM4\n1SCSM/nII49EZ3ExjghIaSy7bNmygO+XvZcbWRqOWD29f0RABnJHiAJw8+ZNnaElaVeSchWq0RyD\nYaa3YRiGBxHTKKX6Zv/+/YCT+Azwl3/5l0F9jpndA5E53nv27AHg+PHjAKxdu9aCNyOkrq4OCNzF\nSiguLuaDDz4AYOrUqYA7d93Mbv8Mpf2iJPSXlpbq9yGNgMVdF+5KPtMoDcMwPIiIRllbW6u+BRkJ\nKq3ch5oOYN2B/NPc3Mzhw4cB2LVrF+COLZBRnsbwaGpq0uvOd8zvYJw5c0ab/4pP2LcJsNGXYBPD\n8/PzddibzLCP1EC8iAjKW7ducfnyZcB1avu7iUWIzp07d4AqbYLSPxUVFXzyySeAm1MmTRrE7DOG\nR2dnZ59ZL16kp6ezefNmwK0yC1QHbgTHhAkTVEBKk+RItakz09swDMODiGiU1dXVKvm/+tWvAu6J\n64uYLXfu3GHGjBmAVTF4UVlZqakr4hjfvn07AEuXLo3WsuKCoZp1kttXU1PDQw89BLizq43gkFne\nvvXwBw4cAJz6+tWrVwOR19RNozQMw/AgIhrl7Nmz+fu//3sgcJqE9OezVIqhk5aWxrp16wC3YakM\nDTNGhpfWsmPHDsBNz2pra9MgpeFNZWUlJ06cAOD8+fMAnD59GnD2vn/gJjs7WzXKSGMapWEYhgcR\n0SiHWr7lrw+lJZgHJjMzk40bNwJOhxow/1g4kDJRKcHdtWuXJvjL9f3EE0/YCJMgaGho0C5AkuIm\n+5uQkMB9990HwAMPPAA45aOpqalRWOkoGAVhjIy0tDQ1TXynUhojp6qqSs1qmRAqVU+HDx/WA0n2\nf+XKleY2CoL6+nptkiMHzO/+7u8CTiBS8ix9p1ZGa3yGmd6GYRgeJASTyJ2QkFAFlIRvOWElr7e3\nd9TOQLC9DR8xvrdg+xtOhrS3QQlKwzCMexEzvQ3DMDwwQWkYhuGBCUrDMAwPTFAahmF4YILSMAzD\nAxOUhmEYHpigNAzD8MAEpWEYhgdB1XpnZ2f3BtMafzRx48YNqqurR20XYNvb8JGdnd0by7NrTp06\nVT2aK3PuhWs3KEGZn5/PsWPHhr+qKDLa58fk5+drb75YQ7p6j1by8vI4ePBgtJcxbFJTU0d1eeC9\ncO2a6W0YhuGBCUrDMAwPTFAahmF4YILSMAzDAxOUhmEYHpigNAzD8CBkM3O6uroAZw5Ge3t7n9ca\nGhoYM8b5VfI4ceJEwBlNK7Mx5DWjLzK35ciRI7q3W7ZsASArKytq64onxo4dS0tLCwCXLl0CoKys\njKamJgCmTp064FHm49h1601NTQ2lpaWAO4567ty50VxSUITsG25rawOcWb2JiY6ieuPGDcC5wRsb\nGwEYP348AJMnO/mzWVlZOhQr0GCmhIQE/fnMzExn8b+9QNPS0pg0aRKACt144ujRowC89dZbJCUl\nAe4M70ggXfDlO4gnysrKAOeAv3r1KgCffvopABUVFVRXVwMwZcoUAN3/2bNn69RQOfQD0dPTo/s4\nffp0AObPn6//jsfr1peysjI98B9//PEoryZ4zPQ2DMPwIGQapWiFS5Ys0edE8ztz5oye3Ldv3wag\nqKgIgOTkZH1/Wloa4Ggu/bUYmauclpam837l82fMmMHKlSsB9LV4QrTHjIwMfvCDHwBQWFgIwJ49\nezh06BCA7sFjjz0GOHOmZRyoaELDGffZ2toKuNZAPHHlyhXAsYjOnz8PoOb2J598opaSXN+iYba3\nt+u1KWNru7u79ToV5BqdN2+eWkBi/WzatEm1q5kzZ4bhrxs9zJ07l6VLl0Z7GcPGNErDMAwPwuqF\nFv/LSy+9pCd3XV0d4J7aiYmJ1NfXA66vJzExUU9yCV6UlDjlrsnJyTQ0NADuUPq5c+cyZ84cID41\nytTUVAA2b97M5s2b+7xWUFBAR0cHAD/+8Y8BVMP8xS9+odrLzZs3AXjggQdUc5f37dixA3B8xPK9\n+FJRUQHArFmzQvdHjRJEoysrK+Opp54CXN/3u+++y/79+wHX3y4Bn7Nnz6qm3dPTAzhau2iUcv3K\na5cuXVIfvFhXkydPZtmyZUD8apQS5K2oqODatWuAa92kp6czbdo0wL1vR6sfPCLhutzcXHJzcwd9\nXS4+cWgnJiaqiVNbWwtAVVUV4Djdf/WrXwFoo4PExET9Qu41Zs6cyT//8z8D6ONQERN99erVANy9\ne3fAe44fP35PRHVnzJgx4Llt27axbdu2QX9GrrmamhrACUzKwS6HkLg+Tp06xa1btwAoLy8HnOs9\n3sdFi4to//79vPnmmwAaBOvo6GD+/PmA07gE3EMqJSVFD2hxX7S2tqr7Rw6kffv2AfDoo4/y+uuv\nh+3vMNPbMAzDg1GhKvgLEoiZIsGeBQsWAI7Z8t577wGuWT5x4sRhBSliGWl3N5L2cQUFBX0efTlw\n4AAA58+f56WXXhr274hnRPuR3Erf5yTPVdi2bZterx9//DHg5MeK6RmvSJ/KY8eO8eSTTwKulZiV\nlcXly5cBJ4UQXLO8o6ND91LyL9PT09XtJma8uJ2effZZv79f0hIlGDdcTKM0DMPwYFRolP4QR3l2\ndnaf5zMyMtQnJEnrixcv1n/HMy0tLeqrFS1837596t+RQJm8Z8yYMXoC/87v/A4AP/3pT/n5z38O\nwMsvvwzA17/+9QG/a+fOnQAsWrRIgz/GyOhvHeXk5GjAM16R+3f79u2q3UksoqOjQ5+7cOGCPgeO\ntShWpcQrfH3lcn2KNv/oo4/6/f0j1SQF0ygNwzA8CJlGKWkQgyFljYJEDP1FVGtrazXS1Z93331X\nU40WL14MwIMPPjjo++OJ8ePHE2j2i6TviEY5adIkTaKW7+fy5cu89tprALzwwgsDPkMih6dOnQLg\nL/7iL0K0+tGJpPEMRv8eBYHo7e3VVDd5lJ/LycnR98l3kZOTo1pmvGdtZGdnD7AOfZFCFcm8GD9+\nvP5b9rChoUGvS/FJyn2/fPny8Cz8t0TM9O4vSEVw+j4vzu4xY8YMuDDlfTt37lSVfNOmTYBT9TBa\n86+iga8bQqohTp8+DcDGjRvZvn2735+rrq7mP/7jPwB45ZVXAGu6IQIskCATV9DVq1c1b1WCGGJK\ndnV1aaqQ5MWOGzeO7u7usKw71pD7XVKHoG/VHjgpQZIPLPv29NNPD/gskSP9f34kmOltGIbhwagK\n5ohW6M8B+1//9V8AXL9+nQcffBBwqkwgcNehex3RxOWUXb9+/aDvffvttzXx+m/+5m/Cv7g4QdJd\npkyZoqagv45CkiTtq6XHe8J5KLl48aLutQQu/Vk8odQkBdMoDcMwPBgVGqWvb3Iw3n77bcDx70i9\nsziAk5OT7WQeBEnoHTt2LIDfdBSpo/+Hf/gHvvGNbwADg2/GQPr7FzMzMwdNIG9ra1NLyTc9xq7b\nwZG9kZLPffv2qY/yD/7gDwa8X6yncFy7UReUXV1dmuvnr6GFBBcqKysBJx9Q2omJU9wuNv9UVlby\n4YcfAvDiiy8O+r633noLcHLSwlkvG29cv34dcK/Dzs5O7VvQv9qsrKxM3RpigntlitzryF5KT4e2\ntjbWrVsHoM1EfAnn4W5qg2EYhgdR0yhFC2xpafGbIiGdVvbs2QO4Ttv169cH7ERkuBrL/v37NU9Q\nRhn4IiaNVDl873vfi9AKYx/fMRFiSqenp6tWIxqlbwtBcX/Ee85kKOju7tZ2dGJNLlmyhC9/+ctR\nWY9plIZhGB5ETaMU/0N9fb0m5/oivkkZSCRpLStWrNBOQeab9I8MxyosLOS73/3uoO/767/+a8AN\n9EiSuTE4kkB+69YtTVGR5xYuXDggVU00yqlTp6pP0q7bwRHr8vbt2xw+fBhwa+MlNhENTKM0DMPw\nIOIapZyq4kfz7eUn7Nq1SzsXS4v8jRs3Ak59rJ3I/pESOenEsmbNGr/7++///u+A23PyT//0TyO0\nwthFrjkZHZydna0a5cKFCwHHR+nbvRvcWuSxY8eab3IISD9KuYbB3Xt/SfzhTAnyJWKCUv5YKXSX\nP0xSK8BVu3/yk59odr20B5Oi95SUFBOU/ZCAjaQCyWB5f/OTy8rK+NGPfgQ4LdQAvvKVr0RimTGN\ntP0Tt8/ly5e1CYmYhv6QFnV2zQZGcqklgFNUVKQNpSUlyB+Ryvc109swDMODiGmUYoqIeehv6pwM\nxzp9+rR2BZEgjiTr2sk8EGk7Jx2C3njjjUHf+7WvfU21+D/7sz8L/+JiHAk6yt5KmtXUqVMDdlaS\nlCGxkszsDoxM/9y9ezfgWJpiVfpandHCNErDMAwPIqJRdnd368ks/fp8kUFZ4jtLTU3loYceAlw/\nmtSBm0bZl9raWm1mGih9QjT6lJQUnn/+eSD8zU7jgdu3bwPu9Sd18fPnz1f/o/gvq6ur1fIR35n1\nmwyMXJfFxcWAu89lZWV885vfjNq6+hMxQSlO8P41sDU1NfzTP/0T4AyJB3jqqae0ltOc4YEpKytT\nE7D/5D9fRJhOnz6dV199NSJri3Vqamo0D1LMP98DRw54ibxWV1fzyCOPAO51a/XcgZE+D7KXkg0j\ngbLRgpnehmEYHkREoxw3btygYfwLFy6otiPdg+6//3410U2TDMzkyZM1PUUqbHz5u7/7O8CtEPGX\nMmT4p7GxUfMg5XqcN2+eviaBHQlQVlRUaJrLaAhAxALSb0Dq5qUa58c//nHU1uQP0ygNwzA8iIhG\nGSgptKmpSSsbxH/5wAMPhKWdezzir4en8LOf/YzPPvsMgDNnzgDoBEbDm4yMDNUMfYdeAX2mfkqF\nzrhx49QXbwwN8fn2txz37t2r1o/sfW1tLRkZGYBbwRMpTKM0DMPwIOodzrOysli1ahXg+oEWLFhg\nJ3MI2L17t84+l8Fts2fPjuaSYgp/YzP8IRpPRkaGakiGN+Xl5dy4cQNwO15JYv/NmzepqKgA4OWX\nXwb8pxZGiqgLyvb2ds2ZzMvLA6CgoEBVawvmDJ/U1FStELHGF+EnMTHR77x6wz/t7e1aVSY9IDZt\n2gTAF77wBb761a9GbW39MdPbMAzDg4RgNLaEhIQqoCR8ywkreb29vZOjvYjBsL0NHzG+t2D7G06G\ntLdBCUrDMIx7ETO9DcMwPDBBaRiG4YEJSsMwDA9MUBqGYXhggtIwDMMDE5SGYRgemKA0DMPwIKgS\nxuzs7N78/PwwLSW83Lhxg+rq6oRor2MwbG/DRyzvLcDJkyerR3PCeSzv71Cv3aAEZX5+PidOnBj+\nqqKI1JOPVmxvw0cs7y1AQkLCqK56yc/P5+jRo9FexrBYvXr1kN5nprdhGIYHJigNwzA8MEFpGIbh\ngQlKwzAMD0xQGoZheGCC0jAMwwMTlIZhGB6EbGZOa2srAFVVVTqnJdAoVWN4yD7LsDAbwhY6ZG/r\n6+sBaGlpoaamBoDJk518b5mJM3XqVJ3rNHbs2EgvNWaQPWpvb+fcuXMA3Lp1C4CamhodB5yTkwO4\nMmPq1KkqR0bD/oZMUN6+fRuAK1euMG3aNMC9ia9fv057ezvgTF30fZwwYYJefMPdkJ6eHhUc8hiP\n1NfXc+HCBcC9cQsKCqK5pLhCrlHhvffeU0Ep11V2djYAubm5epPLrGl/yASBjo4OxoxxbrcpU6YA\nzhx7+bx4PfC6uroA+OCDD/jJT34CwNmzZwFoa2tj1qxZgHs9z5w5E3AG4yUnJwOB72kZ4tbd3a0/\nK5NHZ86cqfsrnzVczPQ2DMPwIGQapajYiYmJVFZWAmhZ0wcffEBJiVOFJdqmaEKzZs1i/PjxAPqY\nkpKip4icGHIyzZgxg2XLljmL/+0JXVtbS1paGuDOWI5HampqdB+rqqoAZ98nTpwIuGahaOtGcGRm\nZgKuZZOdnU1jYyPgXn/V1dUANDQ06PUq12hycrJqpf3H1hYXF6t2mZubC8DSpUtZsmQJEL/z1ouL\niwGoqKjQe1ru/UmTJtHW1qavAxw/fhxw3CDiCpFRtomJifo9iFkuj5/73OdYsWJFn+emTZum7zeN\n0jAMI8yETKMUjW7u3Ll66nZ0dOhrcnK2tLQATtcOgFOnTtHQ0ABAZ2cn4JzscvrevHkTgDt37gDw\n2GOP8fWvf73PZyUkJLBhwwYgvjXK1tZW3RdxiJeXl+s+i1Yv38WUKVNUg5ef6+rqUg1U3i+nbmlp\nKStXrgQC+93inQkTJgDwxS9+Uf8tiEZfU1Oj2pD45ydOnKjWlPjGhCNHjuhr5eXl+rxol/GK+CAT\nEhJ47bXXBrwue3fy5EkALl26BDjXusgD0dI7OztVK5Wgz/Xr1wE4d+6cauXp6emAYxnItS6a/XAJ\nmaCUCyM7O1sXvGrVKgC+853v6PtEuJ0/fx6AsrIyrl27BrgbkpSUpMK2sLAQcAQwwPbt23Xz9+3b\nBziCQdTteEQEoVwAAE1NTQAUFRWpeSiCT1wSOTk5eqOLKdjd3a1miLxf9r+srIzf+73fA+D5558P\n3x8UI/QXkgB5eXl9HvsjwZ/+7o+mpiYOHDgAuNH1zMxMPcjiFbkv582b5/f16dOn93l8+umng/r8\nX//614BzsItJLyZ4cnIyoRrHbaa3YRiGByHTKIeKOMDFxJPH4SBaaUFBwaAnfDwgp3J6erqmPogJ\n0tnZSV1dHeA6vUWruXLlin6G78kqprZo8HIqp6amsnXrVgB1h4h2Cv41rHiku7sbcDXuYBgskHbf\nfffpdyDW17PPPsvUqVOHuUoD3JzXxx57jEcffRTom04k17G4o4aLaZSGYRgeRFyjDAUSyJA0odHe\nYTtUZGZmamKyVDKsW7dONcPm5mbATR0qLy/XgIO85vvvI0eOAK5jfNGiRaqxipO9ublZtfV7RaMc\njibpxc6dO9VH+corrwDud2gEz969ewE3uLZhwwa/iekSzBmpr9I0SsMwDA9GpUbZ0dGhyeqS9iPU\n1dVx+fJlANasWaPPS/hforvximiU8jhp0qQB7/Et65KIuaRadHV1aRKwvCYRyeeee07TVSQta9y4\ncepXNoJH/L+nTp3S5zZu3Bit5UQc8fcOhmiBQ7lv29vb2bFjBwA/+MEPAPjjP/5jYPC+Er7ZHiNh\nVAlKSQk6ePDgoDfnqVOnNP1ITMHu7u6wmEuxilwciYmJA+rnW1tbtQpCLlJxYSxYsEDNdgkMLViw\n4J4xucPBnj17AKfy5KWXXgLQnGLDNYkDCTI5+A8cOMBHH30EwAMPPAA4h/tQPn+kxLf6ZRiGEQJG\nlUYpFQuNjY0DzBPRNpOTkzXhXDBtcuhUVlaq60IS1UXDqa+vV+e4VOZIVxcjON59910Ajh07BjgJ\n1d/85jejuaSYRYpUSktLNXApVT7+Aji+briRVuQIplEahmF4MCo0SkkKLSsrA5z60P4125LSsnTp\n0sguLk6QJPOioiKtOZ4/fz7gdGQCR9uUEzve+ySGC9FgJHgjvrcXX3wx7ssVQ41YimLlnDlzRgtU\nxK/ui/gjwxHQjbqgbG9v1yisRGbXr1+vr8sfL9UnvhU48lo8N+sdKbJHcggVFxdrzbjsswjHuro6\nbX4q+xzvWQShRkxuiXZLNHb79u1RW1MskpSUpI1wJIBz+fLlgLXg/eXASCPdvthdYBiG4UHUNcpb\nt26pRrlw4cIBrx88eBAwTXK4SJrPiRMnAEd7XLduHQD5+fmA4yQH5wQW83A0zCmJNTo7O3n77bcB\np5k0oC0B+7ddM/wj93RHR4dq5b/85S8BJ9939erVnp8RSk1SMI3SMAzDg6hplOIzu3TpktZj+ibi\nSl2sBCGsLjZ4Ojo6NKggAZycnBwNiElPS0kyz8rKMs1nBPzt3/4tn332GeAOuPJKiDb6Ij7xvXv3\n8s477wAkQ4x4AAAPoElEQVTutfv9739/0EKU3t7esFqYplEahmF4EHGNUvwHolGWl5drD0ShtLRU\nE8yfeOKJPq/19PRYJNYDSVG5efOm+ibF57h48WLNLuhf833ffff16T9pDA0ZY/DRRx9p79Bvfetb\nQHyPJgklck/LiJijR4/qHPBvf/vbADz++OOD/nxCQkJYfJNCxO4KCcBITpTcpGvWrNE8PuHQoUNs\n2rQJYMCIBwvgeCOt1Y4cOaL7LvXx48eP5+LFi4CbvypjNmx64/D4z//8T8A54F999VUAPv/5z0dz\nSTFDf6Xn8OHDgBPAkVQgaXwRTUw1MwzD8CBiGqUkjMvUNDEFfQM40ix23bp1A1rkS1DHTMPBkcRx\nCeCUl5eTmpoKuIPeampq9LsQs1BSgmxvg+Nf/uVfAHcW9bJly/xOGjQGRyxECd5++umngHMtSmqV\nP/q3VQyn2Q2mURqGYXgSERWis7NTnbSiNYqG02cxv9Vo/A0CMm1ncMQPKWkUZ8+eBRxf5aJFiwBX\ngx83bpw2+5WEc/NNBs+hQ4f4t3/7N8DVbl599VUL3gRBUlKSdgyTwhKxhtauXat+dV8kEBnpgoiI\nSJ/GxkaNcktwRmZLgzsDR27gtrY2zZcayUS8ewWZE3369GnArbQZM2aM1nX7Nt+VvZQgmgXIgud/\n//d/dR76k08+CWBNL4Kko6ODoqIiAA0wShOWZcuWaX61uIpSU1MHNGkJt8ktmOltGIbhQUQ0ypaW\nFk1ZkZksYva1traqtllYWAjAnDlzVLWWU8UYHDlxxb0hjzNmzNAgmGjtiYmJ2tGmf+qVMXSOHj2q\n1WIyBdR3hpPhTXNzs1pDYkGKdr5582Z9nz93RqQ0ScE0SsMwDA8iolFOnDhRex/6+iHBSUCXEP/y\n5cv1PTbQauhIU2NJD5LW+RUVFZp2IZ2ZcnJyBiT4G0NHugKtWLGCtWvXAm6vSRmfYQyN7u5ulQev\nv/46wIC0QF96enpUk4x0dZ5plIZhGB5ERKPMzMwkMzPT72sLFiyIxBLiGkmnktN4+vTpgNMdSHyT\n4pecNWuWRblHQH19PeDMm9+yZQtgvSaHS3Z2dlB7l5iYGLLxs8EyKpMTw90yKZ7o6enRoIxMpxQT\nvLS0VC9EMbezsrIsJ3UESGDhueees+bGEcI3RVBS2yyYYxiGMcpICEaVTUhIqAJKwrecsJLX29s7\naodU296GjxjfW7D9DSdD2tugBKVhGMa9iJnehmEYHpigNAzD8MAEpWEYhgcmKA3DMDwwQWkYhuGB\nCUrDMAwPTFAahmF4EFQtW3Z2dq/0kYw1bty4QXV19aiti7S9DR+xvLcAJ0+erB7NCeexvL9DvXaD\nEpT5+fk6cS7WWLlyZbSXEBDb2/CRn5/PiRMnor2MYZOQkDCqq17y8/M5duxYtJcxLPzN7vKHmd6G\nYRgemKA0DMPwwASlYRiGByYoDcMwPDBBaRiG4YEJSsMwDA9CNhNA5vM2NzfrhDSZtNja2qqvp6am\n9nnMyMjQUQbWWt8/PT09AHR0dOjeJicnB/UZ1nd0eHR2dgLu/sk17Ttv3saWDI7vtESZFir7JbO8\nvZDrP5qETFA2NTUB0NXVpSM9r1y5AsC5c+d0/KwMwpLZI0lJSSo0A81ykdcmTpyoglW+hMzMTJ0N\nE6wAiQVkTojs03AIxc18rwnbixcv6s0s15UMaQuW3t5eVRZkHzs6OkhPTwfc7zjeECF35swZPv30\nU8Adp5yfn8/cuXMByM3NBWDKlCkDPiMUo2lHKmzN9DYMw/AgZBql70kgGl9hYSHQ16QW9VtO0MbG\nRpX2cuImJSXplLX+J8GECRN0LKsMnM/Ly1ONNR41SiP8tLW1afXOgQMHALh58yaTJzuVg9OmTQNc\njTIlJYWUlBTA1ZCSk5P1GpZrXrShlpYW/XdWVhYAkydP1utbnos3rl69CsCxY8c4d+4cAL/5zW8A\nZ0/679OcOXMAyMnJIS8vD3BlQE9Pj1qWIj/kMSMjg0WLFgHw8MMP93ktFJhGaRiG4UFYBjx3dXUB\nri9myZIlqnGK/1J8lR0dHaqByjzq7u5uPUWqqqr6fGZ3d7cOoRfH+rx588jMzAzHnzIqqKmpARy/\nbij8NcZAysrKaGxsBNz9njp1qvoQ5bq9fv064Gg34vetq6sDHI1SPmPevHmAqyldvXqVgoICAPWn\n33///UyfPj28f1iUER/vF77wBZ599lnAnTFfXFxMWVkZALdu3QLg8OHDAOzdu1ctRpEL7e3t+nkS\n45DXWlpaWLdunf4b4Omnnw7Z3xEyQSlCsauri9u3bwPuBecbCReT+s6dO84CxowZEAmvq6tTIagL\n/a3K3dbWxscffwzAggULAHjyySf1oo3HgMOFCxcAqKys5MaNGwCcP38ecG5c+dvl5pQLccGCBfrc\npEmTAOcGl4NJDivfCO69hgjAuro6DUhu2LABgDVr1ugBL9dtcXEx4Ag+uc7lmqutrdUbOCcnB0C/\nr5kzZ1JZWQm4JuGYMWPUZRSvzJw5c8Bzr7/++oDnJLtAzPPfdvUBXHdabW2tHlzy2kcffQQ494bs\nq8iW0tJSv79/OJh6YhiG4UHINErRarq6utQMXrZsGdA3/080RVGPOzo69DNEs7lz546a19LnTk7t\nDz/8kN27dwOuBjXclI1YQbTp5ORk1XDEZPH9t5gvsselpaUDUqkADULId/bYY48B8JWvfIVnnnlm\nwO8X0zIe3Rui0SUmJrJ06VIATVnxTVcTbUWuOXn0QjSf8vJyioqKAHjooYcANFhhuMGvFStW9Hn0\n4uWXXwYc60isLAkYi8UUCkyjNAzD8CDkwZzU1FTV8CSNJykpSTXChoYGwJX6ra2tqlWKpjNp0qQB\nWfvipysqKtKTfvbs2UD8V0YsXrwYcLRw0WxWr14NOL4Z0VpE8xN/WlFRkaaryL6PHTtW/TyitYsT\n/JFHHtHfKa9duHBBv8d41CjF/7VkyRINIAYqfBju558/f1730TTJ0OFrTS5cuBBwLdNQygXTKA3D\nMDwIedQb/JfaiXSX10SLTEpK0pNcfGtpaWmagCuf+8tf/hKAkpISNm7cCMDWrVtDtfxRjZyQvtFp\nSTEpKChQDVL8vrLHDQ0NGk2UR9/vRlIr/EUGRUttb2/XiHk8I75cLyRNzUvrPHXqFAD/+I//CDjZ\nCb//+78/ghXGJl6lg8Gmux06dKjP/8Uaam5u1iwEyYYJJWHJo/RH/7Qd32qd/ps1bdo0vfnPnj0L\noPNk0tPTNR9LUjD8ff69hJjj/dN8fKs9fM0QOZBE2PoiAlXSWnJyctRUN7wFpKQP/eIXvwDg9OnT\nACxfvpwXX3wxvIuLQYZSgy339tGjRzXP8qmnnurznqqqKr8BtlDJBTO9DcMwPIiYRhkIkfqiGY0d\nO1ZPmnfeeQdAp7w9/fTTqlEawdPe3q6BNAme+SIn9t27dwFH67RqoKFz8OBBwK1nlhSVb3zjG1Fb\nU6wiVpBUO1VUVPDggw8CMH/+fMB1N40fP97vdWoapWEYRoQYFRqlaC++gQbx7YjzVvxkW7du1RI9\nwxvRzOVkraqq8qtJglNyKilG0sXlXi5vDJaSkhIuXbrU57k1a9YAblK/MXREo5R+D8nJyZoWJ4hG\n6a+PZSjjFlEXlG1tbRp9FWHY0tLCz3/+cwBtffX8888DTnF9f+7lQE4genp61J0heZHJycl+gzjg\nBCLEVJQGDtZ1fugcO3aMzz77DHCbQZjJPTwSExO19l5yqKdPnz4go8ZfMEjkQSjlgpnehmEYHkRN\no5SToLGxUTuuSErQxx9/rF1BJAVIAji+pqBpkoHxbVwqGo6/fEEJ7tTV1Wmd82DmuTEQSQk6fvy4\n1nNL+sry5cujtq5YRMztnp4ejhw5Arj7u379en2fpK9JLwhfwiEXTKM0DMPwIGoapVR+TJw4UU8R\naYr6wx/+ULPsxTd5r1ThhALRzH2TowNVnogPaNy4cRYoCwKxiiSlaseOHToy4rnnnovaumIRkQHy\neP78eT755BPArb7JysqivLy8z/t8CaeFaRqlYRiGBxHXKKVWVqKpY8eO1eiWRLpPnz6t6SlyMvt2\ngjbfZGCG2v1GBj9VVFQAsGjRIhvOFgS7du0C4Kc//SngZGs88cQTgKsFGUNDNES5Fj/66CO1gu6/\n/359n6QSSlaGL+GUCxETlGIOyngIaY/U1dXF0aNHAfjRj34EOPmUL730EuDmoRne+DO5/SFuDUll\nkaYX8T6/JZQcP35c+w+8//77ADzzzDO88cYb0VxWzNG/mkaa77a3t/Paa68B6CTMxsbGQa/RcCtP\nZnobhmF4EDGNUjLoJe1E6jerq6v57//+b8AdZfDCCy+wZcsWYGAQwszuwRFNUhL4BzOjT548Cbga\nqLg5zOz2RgI4Fy5c4N133wXctl5/+Id/GLV1xTpS0SS18ps3b1ZNUgjUxco0SsMwjCgTEY2yvb1d\ngzjiD5ORkj/84Q/59a9/Dbg+si1btjBr1qxILC3m6e3tHeDnCdQCv7KyUntOSv8+GVFgeCM+tP/5\nn//RwMOf//mfA/Dwww9Ha1kxSWJiolqW4u8V60YGvQE6zsRfQ/BIWZgREZRdXV1aQyzBnMuXLwOw\nf/9+veAkV3L9+vV9qkrATO7B8NdaKlDO5OTJk3W2iMxuCeWMmHhFZnLLDX3w4EFt9eU7a8gYOl1d\nXXrwiOIk0xcLCwt1ztPatWuBvoIyHPXcgTDT2zAMw4OIqBK+83XlBJAcvsLCQnXaysmRm5s74P3G\n4EjwRk5gfy2nhJaWFt1ff6aM4R9p+/fmm28CTlDnS1/6EmD13MOlsbFRc6iljlsq9mpqatQVd+3a\nNaDv6JJIywXTKA3DMDyIuHNKfGrSESQnJ0d9k1LVYAyd7u5u9Z+JZil+n82bN1NWVga4qVddXV2m\nAQVBYWEhAPv27QPcypANGzbw6quvRmtZcUFLS4sGbSWII8UQt2/f1lSsQBZSpDCN0jAMw4OIa5QS\n6hcfw7Jly1i1ahXgv7ecEZjLly/r4DVJ6i8pKQEcLUjSL8QvOWvWrCGNCDUcJJ1NUqnkun3hhRfI\nzMyM2rriAX/z5KV71WjrYhUxQSk3rMxkkaYBc+bM0ZvY5rMET2dnp+aoyt7KzVxeXq7NROSmXrJk\niTXlHSJ3797l5s2bgFtvLG4i6VVgRJZoBXfN9DYMw/AgIRgJnZCQUAWUhG85YSWvt7d3svfbooPt\nbfiI8b0F299wMqS9DUpQGoZh3IuY6W0YhuGBCUrDMAwPTFAahmF4YILSMAzDAxOUhmEYHpigNAzD\n8MAEpWEYhgcmKA3DMDwwQWkYhuHB/wPu2gEcRxkXGwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2cc21a2e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_output(values=layer_output1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of Convolutional Layer - Method 2\n",
    "\n",
    "Keras also has another method for getting the output of a layer inside the model. This creates another Functional Model using the same input as the original model, but the output is now taken from the convolutional layer that we are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_conv2 = Model(inputs=layer_input.input,\n",
    "                     outputs=layer_conv2.output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates a new model-object where we can call the typical Keras functions. To get the output of the convoloutional layer we call the `predict()` function with the input image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 14, 14, 36)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_output2 = output_conv2.predict(np.array([image1]))\n",
    "layer_output2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then plot the images for all 36 channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAADuCAYAAACEaORrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmUFNXVwH/FMAMDDIsM+zIDCAIqi4K4gohbIhoxmojB\nJSoeTeDk4CFGRT/M0WMWjdGgMQoxeojrMa6IIgjigqiMsimiCIMgggyLAzLDDDP9/cG5VdVDT09V\n96vq6uH+/uk31W+q7uuqfn3ve3exYrEYiqIoSno0ybQAiqIojQGdTBVFUQygk6miKIoBdDJVFEUx\ngE6miqIoBtDJVFEUxQA6mSqKohhAJ1NFURQD6GSqKIpigKZ+OhcWFsaKi4sDEiU5paWllJWVWUFf\nJ9EYa2trAaiuro47npOTc0jbslIXMZNjDJOSkpKyWCzWIejr+Bmn3GNw7qG8eokSrHvfw76XiWRM\n51n0QlhjBGec7vtUU1MDQNOmB6exoMbrdZy+JtPi4mKWLVuW8L0VK1YAsGnTJgDy8/Pt944//ngA\n2rZt6+dycQwbNizl//VDsjGuWbMGgNdeew2AH374wX6vsLAQgPbt2wNQWVlpv9eiRQsAOnXqBMCQ\nIUPi+gphj7GiosI+tnnzZsAZkzyg3bp1s/vk5ubGncf9d8uWLT1f37Ksjf6l9k+ye5kue/fuBaCq\nqgqA1q1b2+81bdo01Hv58ccfU1ZWZh/bvn074Nwfee7cMpogrDEC9OzZk0WLFrFu3Tr72LfffgvA\nnj17ABg/fnwg1/Y6TjXzFUVRDKCTqaIoigF8mfnJGDx4MAAfffQRANOmTbPfE7OjqKgIgIEDB9rv\n7dixA4DevXsDcMkllwBw0UUXmRLNGAMGDIh7dbNv3z7AMell2QPgiSeeAGDr1q0A/PSnP417BTji\niCMCkDg57qWYvn37hn79oKmtrWXPnj2Ulpbax+S+7Nq1C4ADBw4A0KxZM7tPu3btADjyyCMBZwkH\nYNu2bQDs3r077lrDhw+322Hey5qaGvbs2WPLBTB//nwA5s2bB8AHH3wAQHl5+SEyyneyX79+9nun\nnnoqABdffDEAnTt3Dkp8zzRp0oSCggKGDh1qH5P2u+++Czjr3k2aZEZHVM1UURTFAMY0U2HixIlx\nr4n44osv7Pa//vUvAB544AEAnnnmGQBeeOEFu8+5554bt4sXRUQjFURTB5g6dSoAkyZNAmDRokUA\ntGrVyu5z2mmn2VqSYgbR2r788kv72MKFCwF4//33Aez33DvBsjOel5cHwCmnnGK/Jxs9orVeeOGF\nQPyG44EDBzx5AJggJyeH1q1bc8wxx9jHpD1lyhTA2aCR5w5g6dKlca+ymQPOZyKf0XHHHQfEbyC3\natUqtDE2xGmnnZbS/4n8prwAVDNVFEUxgC/NtKamhl27drF69Wr7mKwvuX8ZG6J///52+/777497\nFXedLVu22H12795t+5QFTUVFBatXr45bgxozZkxa5+zatSsQr23XZd++fRlb6wkC0eDc641hk5ub\nS9euXfn5z39uH3O3U0HWWkWbqc/dL2gfTz8UFBQAcMEFF9jH3O1UidIYU+G3v/0tACNGjADgyiuv\nTOt8jefbqyiKkkF8aaa1tbVUVVXZO/AAs2bNApx1Ftn5k7UkgMsvvxzwtssmO8x9+vSJO17XYTwo\nmjdvTr9+/eKc1WVdSXbx27RpY/y6LVq0aBSa6ccffww4Tu2jR4/OpDjGEUssKtTU1FBeXm4HlADs\n378fgJEjR2ZKrKzgn//8p9HzZf+3V1EUJQLoZKooimIAX2Z+bm4unTp1ijPhpS2O0Z9//jkAJ598\nst0nm8xXy7LIy8uz3WLAic0VV5GSkhIgPvGJbLQsX74cgAcffNB+LyouJGHw2GOPAXD99ddnWJKD\ny1IVFRV23gFwXNjcyzimqa6uDu2eW5aFZVlxG32yxCKbgBKTL3kzwAmWOeGEEzxfy/28h7Xslk1k\nzyynKIoSYYw57Uuqs0ymdgsKyaAkLhSjRo0CnIV+gPXr1wOO87OEjnqlsrIykhqsO1hCggrcWrtw\n9dVXA9hZmh5++OEQpEuOZVk0adIkLqOVjGHnzp2As5noTqcoiLXl95nOzc0NzW1IwizF/SkZ7k3d\nuhu8XlBtNDmqmSqKohjAeDhpqpgO7QqC5s2bx/3tTo4hblP/93//5/l87nyo7qQjUcK93p1IIxWe\nfvppAJ566qnAZfKKZVk0a9bMDprwilgV3333HeBNM/3xxx/ttp/crkEiltOGDRuA+Oe1V69eGZEp\nCGQtN9Oas2qmiqIoBtDJVFEUxQCRMfOjbN57QUx2P9FRsrEFB03obPsM3PlYJQfmuHHjMiWOMcS9\nT7IleSEqpr0bcX8S2Xr06JFJcQIj0+a9oJqpoiiKAULVTDNRQTEs/GikkhkrqptODfH2228DsGDB\nAvuYxORnC7I5496UEc444wzP55FcoV5ck8JCqj7Id8udN/dwwJ1xzu/mYzqoZqooimKAUDVTKYsr\nNWrAyRWarVqaHyQbu7gbZVOYrRvJBCWO+hBfWSDKSGip1HASdy+39uZFm5FgBtFI3VZXpq0t0ba7\ndOmSUTkyhfv+SR7kuXPnAnD++ecHdt3s/DYriqJEDGOaqVQ+lKQKiZBfzLFjx5q6bCh88803cX/3\n7NkzpfOIY3f79u3TlikTSPKWjh07AjBjxoxMipMSorV07949rfPUXf/PtDbqJlFo7OHK888/D8Cl\nl14KOJVMwfFAMYVqpoqiKAbQyVRRFMUAaZv54oYguT7Fed1tCp999tnpXiajSKYhyYWZah6BqJW8\n8MuTTz4JwI033ggcWt46GzC16aemdHYgG45SQNC0ae9GNVNFURQDWH5yaFqWtR3YGJw4SSmKxWId\ngr6IjjEUDodx6hgNkg3j9DWZKoqiKIlRM19RFMUAOpkqiqIYQCdTRVEUA+hkqiiKYgBffqaFhYWx\nuvVwZANLEj/I325/PhO+faWlpZSVlQUes5dojGFxOIwRoKSkpCyMXeBE45TEF3WfV7fPsPiQpvPc\nHg73MqwxQnaM09dkWlxczLJly+wHEmD79u0AfP/99wCsW7cOiH84JRO2ZJZ3Z4iSInUdOhz8bkk2\n8LrZs4cNG+ZH1JSRMXpBclmCMyYZYyIvCcmaJZ+f+3OwLCuSYwwCy7JCcXGRcUrJZoDFixcDsGLF\nCsDJ/ekulihBGoK7IkJRUREAvXv3tq8B0K1bN7tP27Zt7bLgQZPJexnW8wr+xvnll1/abckZUnee\nAef7Jz+e8iy473e7du08j1PNfEVRFAPoZKooimKAlGLz3XHJnTt3jns9+uijAcfsBygrKwOcuH0p\n9OU+V93a19lQ/CtZqYpEcfuSglDW66KUtq0xUltbS0VFRVzi5+HDhwNw4oknAs7ztnPnTrvPe++9\nBzjmorsMhizfiFmfKKn5/v37Ey7zZCuyPCXPLcQvi0SNfv36HXJMxiBzEThzjpSw+fTTT4F4M3/E\niBFx406GaqaKoigGMF62RDRNd8mEw7V8Ql02bjy47yK/iEceeWQmxQkU+bV3/6qHXZqmSZMm5Ofn\nx123sLAwYV/ZUAIzGyvZWpImEVLaRcruZCMyhmQlaSS7nWxEwcHMaF7vZeO544qiKBkkJc3UveYp\na0P1/eIrDrfffjsA06dPz6gc1dXVbNmyhYULF9rH5s2bB8DatWsBp3SzW6uT4ofyCz558uR6ryFa\neKL1KyU7icI6qRRCBEf7T1YqKRVSzdOrmqmiKIoBfGmm1dXVbN26lQULFtjHZOdTdu83bdoEwFdf\nfWX3kV38/v37A3DTTTfZ78nu/wknnOBb+GxBPpO+ffsC0KdPn0yKQ25uLl27dmXChAn2MXc7HSSz\n+eeffw5EQzN176xLUMm2bdsAZ01X1rHB2fmVygiy8x9lJHgG4h3TGwuxWIzKykr7/gHMmTMHgFdf\nfRWATz75BIhfr5b9GrGmHnvsscBkVM1UURTFADqZKoqiGMCXmZ+bm0vnzp0DMQ/r4napyYSbSUVF\nhd3+5ptvACemXpztf/zxR7tPMnea8847D4C33nrLuJxRQ+rRp1uX3iRitoMTdy9LL1JH/fXXX7f7\nSCy/OG9PmzbNfu+qq64CnJj8RHh18jZJuhvAb7/9tt0+/fTT0xMmACzLonnz5nHfM2nfcccdALzz\nzjsArFy50u4jARdBzVNuVDNVFEUxgHGnfVNkyuk5FotRVVVlb5oB7N27F3BcwuTXrm3btnafuprp\nyJEj7baMpTFuDGQDEsYLjkbZsmVLAI444ggAjjvuOLvP119/DTibo361mh07dhySeSpoUg1NPumk\nk4B4Z/YoaqZekO+c+7uXLn6sDNVMFUVRDBA5zVTCEN3aRJhYlkVeXp6duAWIa4OzdupO+FIXWYsD\nZ32uMbN8+XLAWSuNahCHBCF07NgRcKwO0VQBTjnlFAAGDBjg+bzffvut3e7YsWPSZyMKSD7XpUuX\nAonz7yYjE+vCYSFzEPibh1QzVRRFMYBOpoqiKAaInJmfKfPeC17Me1nQnzJlin0sSm5CQfHhhx8C\nMGTIkAxL4g3ZYJRsQu6cp37Me0HOAwddCKOeq1aiuh5++OGU/r8xZcUSZOki1bE1vk9EURQlA4Si\nmYrTtPvXO5U+mSaZRiqamWRL+uCDD0KRKZPcdddddvu2227LoCTekWCMNm3aAI4WkmpuWYmJFxer\nKPPoo4/abQlIuP766zMlTmSpW8zTK6qZKoqiGCAUzTSZtinrkFHWSL3w0EMPAXDvvfdmWJLgEdeR\nbKwUUDfbf6o5OqVmlGi2UXaFkiCTW265xT7mLlN+uCOfhds9LhVUM1UURTFAxnfzo/yL7gfJx3rZ\nZZdlWJLUKC8vB2Dr1q2AEzor3gluXnvtNQAuvfTSkKSLHhIuKs7/UWbq1KkAPP/88xmWJJqIB1G6\nHgqqmSqKohhAJ1NFURQDpG3m79q1C4D58+cDTu7Pc845x+5z7LHHAk4JVTEp4dC492zCnc/0ggsu\nyKAk6SOx2eIylKzcSLaP1Y24SkmmKHA21pJtTrVv3z5YwQywZs0awHHQHz16tK//l6UMcaOKGhJ4\nIffwu+++A+LN9cGDBzd4HlOb36qZKoqiGMDyky3GsqztwMbgxElKUSwWCzwhqI4xFA6HceoYDZIN\n4/Q1mSqKoiiJUTNfURTFADqZKoqiGEAnU0VRFAPoZKooimIAXw5khYWFsWT1wt1IAhNwKiemE65V\nWlpKWVlZ4Bl3/YzRNIfDGAFKSkrKwtgFPtzvpSQ7llfT/qJhjRGy4176+nSLi4tZtmyZp77ivA+O\ns34i51jJaCMO8JLVx52RqHPnznbse9AkG2NlZSUAGzZsAOILb0mm9hYtWgDElfqV/IhdunRJeu26\n5aKDws999Ip4hSTLMC9f6pycnFBcXIqKiliyZEmcTHVzVXqpnpAKUbiXq1atAqBbt26A95yr4gRf\nN8NWXcIaIyQf5+7duwEoKysD4rM/SXn1dH5IvI5TzXxFURQD6GSqKIpigMCCbnv27HnIMYmldZsP\n0l68eDEAc+bMAeLjiEeMGEF1dXVQonpGYrUTFVxbvXo14Jj5b7zxhv3e3LlzAWesEts+cuRIu4+Y\nYtmKlwJyklA5TJo0aZLUxGssKSDdfPTRRwC0bdsW8F9SpSHzPirIstu7774LwLPPPgvA5s2b7T51\nv7Puz0LGKUtyQ4cOBeC4446z+xQUFOA1sEk1U0VRFAP40kxra2vZu3dv3DF3idyGkIxEbmRTRhIN\ny+u2bdv8iJZxjjnmmLi/r7vuOrs9cOBAAF5++WUA1q5de8j/jx071t6gaayEXcbbsqzAMh6JlZXo\nmQ6T2tpa9uzZw/r16+1jom3LZpt8Z2WDBhy527VrBxCnfS1fvhxwNLUo4ZZTNg87deoEOBmi3Pd8\nxYoVAMybN++Qc4k1KOeRROjuBO+jRo2K80xKhmqmiqIoBvD1s92kSRNfmmg6yK+NkGr5Vb/U1NRQ\nXl7OPffcYx8bPnw4kHoez1NPPTXuVZDSIHBwbSbdsglRp6CgINMiGENKt2S6TI1lWeTn53vK25no\nuyuam7sETa9evQBnrT8R+/bty4gl5V6bFxcocZsMyn3Sq3XTuL+9iqIoIZH2gpLssoelOQZNTk4O\nLVu25M477wz8WtmQrV2Jp3///gCMHz8+w5IcJN11YdFoJSu/V5o1a9boLSm/6KehKIpiAJ1MFUVR\nDJC2mV/XvK8bJwuO+8Xs2bMBmDJliv1eFDP9N0ZH7iC499577bbUZm/syPM6ffr0DEtykFgsRmVl\npV3YEpzvnhSrlPj0RIib3lFHHeXrumF/R2pqavjhhx/s4B6ARYsWAU4CJXFB3LNnj91H3PFuuOGG\nwGVUzVRRFMUAxj2aJYRNXsEpB/3iiy8C8Pjjj5u+bCiI1i1jczvfy8aEuJg8/fTTh/z/gw8+CMCk\nSZMClTNoxD1FSnhD49ZMZ8yYYbeffPLJDErijaqqKsApXy1Z2xYuXGj3ufrqqwHHcd0dghlFcnJy\naNWqVVwggYSGSuY5ef3+++/tPqKtikbr1lrHjh3b4HX37Nnj2QVMNVNFURQDBJboxE3r1q0BZ+30\nyiuv9PX/Bw4ciMTaqlvbBti+fbvdvvbaawGYOXPmIf8n+U8bymcadSQcVkJnV65cmUlx0kK0NUlM\nk4ilS5cCMGvWLPvY5MmTgxXMJ5Zl0bx587hnq77nzJ0sSKyKVO9hZWVl6N/JnJwcevToYf/tboOT\nSMftKiZzj4zdixvZF198YbeLioo8y6eaqaIoigF0MlUURTFAKGb+p59+CsBLL72U0v83bdrUU77M\nsJk2bZrdvuOOO+rt98477wD+lzeixn333QfACy+8kGFJUkc2E7w8TxdeeCEAb7/9dpAiBU5paSkA\nDz30kH3sd7/7ne/zuMv0NGvWLDLfSclrKsuIieSSPsnyQ8jGlTsTWH5+vudIL9VMFUVRDBCKZhpm\n4a0wEJeoiy66yD7mrgwA8N5779ltP4vYUePWW2+127JZ07dv30yJkzaioSTbeJJMUOJWI25v2Yq4\nBZ177rn2sWuuucb3eRJVHI4Ckk0/GV4ylklRz65du6Ykh2qmiqIoBkhbMxUHYcmML7N7tv+aJ0Pc\nK5KtO0ldGoBbbrklcJmCwh188Nlnn2VQEjMk00gFGbM4vWcbokFKuXWpx5bqmr18x718dlFDcgYn\ny9Am91nWSlOtCKGaqaIoigHS1kzz8vIAJyxNaq00Js1UfuknTpwIQO/evQG47bbb6v2fbNZGAbum\nkIwVslMz8cq6devs9i9/+UsgfuzZRHl5OeBkzJdXv4iju1TvlO96NuElZ7B8v+sG5fhFNVNFURQD\n6GSqKIpiAGOuUeLYKg7CjQlZyD/99NMBuOKKKzIoTThIpq/rr78+w5KEgztGPRW3oSghzuvpUlFR\nATjx7Y0VWc5Jtyy4aqaKoigGsPxkfrEsazuwMThxklIUi8XqTxluCB1jKBwO49QxGiQbxulrMlUU\nRVESo2a+oiiKAXQyVRRFMYBOpoqiKAbQyVRRFMUAOpkqiqIYwJeXamFhYay4uDhhIa26+Q3d5VHl\nvXRyIJaWllJWVhZ4EkUZYzK8jD8VojTGICkpKSkLw6Umk+OM0r3M9ucVsuNe+ppMu3fvzptvvsmm\nTZvsYxs3HnT9kmqPUrvanRRBPgQ55k7UKskzpAa2UFhYaLfz8/NDSzBdXFzMsmXLkvaRtF4SIQJO\nRcicnJxD+kuiiL179wJOREndcghhjbF79+688cYbccfcn3fQWJYVir+gl3spteTPOOMMo9eO0vMq\nteIlPSY4JUjkeXVH/8hzKQlAJCVdq1at7D55eXmhJn33Ms6g8DpONfMVRVEMoJOpoiiKAXyZ+U2a\nNCE3N9fOlwiwevVqABYsWADAhg0bAMe0Bdi9ezfgmLnuXKfnnHMOAGPGjAEcc7lDh1Ci1FJC1pu+\n/PJL+5gsfQwePBiIz/0pppTUhZKxjRgxInhhE5CTk0NBQUHKGcWzhVgsRmVlZdIaQbK2v3TpUvvY\niSee2OC55X736NEjTSmDR5bVvNRBciPJbuR5dy9hHThwIOFa7OGMaqaKoigG8K2ZFhQUcNppp9nH\npO2uYpkKUkNK6rB4qTiYKWSzbPjw4fax2bNnA07tbUnXBzBz5kwAfv/73wNOirSnnnrK7nPWWWcF\nJ3AdLMtq9FopHBxnQ8/RmWeeecgxyTCfm5tb7/9JXaX//Oc/AFx11VUpShldfvjhB8DZQK5LVCqU\nfvfdd4Dz3WvZsqX9nmz2yga525ocNGgQAJ07d6733FVVVZ41cNVMFUVRDGAsOXS6dOrUKdMi+Ma9\nBvWb3/ym3n5Tp04FHDecTz/9FIAlS5bYfbp16xbnahU24ipTUlICOO5f7l9yke+SSy4B4Kijjqr3\nfOJW415nc/seR5lkGqlw5513AjB06NCgxUkZ0SzF2vOLe28kKmzdutVur1ixAoBPPvkEcJK4b968\n2e4ja9vS1504W6xpcX2SOnZimcBBdzD338lQzVRRFMUAvjTT6upqtm7das/yAN9++y0AO3fuBJzy\nD+4yEOKsf/HFFwMwYcIE+72uXbs2eN3KysrQdg5ra2vZu3evvf4Czi9VMk3MC3Pnzo37Wz4zOLjz\n7EUjMkEsFqOqqopVq1bZxxYvXgw4a9fyiy7VZsGRVzw1brrpJvs9Cd4QzUGCANz3Ld3qjyYQDVy0\n5Pz8/EP6yHt1gyrAed6lMq2sxWUasSTA8Z7p2LFjWueUNcUo4V7flLZ4BJlizZo1drugoCBhIE4i\nVDNVFEUxgE6miqIoBvBl5ufm5tKpUydOOukk+5g44peVlQGOy4h7YV7MQzEb3DHC9SGmChw0zdxB\nAEGSyG1o/vz5APzxj38EnM2VPn362H3uvvtu39eqm48g3eqIXrEsi7y8PI4//nj7mLudCgMGDIh7\nFdwbWO5AhjBxy7B+/XrAeRYlbv25556z+4wcOTKuj5t3330XgEsvvTTub9Ompl/at2+fsJ0K8ln8\n4he/SOs82Yo7YKhdu3Zq5iuKooSJb1XIsqy4OtrSlo0kU4vWbjea9u3bh6q15ebm2mGt4DjUFxUV\nAbB27VogcSihaN3vvPOOfUw2Y9xO/ocLbu07zMxUQm1tLd27d7f/7tevX8J+kyZN8nQ+0UiFTGuk\npnBvBJ933nnA4aeZiitZqs+paqaKoigGSFvdE9cXcZiV8D13SJc7D6JX6q4nhh265naZEZcoeT3/\n/PPr/T8Ztzs/ZjaFborrjzgwp0vd+xg2TZo0SbhWK3l3xYk7kYuU4HZed1tljYFXX30ViHfsl/Dn\nZEQ5yYnbyd6Lu2FVVZWR66pmqiiKYgCdTBVFUQyQtpkvJpBsGKUbY58s+iQbEPPHXbYlG5DFd1Ny\nS2RQptyhGkLuUzLzfs6cOQCMHTs2FJkywfTp0wH/rn1RyRiVCHHXhPhY/PooLS0F6t+c9Ep2zliK\noigRI23NVDQQU1mfoqiRVlZWAt5yrMqvot+s5plGxpjufZTY96jmoxXXNS+alWTQaoyaqeRflfj2\ncePGNfg/bnfFKH5PZS7yuuEtlQTE5TFdoveJKIqiZCFpa6Zu5/Z0kHDRsJzz/eDHtSmqa4QNYcqy\niLobmLiuuV336iPbrAs/PPHEE0B8qG1DRFEbdSPr4G5Xp7quURIKDk7Iupd1VS9E+9NRFEXJEiKj\nBkZRIxX87FxKUgS347B4PKSbgCITvPjii3Zb1hslpNJ9zyRhSGPS5m688UYgPulOFHKypsojjzxi\nt2fNmgVA3759MyWOMaQG1Oeffw44zyJAr169ACd5ieTdBTj66KONyqGaqaIoigF0MlUURTFA2ra1\nuEtImQuJ6U5ktouK7X5PTN+outKAszkmbj9eNi/cC9/ZYN7XzbEgJqE7Lv2aa64BnPsncfzgLOw3\nJjNfyFbTXr6bUtLEnW+hMbh7yfjk/owZM6bB//FSJilVVDNVFEUxgOUn+4tlWduBjQ12DIaiWCzW\noeFu6aFjDIXDYZw6RoNkwzh9TaaKoihKYtTMVxRFMYBOpoqiKAbQyVRRFMUAOpkqiqIYwJefaWFh\nYay4uDggUZJTWlpKWVlZ4BlpUx2jbOSlkzQ36mM0RUlJSVkYu8CJxhlW8vHD4V6GNUbIjnH6mkyL\ni4tZtmxZ6lKlwbBhw0K5TpBjFMd2yWrvzruYl5cX+hjFmRtg27ZtAAwcODDw61uWFYqLi4zTHY99\nzz33AHDKKacAMH78+ECu3Rie14YIa4yQHeNUM19RFMUAOpkqiqIYILp57xohkp4v07Xkd+zYweOP\nP87NN99sHxMz/6WXXgLgZz/7WUZkCwJ3WYpf//rXgFNATta4Ja1gtlFbW0tFRYV9/wA2bNgAOOv4\nkmrOVAJwJTGqmSqKohhANdPDkGbNmtGvXz8WLlxoHwtj4ykKHH/88QD873//8/w/X331ld2WHeW6\n5TAyhWVZWJYVV9549erVALz33nsAvP/++0B8li/J/NWnTx8AzjrrLPu9888/H4ARI0YA0KZNm6DE\n98yBAwcoKyuLO9a6dWsg/fLkFRUVgFPyecCAASmdRzVTRVEUA6hm6gHxTdy6dSsAjz32GABvvvmm\n3eekk04CYPTo0YCTAxXggQceAODkk08G4I9//GPAEienVatWtixKw/gt7VFRURFXFjlILMuiefPm\nHHPMMfYxaU+ePBlw1lDdxfPETU/c89yarZRAXr58OeBo4+5ikfn5+aGNEQ6WAdqyZQsvv/yyfeyz\nzz4DnHIlkmf4yiuvtPuIW1Mi96Y1a9YAB8cCqWukgmqmiqIoBoisZuouYrZx40Z7XSMTSLSMZOm+\n7bbb4l7dSJnZK664wj62YMECAH71q18FKqdfVq5cabfFsV3Wy0yxbt06u+2nrHA2s2XLlriCiplG\nisrJa6q4yyTX1NSkFe3nl/z8fAYNGsSgQYOMnVMK8cn3Uzw8UkU1U0VRFAPoZKooimKAUM182ZxZ\ntGhRvX2ewP+8AAAKXklEQVTWrl0LOG4dcNC8FvM56oibxjPPPGMfc7fro7y8PM6MChJZzHeb+R9+\n+CHgfO5i7i9ZssTuI65Ub7zxhudrySI/YNREixqXXXaZ3Z49ezbNmjXLoDTBIEEn0g7TzA8Ccf06\n44wzjJxPNVNFURQDhKKZisuBl1+y77//HojfrBgzZoztvtBYad26ddwvf5A0bdqUjh07xpX7veCC\nC2w53Lid+W+66Sbf15LsTJD5MNogkE2Lc8891z4W1n083KitrTWaOlGyUI0aNcrI+VQzVRRFMUBg\nmumzzz5rt7/44gvASbyQjHnz5gHQr18/+1iPHj3SDhkLixkzZgCOw3QUsSyLpk2b0rZt20PeE5ee\nadOmAbBq1Sr7vddff933tRqjNupG7re42TQG5J737t0bcJzhM006WumPP/4IEGeNzZo1K22Z3Khm\nqiiKYgCdTBVFUQxg3Mzft28fABMnTrSPzZ49u8H/e+655wAnDv7CCy+038uGBX1Z1oiyee+FF198\nEXAil1Ix7QEqKysBaN68uRnBAkYyBoETi54M2VS96667ApIoXCR+H5yN4qiY9+kgOQdk2apDB6f0\nmGTMMoVqpoqiKAYwrpnecMMNhxybMGFCvf3Ly8sBxxF83LhxALRv3960aMZx/5p36dIlg5Kkj2Te\nmTt3LuBkpPeLuLZ17NjRjGAh8Y9//MNu33fffQ32l3ym11xzTWAypYoEf8iGr+QuTYY7P6s7A1W2\nI7lc5f562QRPFdVMFUVRDGBcM126dCngaJwN8corrwDQv39/ID7jd9RxZ0TKJrkTUVJSAsCZZ54J\npJ49Kts0UsG9Rp8MWWf773//G6Q4aSHr1fv37wec0uI9e/a0+4hVtWLFCgAGDx4cpoihIS5rM2fO\nDPxaqpkqiqIYwJhmKr9wZ599doN9n3rqKbstCTZk7Slbdn8h+7VRN5dffrnv/3HX5BFNZ/jw4cZk\nCpORI0fW+96f/vQnuy07/VJZIYrILry8JgqckByeUhHCy/c2akhtrk2bNgHOvdmxY4fdR/YC/vrX\nvwYuj2qmiqIoBtDJVFEUxQDGzHxRtW+//fZ6+4hDvzsmVnIJNhZ3DFn0l+J7AEVFRZkSJ1Dc5qPJ\nbD5BIpsxXsoXyzN966232sfEbMx2JJuXfO+yMTdpt27dgEMLHkpOAQh32Sk7vgGKoigRx/LjxGpZ\n1nZgY3DiJKUoFot1aLhbeugYQ+FwGKeO0SDZME5fk6miKIqSGDXzFUVRDKCTqaIoigF0MlUURTGA\nTqaKoigG8OVnWlhYGPOSODcISktLKSsrC9wZ7nAZY1FRkR1KCE6atjD8DUtKSsrC2AX2cy8lKTk4\nn0E6n4U+r2bJhnH6mkyLi4vt8qhhM2zYsFCucziMsWPHjjzyyCPs2rXLPhZmngHLskJxcfFyL+UH\nxUvOT4CKigoA+7OTKhCdOnWK66fPq1myYZxq5iuKohhAJ1NFURQDpB2bv2XLFsCJze7cuXO6pzyE\nHTt2xK3vBUlNTQ3l5eXs2bPHPiYmYF1TLltp0aJFaCaau1Bd165dQ7mmH7ya90J+fj4AzZo1A6Cq\nqsq4TEp2opqpoiiKAdLWTIPSNty7qxs2bAhNA9i/fz/r16/n+eeft489+eSTgJO4evTo0QAMGjTI\n7jNixAgAhg4dGoqc2YKUjgaYMmVKRmTYvXu33RaLQzTLdMusZDqZeW1tLXv37uWjjz6yjy1atAhw\nMlx98MEHgFPCAxwNW7IqucvU9OrVC4BRo0YBjvbuHmumxx1FVDNVFEUxgPGCen6REgN1SztLSQI4\nWJxPytcGTYsWLRgyZAhHH320fUy0zqeffhqAhx9+GIATTjjB7tOvX7+483z99dd2W37po5Lzs6Ki\nglWrVtnltcHxqTz11FMByMvLA2Dt2rV2HxmvFJVLhmj2PXr0MCN0isRiMbt8BzhaVqtWrdI6b7J7\nuX///kBLCruxLIu8vLw4K0ny54omKuXXP/74Y7vPypUrAWjXrh1A3J6EaO9SZHH79u0AdOjguAaf\ndtppcdZjNjJ//nzAnFtgNL7diqIoWU7GNdO6GqnwzTff2O3i4mJbUwqL3Nxcuy3rSfLqLghYH27t\nTX71RZsRrbtuhvCwyM/PZ+DAgXHVDWRNetWqVYDjlN6lSxe7jxeNVPjDH/4AxGvomcCyrLh76W6n\nglRSkDXXRFRXV4eumRYWFtrHpF33fkkZ71TZtm2b3W7atGnksvPLnOEuqFd3D+P++++327KGL2vL\nAwYMSOv6qpkqiqIYQCdTRVEUA2TczK8Pd1KDvn37JjWroo5setx5552AY0o/99xzGZNJYsoF+XzT\ndeafOHEiAJdcckla54kSf/nLX+y2FIBMVqgtLy8vciawCeoGrURtjD179ox7TcQDDzxgtydMmACk\nb94LqpkqiqIYIFTNdPLkyQDcfffd9rGCgoK4PmVlZQAceeSR4QnmIhaLUVVVFciG1+uvvw7AI488\nYvzcUUHccWbOnJlhSdLnpZdeAuDmm2+2j9W3seR2E8rJyYmc1qYcxD33jB8/3ui5VTNVFEUxQCia\n6S233ALAEUccARyqjbqR0D+3q0eYiKuJSSQPY//+/QE49thjjZ7fJOLYnWxNMBHi5jVt2jTjMmWK\n6dOnA/DCCy802DcqARlBIAEzddfZs4lHH30UgOuuuy6wazTeJ0BRFCVEdDJVFEUxQGBmvjume86c\nOYDjEpSInTt3AvEuUY2FGTNmAPDEE09kWJL62bx5M+AvZv3f//633ZYY/KuuusqoXJngwQcfBByX\ntnHjxtXbVzae3BtOjW3zqbq6GshOM3/x4sVAOLl0VTNVFEUxQGCaqWy2gKOZJUNcTvxmPo8qoo3D\nwQw7UUdi8Lt37+75f6699lq7/cknnxiXKVPMnTsXcLImJaMxbzxJTolszl36t7/9DYBXXnkl8Gs1\n3idBURQlRIyrgbIu6K4FNWnSpHr7S15Fdz7GqCF1riSvo4ReurVvobKyEoC33nrLPvb3v/89aBHT\nxs96mDjmu91Msr3CwJtvvmm3b7zxxgxKkj5i5UlZ6jVr1gBO7lJw9iaSadbZaiW6qzv85Cc/Ce26\nqpkqiqIYwPhPj2Sl9+LoDE7ezCjTtm1bwNuO4MUXXwzAn//850BlMo3kgFyyZAngZNo/55xzDukr\n96wxhMWK1SEWEsDUqVPr7S+796L9RXGHu7y8HDhYOw2c3Xh3HbW6Gqk7HHbjxo2Ak7E/29aF3RUB\nwrSYsutTUhRFiSg6mSqKohjAuJmfaFMmGSeffLJpEYzTokWLBvuIo/dnn30GEFcSJBsQs15Mu/rK\nyQAMHDgwFJnCxG8ugiia90KbNm0AGDJkiOf/cZvysjmVrcEH7jlIck1IPpBkz3W6qGaqKIpiAMtP\n4S/LsrYDG4MTJylFsVisQ8Pd0kPHGAqHwzh1jAbJhnH6mkwVRVGUxKiZryiKYgCdTBVFUQygk6mi\nKIoBdDJVFEUxgE6miqIoBtDJVFEUxQA6mSqKohhAJ1NFURQD6GSqKIpigP8HPo29P1u2SNEAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2a94760f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_output(values=layer_output2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This tutorial showed how to use the so-called *Keras API* for easily building Convolutional Neural Networks in TensorFlow. Keras is by far the most complete and best designed API for TensorFlow.\n",
    "\n",
    "This tutorial also showed how to use Keras to save and load a model, as well as getting the weights and outputs of convolutional layers.\n",
    "\n",
    "It seems likely that Keras will be the standard API for TensorFlow in the future, for the simple reason that is already very good and it is constantly being improved. So it is recommended that you use Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.\n",
    "\n",
    "You may want to backup this Notebook before making any changes.\n",
    "\n",
    "* Train for more epochs. Does it improve the classification accuracy?\n",
    "* Change the activation function to sigmoid for some of the layers.\n",
    "* Can you find a simple way of changing the activation function for all the layers?\n",
    "* Plot the output of the max-pooling layers instead of the conv-layers.\n",
    "* Replace the 2x2 max-pooling layers with stride=2 in the convolutional layers. Is there a difference in classification accuracy? What if you optimize it again and again? The difference is random, so how would you measure if there really is a difference? What are the pros and cons of using max-pooling vs. stride in the conv-layer?\n",
    "* Change the parameters for the layers, e.g. the kernel, depth, size, etc. What is the difference in time usage and classification accuracy?\n",
    "* Add and remove some convolutional and fully-connected layers.\n",
    "* What is the simplest network you can design that still performs well?\n",
    "* Change the Functional Model so it has another convolutional layer that connects in parallel to the existing conv-layers before going into the dense layers.\n",
    "* Change the Functional Model so it outputs the predicted class both as a One-Hot encoded array and as an integer, so we don't have to use `numpy.argmax()` afterwards.\n",
    "* Remake the program yourself without looking too much at this source-code.\n",
    "* Explain to a friend how the program works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2016-2017 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}