GeronCh11_training_deep_neural_networks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GeronCh11_training_deep_neural_networks.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ActionPace/DeepLearningTests/blob/master/GeronCh11_training_deep_neural_networks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Il3MCrnsRwFO",
        "colab_type": "text"
      },
      "source": [
        "**Chapter 11 – Training Deep Neural Networks**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8NeNMVrjRwFU",
        "colab_type": "text"
      },
      "source": [
        "_This notebook contains all the sample code and solutions to the exercises in chapter 11._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBIFuMjORwFY",
        "colab_type": "text"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xauVTq4FRwFa",
        "colab_type": "text"
      },
      "source": [
        "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0-preview."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KFyw22JISZfJ",
        "colab_type": "code",
        "outputId": "658adf54-1d76-4546-d449-f0ec539e97a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 721
        }
      },
      "source": [
        "!pip install tensorflow==2.0.0-alpha0 "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting tensorflow==2.0.0-alpha0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/29/39/f99185d39131b8333afcfe1dcdb0629c2ffc4ecfb0e4c14ca210d620e56c/tensorflow-2.0.0a0-cp36-cp36m-manylinux1_x86_64.whl (79.9MB)\n",
            "\u001b[K    100% |████████████████████████████████| 79.9MB 607kB/s \n",
            "\u001b[?25hCollecting tf-estimator-nightly<1.14.0.dev2019030116,>=1.14.0.dev2019030115 (from tensorflow==2.0.0-alpha0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/13/82/f16063b4eed210dc2ab057930ac1da4fbe1e91b7b051a6c8370b401e6ae7/tf_estimator_nightly-1.14.0.dev2019030115-py2.py3-none-any.whl (411kB)\n",
            "\u001b[K    100% |████████████████████████████████| 419kB 19.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.16.2)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (0.2.2)\n",
            "Collecting tb-nightly<1.14.0a20190302,>=1.14.0a20190301 (from tensorflow==2.0.0-alpha0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a9/51/aa1d756644bf4624c03844115e4ac4058eff77acd786b26315f051a4b195/tb_nightly-1.14.0a20190301-py3-none-any.whl (3.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 3.0MB 11.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.0.9)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (0.7.1)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (3.7.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.1.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (0.7.1)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.11.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.0.7)\n",
            "Collecting google-pasta>=0.1.2 (from tensorflow==2.0.0-alpha0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/bb/f1bbc131d6294baa6085a222d29abadd012696b73dcbf8cf1bf56b9f082a/google_pasta-0.1.5-py3-none-any.whl (51kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 31.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (1.15.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0) (0.33.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow==2.0.0-alpha0) (0.15.2)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow==2.0.0-alpha0) (3.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow==2.0.0-alpha0) (40.9.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow==2.0.0-alpha0) (2.8.0)\n",
            "Installing collected packages: tf-estimator-nightly, tb-nightly, google-pasta, tensorflow\n",
            "  Found existing installation: tensorflow 1.13.1\n",
            "    Uninstalling tensorflow-1.13.1:\n",
            "      Successfully uninstalled tensorflow-1.13.1\n",
            "Successfully installed google-pasta-0.1.5 tb-nightly-1.14.0a20190301 tensorflow-2.0.0a0 tf-estimator-nightly-1.14.0.dev2019030115\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "tensorflow",
                  "tensorflow_estimator"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "11uj2beiRwFe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Python ≥3.5 is required\n",
        "import sys\n",
        "assert sys.version_info >= (3, 5)\n",
        "\n",
        "# Scikit-Learn ≥0.20 is required\n",
        "import sklearn\n",
        "assert sklearn.__version__ >= \"0.20\"\n",
        "\n",
        "# TensorFlow ≥2.0-preview is required\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "assert tf.__version__ >= \"2.0\"\n",
        "\n",
        "# Common imports\n",
        "import numpy as np\n",
        "import os\n",
        "\n",
        "# to make this notebook's output stable across runs\n",
        "np.random.seed(42)\n",
        "\n",
        "# To plot pretty figures\n",
        "%matplotlib inline\n",
        "import matplotlib as mpl\n",
        "import matplotlib.pyplot as plt\n",
        "mpl.rc('axes', labelsize=14)\n",
        "mpl.rc('xtick', labelsize=12)\n",
        "mpl.rc('ytick', labelsize=12)\n",
        "\n",
        "# Where to save the figures\n",
        "PROJECT_ROOT_DIR = \".\"\n",
        "CHAPTER_ID = \"deep\"\n",
        "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
        "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
        "\n",
        "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
        "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
        "    print(\"Saving figure\", fig_id)\n",
        "    if tight_layout:\n",
        "        plt.tight_layout()\n",
        "    plt.savefig(path, format=fig_extension, dpi=resolution)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Li5__nBDRwFo",
        "colab_type": "text"
      },
      "source": [
        "# Vanishing/Exploding Gradients Problem"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S0UYQx0_RwFq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def logit(z):\n",
        "    return 1 / (1 + np.exp(-z))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMYzl90HRwFy",
        "colab_type": "code",
        "outputId": "81a39209-b010-4518-9f06-6ebc6ca922a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        }
      },
      "source": [
        "z = np.linspace(-5, 5, 200)\n",
        "\n",
        "plt.plot([-5, 5], [0, 0], 'k-')\n",
        "plt.plot([-5, 5], [1, 1], 'k--')\n",
        "plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
        "plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
        "plt.plot(z, logit(z), \"b-\", linewidth=2)\n",
        "props = dict(facecolor='black', shrink=0.1)\n",
        "plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha=\"center\")\n",
        "plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha=\"center\")\n",
        "plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha=\"center\")\n",
        "plt.grid(True)\n",
        "plt.title(\"Sigmoid activation function\", fontsize=14)\n",
        "plt.axis([-5, 5, -0.2, 1.2])\n",
        "\n",
        "save_fig(\"sigmoid_saturation_plot\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving figure sigmoid_saturation_plot\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl4Tdf6wPHvisyDOVJEDTXGPLXI\nLTFXUUNoqbHaKlo/LUqvoVeq1dYY91YHnaKCKkUNNZaosYSG0opWY4gIgpDIJMn6/bGPNMMJCSc5\nGd7P8+wn2Xuvs9d7tiPvWXuvvZbSWiOEEEIUNDbWDkAIIYQwRxKUEEKIAkkSlBBCiAJJEpQQQogC\nSRKUEEKIAkkSlBBCiAJJEpR4IEqpIKXUR9aOA3IWi1LqhFJqRj6FlL7eAKXUxnyox0cppZVS5fOh\nrpFKqfNKqVRrnNNMsQxXSsVaMwaRd5Q8ByUyU0q5A37A00BFIBo4AXygtd5uKlMWuKO1jrFaoCY5\niUUpdQJYrbWekUcx+AC7AHetdVS67aUw/p9FW7Cus8BHWuu56bbZA2WByzoP/1MrpcoAV4DxwGog\nRmudLwlCKaWB/lrr1em2OQFuWusr+RGDyF+21g5AFEjfA87Ai8BfQAWgHVDubgGt9XXrhJZVQYol\nM631zXyqJwmIzIeqqmL83diotb6UD/Xdk9Y6Hoi3dhwij2itZZElbQFKAxrodJ9yQRjf4u+uewDr\nMf5YnANewGh1zUhXRgOjgR+AOOA00B7wBLYCt4EQoFmmuvoCvwGJwAVgKqbWfzaxVDDVcTeWEZlj\nMfN+HjO9JtIUx1GgR6Yy9sAs0zETgb+B/wOqmd5b+iXA9JoAjD/mACOBy0CJTMddDqzPSRym95qh\nLtN2H9N6+Vyct7PANOAz4BYQDrx5j3M03Mz7rAbMAE6YKRubbn2G6d9gAHAGiAHWpY/XVG5Yupgv\nA0vSxZq+3rPm6jFtewXji1WS6efLmfZr07/FKtM5/hsYbO3/e7JkXeQelMgs1rQ8o5RyzMXrlmB8\nu+4A9AIGm9YzmwZ8CzQGgk2/fwl8DDQFIjD+qAOglGqO8YdkDdAQeAv4N/DaPWIJAGoCnYDewFCM\nP6T34gpsBjqbYvseWKOUqpvpPQ7FuLxVD6OFGY3xx9/XVKY+xmXRcWbqWAWUMtVx9/25YpyvwBzG\n0RcjkbxjqqeiuTeTi/P2BkZCaAZ8CMxWSrU2d0xgJfCU6ffHTXVfyKasOdWA54A+QBeMf+/30sX8\nCkay/BpohHGJ+YRpd0vTz5dN9d5dz0Ap1Qf4CPAHGgALgY+VUj0zFX0b44tAY9P7+kop9Wgu3ovI\nD9bOkLIUvAXjj+11IAE4AMwFnshUJghTqwWog/GttFW6/VWAFLK2oN5Pt97AtG18um0+pGsJAMuA\nnZnqngGEZxNLbdPrvdPtr5o5lhyeh4PANNPvtUzHfSqbshniTrc9AFMLyrS+Bliabn0wcBNwzEkc\npvWzwMR71Z/D83YWWJGpzJ/p6zITSwtTPdUyHTcnLagEoFS6bVOBv9Kth2Pc58yubg30u089+4Cv\nzPwb7L3H59AWo0UvragCtkgLSmShtf4eqAT0xPg23wY4qJSaks1L6gKpGC2iu8e4gNEayux4ut8v\nm37+ZmZbBdPPehh/dNLbC1RWSpU0c/x6plgOpYvlXDaxpFFKuSilZiulfldK3TD1DGsB3P1W3dR0\n3F33Ok4OBAK9lVLOpvVBwPda64QcxpFTOT1vxzOVieCfc29p53TGe3JpdSmlKgCVgZ8eso7s3rdX\npm1p71trnQxcJe/et3hAkqCEWVrrBK31dq31O1rrNhiX4WaYeos9jDvpq7nHtpx8Nu/VWy23Pdnm\nAv2B6RgdQppgJLmHfb+ZbQKSgV6mP8qd+OfyXn7Fkf7c3DGzL7d/F1IBlWmbnZlylqjrQWX+PFgz\nFpFD8g8icup3jEsh5u5LncL4LDW/u0Ep5YnRCntYfwDembb9C+NSlblu5XdjeTxdLI/mIJZ/Ad9o\nrb/XWh/HuNz0WLr9Iabjts/m9UmmnyXuVYnWOhHj3tAgjPsxkRiXKHMax9267lkPuT9vD+Mq4KGU\nSp+kmuTmANroJn4R6HiPYnd48Pf9e27iEQWDJCiRgVKqnFJqp1JqsFKqkVKqulKqPzAJ+ElrfSvz\na7TWoRi98D5VSrVSSjXBuNEdR+5bMpnNA9oppWYopWorpQYBE4DZ5gqbYtkCfKaUam2KJYD7d0U+\nDfRRSjVTSjXEaNWkJWOt9WngO+ALpZSv6bw8qZQaYipyDuO9dldKuZs6P2QnEOgKjMK4B5Sa0zhM\nzgJPKqUq3+PB3Fydt4cUhPEM1hSl1GNKqReBfg9wnPeA15VSb5hibqKUmpBu/1mgo1LqEdPzWObM\nAYYopV5VStVSSo3F+DKQF+9b5DFJUCKzWIyb8uOA3cBJjK7VyzG+8WdnOMa3/SCM7ubLMB7oTHiY\nYLTWRzEueflieljYtNxr5IjhQBiwE9hgiv3sfaoab4p3D8Z9t4Om39MbajrWfzFaagEYvfLQWl8E\n/oPxR/byfeLbg9Fa8CLj5b2cxvE2RieUMxitlywe8Lw9EK31HxiPD4zEuLfTGeMzk9vjfAK8itFT\n7wTGF4366YpMwGjBXgB+zeYY64CxGL0Tf8f4HI/RWm/IbTzC+mQkCZEnTN/sI4CBpk4XQgiRKzKS\nhLAIpVQHwA2jR14FjJZEFMa3YCGEyDWLXeJTSr2mlApWSiUqpQLuUW6YUuqIUuqWUirc1KVWEmXh\nZwe8i5GgNmDcf2qrtb5t1aiEEIWWxS7xKaX6YnQ37Qo4aa2HZ1NuNMb15V8Ad4z7Fau01h9YJBAh\nhBBFgsVaLlrrNQBKqRYYY6tlV+6TdKsXlVLLyL7rrhBCiGKqIFxaa4vRU8wspdRIjN5BODk5Na9S\npUp+xZUjqamp2NhIZ8j7kfOUMxcuXEBrzaOPyrBw95Pfn6nIhEicSzhT0s7cACYFV0H8v3f69Oko\nrbX7/cpZNUEppUZgDOPyUnZltNaLgcUALVq00MHBwdkVtYqgoCB8fHysHUaBJ+cpZ3x8fIiOjiYk\nJMTaoRR4+fmZmrx9MrP3z2aCzwTebvd2vtRpKQXx/55S6lxOylktQSmlegPvY0zrEHW/8kIIYQ3z\nD8xn9v7ZjGkxhultp1s7nGLFKglKKfUU8DnQXWv92/3KCyGENSw7vowJ2ybQz6sf/+32XzKO5iTy\nmsUSlKmruC3GWFklTHMJJZtGCk5frgPGKAN9tNaHsh5JCCEKhrDoMNpXa09gn0BK2NxvGEBhaZa8\nczYNY7yztzDmuIkHpimlHlVKxaabDGw6xvAwP5q2xyqlNlswDiGEeCgpqSkATGs7ja2Dt+Jg62Dl\niIoniyUorfUMrbXKtMzQWp/XWrtqrc+byrXXWtuatt1dulkqDiGEeBihUaE0+KQBhy4aF3jsSpib\nOUTkh4LQzVwIIQqEiJgIugZ2Je5OHGUcsxswXeQXSVBCCAFEJ0TzVOBTXIu/RtCwIGqVq2XtkIo9\nSVBCiGIvITmBXt/24lTUKTY9v4nmlZrf/0UizxWsx4uFEMJKyjuX55s+39D5sc7WDkWYSAtKCFFs\naa2JuxOHi70Lq/uvluecChhpQQkhii2/3X60/rI10QnRkpwKIElQQohi6dPgT/Hb7UeLSi0o5VDK\n2uEIMyRBCSGKne9//54xm8bQo3YPFvdcLK2nAkoSlBCiWNlzbg/Pr3meVp6tWNlvJbY2ciu+oJIE\nJYQoVqqXqc4zdZ5h4/MbcbZztnY44h7kq4MQoli4HHuZ8s7l8Szpyar+q6wdjsgBaUEJIYq8K7ev\n8K+v/8UrG1+xdigiFyRBCSGKtJjEGLov787FWxcZ0XSEtcMRuSCX+IQQRVZSShK+3/ny66VfWfvc\nWtpUaWPtkEQuSIISQhRZozeOZvvf2/nqma/oWaentcMRuSQJSghRZL3U7CUaeTTihaYvWDsU8QAk\nQQkhipzfLv9GQ4+GtK7SmtZVWls7HPGApJOEEKJIWRKyhEafNmLtH2utHYp4SJKghBBFxqbTm3hx\n/Yt0qtGJ7rW7Wzsc8ZAkQQkhioSD4Qfpv6o/TR5pwppn12Bfwt7aIYmHJAlKCFHoXY+/To/lPahc\nsjI/DvoRNwc3a4ckLEA6SQghCr2yTmVZ0HUB3o96U8GlgrXDERYiLSghRKF1Pf46v4T/AsCQxkOo\nUaaGlSMSlmTRBKWUek0pFayUSlRKBdyn7BtKqUil1C2l1FdKKQdLxiKEKNoSUhLouaInTy17iuiE\naGuHI/KApVtQEcC7wFf3KqSU6gq8BXQEqgI1AD8LxyKEKKKSU5OZ+cdMDlw4wOc9P6e0Y2lrhyTy\ngNJaW/6gSr0LeGqth2ezfzlwVms9xbTeEVimtX7kXsd1c3PTzZs3z7Dt2WefZcyYMcTFxfH0009n\nec3w4cMZPnw4UVFR9OvXL8v+0aNH89xzz3HhwgWGDBmSZf+ECRPo2bMnoaGhvPJK1pGQe/bsyYQJ\nEwgJCeH111/Psn/WrFm0adOG/fv3M2XKlCz7/f39adKkCTt27ODdd9/Nsv+zzz6jTp06bNiwgXnz\n5mXZv3TpUqpUqcLKlSv55JNPsuxfvXo15cuXJyAggICAgCz7f/zxR5ydnfn444/57rvvsuwPCgoC\nYO7cuWzcuDHDPicnJzZv3gzAzJkz+emnnzLsL1euHN9//z0AgwYN4uLFixn2e3p6EhgYCMDrr79O\nSEhIhv21a9dm8eLFAIwcOZLTp09n2N+kSRP8/f0BGDx4MOHh4Rn2t27dmvfffx8AX19frl27lmF/\nx44dmT59OgDdunUjPj4+w/4ePXowceJEAHx8fMgsLz57ISEhJCcn06JFi/t+9qZNm0anTp2K3WdP\nozlT/wwXK1zk46c/JmpL1D0/e//+9785cOBAhv3F6bPXqVMnSpfOmMAf9u/ew372du/efURr3SLL\njkys1UmiPvBDuvVjgIdSqpzWOsO/pFJqJDASwM7OjujojE3506dPExQUREJCQpZ9AKdOnSIoKIib\nN2+a3X/y5EmCgoK4cuWK2f2//fYbbm5unD9/3uz++Ph4goKC+Ouvv8zuP3r0KElJSZw4ccLs/uDg\nYKKjozl27JjZ/b/88guXLl3it99+M7v/wIEDnDlzhpMnT5rdv2/fPkqVKsWpU6fM7v/5559xdHTk\n9OnTZvff/SNx5syZLPvvvneAsLCwLPtTU1PT9iclJWXZb2dnl7Y/PDw8y/6IiIi0/REREVn2h4eH\np+2/fPlylv3nz59P23/16lVu3bqVYX9YWFja/uvXr5OYmJhh/5kzZ9L2mzs3efHZS05ORmtNdHT0\nfT97x44dw9bWtth99m543uBihYsMqDiAerfr8U3YN/f87Jk7f8Xps5eSkpKlzIP83dPahtRUF1JS\nnNm69QJ//nmEv/++xPnz9UlNdUBrR1JTHUhNdeTDD6FMmbNcvOjAyZMjSU11JDXVEa0dSE11AJ7M\nUqc51mpBnQFe1VpvMa3bAUlAda312eyO26JFCx0cHGzxeB9GUFCQ2W84IiM5Tznj4+NDdHR0lm/0\n4h8pqSms+n0VHlc9aN++vbXDKfCCgoJo186H27fh2jW4ft1Y0v9+4wbExBjLrVv//J5+PS7OklGp\nAt2CigVKplu/+3uMFWIRQhQCG09vpLFHY6qUqsKABgPSWhjFVXw8XL4MkZH//Ey/REUZSSgysg2x\nsXDnzsPX6eb2z+LqCs7O4OT0z5J5PbttPXrkrD5rJaiTQGPg7oXnxsDlzJf3hBACYMffO+i7si++\nXr6s8F1h7XDyXFIShIfDuXNw/ryx3P39wgW4dAlu3szp0YwRNZycoGxZKFfO+Hl3KVcOSpeGkiX/\nST7mfndxAZuH6FZ3+vRpzp8/T6dOnXL8GosmKKWUremYJYASSilHIFlrnZyp6DdAgFJqGUbPv2lA\ngCVjEUIUDUcijtBnZR/qlq/LJ92zdsYorKKj4c8/jeX0aePnmTNGEoqMhPvdfbGzAw8PeOSRf5b0\n6+XLG8knNHQ/3bu3wckpf96XOcuXL+eFF16gWbNm1ktQGInmP+nWBwN+SqmvgN8BL631ea31FqXU\nbGAX4AR8n+l1QgjBX9f/4unlT1POqRxbBm8pdN3JtTYuv/32m7GcOAGhoUYyuno1+9eVKAGVK8Oj\nj0LVqsbPu0uVKsa+MmVAqfvHcO1aktWSU0JCAmPGjGHlypUkJSWRmpqaq9dbNEFprWcAM7LZ7Zqp\n7HxgviXrF0IULW9uf5OU1BS2Dt5KJbdK1g7nnpKTjQR05AgcP/5PUoqKMl/eyQlq1TKW2rWNnzVr\nQrVqULEi2Bbygej+/vtvnn76ac6fP5/WjT63nfIK+SkQQhRlAb0COHfzHHXK17F2KBloDWFhcOjQ\nP8vRo0bHhcxKloQGDaBhQ2OpV89ISJUqPdw9nYJszZo1DBs2jLi4uAytJqu2oIQQ4mElJifywd4P\nmOQ9iVKOpWjk2MjaIZGcbCSg3bvh55/h4EHzLaPHHoOWLaFRo38S0qOP5uxSXFFw584d3njjDb76\n6qssDx+DtKCEEIVYSmoKg9cOZvXvq2lRqYXVJh1MTTUS0o4dRlLauxdiYzOWcXeHxx//Z2nZ0uiU\nUFydP3+eHj168Ndff5lNTiAJSghRSGmtGbdlHKt/X83cznPzPTlduQLbtsGWLcbPzJ0YatWCdu2M\nxdvbuFdUXFpG97Np0yYGDhxIXFwcKSkp2ZaTS3xCiEJp1p5ZLDq8iImtJzKhzYR8qfOPP2DNGli3\nDjIPUlO1KnTpAh06QNu2xj0jkdWUKVPw9/fPttWUnrSghBCFzrW4a/j/4s+QRkP4sPOHeVaP1kYv\nuzVrYO1aOHXqn32Ojkbr6KmnjKVOHWkh5UR4eDhaa0qUKHHP1hNIghJCFELlnMtx6KVDeJb0xEZZ\nvmvbn3/C0qUQGGj0vrurbFl45hno2xc6djSG5RG588033zB16lSmTZvGxo0bSUxMzDYRSYISQhQa\ne87tYdfZXUxvO53qZapb9NhRUbBypZGYfvnln+0VKxoJqW9f49JdYX/eqCCoU6cO3333HY0aNeLE\niRPZlpMEJYQoFH67/BvPfPsMHi4evN7qdUo6lLz/i+5DawgKgk8/NS7h3R0g1dUVfH1hyBDw8TFG\nahCWtXPnTsLSN08x5oy7c+cOycnGaHfSSUIIUeCdiz7HU8uewtnOma2Dtz50coqOhiVLjMR0976S\njY1xL2nIEOjdWy7f5bVJkyZx+/btDNsqVKiAj48PK1eu5M6dO9KCEkIUbFFxUXQN7MrtpNvseWEP\nVUtXfeBjnToF8+bBsmX/jOJQsSK8/LKxeHpaKGhxT7t37yY0NDTDNldXV2bPns2zzz7LzJkz8fPz\ny1FPv/QkQQkh8tXB8INcjLnIj8//SEOPhg90jH37YNq0Buzb98+2Tp1g9Gjo2dMY6VvknzfffDNL\n66ls2bL069cPgCpVqvDFF1/k+riSoIQQ+apH7R6cHXeWcs65G3YhNRU2bIDZs2H/foDyODjA8OHw\nxhtGt3CR//bs2cPJkyczbHNxceGDDz7A5iEHGyyiQxUKIQqSVJ3KyA0jWfPHGoBcJSet4YcfoEkT\n417S/v3GVBNDhpzl3DnjvpMkJ+uZNGkScZnmgy9TpgzPPvvsQx9bEpQQIs+9teMtPj/6Ob9f/T3H\nr9HaGHbo8ceNxPTbb8Y9JX9/Y1K/ESPO4uGRh0GL+9q/fz/Hjx/PsM3V1ZX333+fEhboKimX+IQQ\neWre/nnM2T+HV1u+ytQnp+boNbt3w7RpxiCtYMwUO3Wq0fHB0TEPgxW5Yq71VLJkSQYMGGCR40uC\nEkLkmcDjgUzcPpH+Xv1Z+NRC1H3GDvrrL5g40bikB8bo4JMnw6uvSjfxguaXX37h119/zbDN1dWV\nWbNmYWuhp58lQQkh8syxyGO0r9aepX2WUsIm+0s+N2/Cu+/CwoXGw7UuLjBpErz+ujHhnyh4Jk+e\nnKX15OLiwqBBgyxWhyQoIYTFpepUbJQNszvPJiklCQdbB7PlUlLgq6+My3lXrhjbhg+HWbOM55lE\nwRQcHMyhQ4cybHN1deW9996zWOsJpJOEEMLCQqNCafpZU367/BtKqWyT06+/whNPwMiRRnLy9obD\nh+HrryU5FXSTJ08mISEhwzYnJyeGDh1q0XokQQkhLObirYt0CezCpZhLONk5mS0TF2dcvmvZ0pj6\nokoV+PZb2LMHWrTI54BFrv36668cOHAgw7BFLi4uzJw5EzsLPyEtl/iEEBYRnRDNU8ue4nr8dYKG\nBVGzbM0sZbZtg1GjjCkvbGyMe0wzZxqDuYrC4a233jLbenrhhRcsXpckKCHEQ4u/E88zK54hNCqU\nHwf9SPNKzTPsv34dxo0z5mMCaNQIvvjCaEWJwuP48ePs2bMnS+vJz88Pe3t7i9dn0Ut8SqmySqm1\nSqnbSqlzSqnnsynnoJT6VCl1WSl1XSm1QSlV2ZKxCCHyT3JqMo62jizts5RONTpl2LdtGzRoYCQn\nR0f44ANjenVJToXPW2+9RWJiYoZtDg4OvPjii3lSn6VbUIuAJMADaAJsUkod01qfzFRuHNAaaATc\nBBYD/wP6WjgeIUQe0lqTmJKIm4MbWwdvzfCcU1yc8QzTRx8Z697eEBAANbNe+ROFwPXr19myZUuW\n1tOMGTNwcDDfEeZhWawFpZRyAXyB6VrrWK31XmA9MMRM8erAVq31Za11ArASqG+pWIQQ+cNvtx8+\nAT7EJMZkSE5HjkDz5kZysrU1uo3v3i3JqTArW7Ys27dvp2nTpri4uABgb2/Pyy+/nGd1WrIFVRtI\n1lqfTrftGNDOTNkvgYVKqUpANDAI2GzuoEqpkcBIAA8PD4KCgiwY8sOLjY0tcDEVRHKeciY6OpqU\nlJRCca5+iPgB/z/96fZIN4L3B6OUIjUVVqx4lK+/rkZKig1Vq95mypQ/qF07lj17LFu/fKZyxpLn\nqUSJEsyfP5+QkBA+//xzunXrxsGDBy1ybLO01hZZgCeByEzbXgaCzJQtBXwLaCAZ+BUoe786mjdv\nrguaXbt2WTuEQkHOU860a9dON27c2Nph3Neqk6u0mqF0j+U99J2UO1prra9e1fqpp7Q2hnnV+v/+\nT+u4uLyLQT5TOVMQzxMQrHOQVyzZSSIWyDwoSUkgxkzZRYADUA5wAdaQTQtKCFGw7D67m0FrBtG6\nSmtW9luJrY0tBw5A06bG6OPlysGPPxrDFjmZfxRKiByxZII6DdgqpWql29YYyNxBAowOFAFa6+ta\n60SMDhKPK6XKWzAeIUQeqFyyMl0e68KGgRtwsnXG3x/atoXwcGjd2hghols3a0cpigKLJSit9W2M\nltA7SikXpZQ30AtYaqb4YWCoUqqUUsoOGANEaK2jLBWPEMKyouKi0FpTs2xNNgzcQImksvTvb8xm\nm5xs/AwKMkaGEMISLD3U0RjACbgCrABGa61PKqWeVErFpis3EUgA/gSuAk8DfSwcixDCQq7cvkLr\nL1szfut4AP78E1q1gu+/N0YbX70a5s+HPHhWUxRjFn0OSmt9HehtZvsewDXd+jWMnntCiAIuJjGG\np5c9zcVbF3m2/rP89BP07w83bhgP4K5dK93HCyIfHx8aNGhAv379rB3KA5PBYoUQ2UpKSaLvd30J\niQzhu36rOPJDa7p2NZJTz56wf3/RSk5Xr15lzJgxVKtWDQcHBzw8POjYsSPbt2/P0euDgoJQShEV\nlX93KwICAnA1M5jhmjVreP/99/MtjrwgY/EJIbL18oaX2fH3Dj5/egkb5ndn8WJj+7//bUwwaFPE\nvuL6+voSFxfHl19+Sc2aNbly5Qq7d+/m2rVr+R5LUlLSQ41vV7ZsWQtGYx1F7OMlhLCkwQ0HM7PV\nIpZNGsrixeDgAMuWGSNDFLXkFB0dzZ49e/jggw/o2LEjVatWpWXLlkycOJEBAwYAEBgYSMuWLXFz\nc6NChQr079+fixcvAnD27Fnat28PgLu7O0ophg8fDhiX21577bUM9Q0fPpwePXqkrfv4+DB69Ggm\nTpyIu7s73t7eAMyfP59GjRrh4uJC5cqVeemll4iOjgaMFtsLL7zA7du3UUqhlGLGjBlm66xWrRrv\nvvsur7zyCiVLlsTT05M5c+ZkiOn06dO0a9cOR0dH6tSpw48//oirqysBAQGWOcm5VMQ+YkIISwiN\nCgWgtm1nlr8xhqAgYxLBPXvgebNDQBd+rq6uuLq6sn79+izTSdyVlJSEn58fx44dY+PGjURFRTFw\n4EAAqlSpwvfffw/AyZMnuXTpEgsXLsxVDIGBgWit2bNnD9988w0ANjY2+Pv7c/LkSZYvX86hQ4cY\nO3YsAG3atMHf3x9nZ2cuXbrEpUuXmDhxYrbHX7BgAQ0bNuTo0aNMnjyZSZMmceDAAQBSU1Pp06cP\ntra2HDx4kICAAPz8/LIMDpuf5BKfECKDgJAAXlz/Iv9t8jPvvuJNZKTRGWLzZvD0tHZ0ecfW1paA\ngABefvllFi9eTNOmTfH29qZ///488cQTAIwYMSKtfI0aNfjkk0+oV68e4eHheHp6pl1Wq1ChAuXL\n5/6xzurVqzNv3rwM215//fW036tVq8bs2bPp1asXS5Yswd7enlKlSqGU4pFHHrnv8bt06ZLWqho7\ndiz//e9/+emnn2jdujXbt28nNDSUbdu2UbmyMbnEggUL0lpy1iAtKCFEmk2nN/HS+pdocvtN3hrU\nhshI8PExWk5FOTnd5evrS0REBBs2bKBbt27s37+fVq1aMWvWLACOHj1Kr169qFq1Km5ubrQwTQF8\n/vx5i9TfvHnzLNt27txJ586d8fT0xM3Njb59+5KUlERkZGSuj9+oUaMM65UqVeLKlSsAnDp1ikqV\nKqUlJ4CWLVtiY8VruZKghBAAHLhwgP6r+lPl7FSOz3+f2FjFgAHG8EWlS1s7uvzj6OhI586defvt\nt9m/fz8vvvgiM2bM4ObNm3Tt2hVnZ2eWLl3K4cOH2bJlC2Bc+rsXGxubDNNUANy5cydLubujhN91\n7tw5unfvTr169Vi1ahVHjhxrAd4+AAAgAElEQVThq6++ylGd5mSekt0Y4Dc118fJL5KghBBcuX2F\nHit64PzLO5z92o/kZMWbbxodIvJoqp9Cw8vLi+TkZEJCQoiKimLWrFm0bduWunXrprU+7rrb6y4l\nJSXDdnd3dy5dupRh27Fjx+5bd3BwMElJSSxYsIDWrVtTu3ZtIiIistSZub4HUbduXSIiIjIcPzg4\n2KoJTBKUEAJ35wq0OLmdaxsmopQx0Ovs2UWvp969XLt2jQ4dOhAYGMjx48cJCwtj1apVzJ49m44d\nO+Ll5YWDgwMfffQRf//9N5s2bWL69OkZjlG1alWUUmzatImrV68SG2sMoNOhQwc2b97M+vXrCQ0N\nZfz48Vy4cOG+MdWqVYvU1FT8/f0JCwtjxYoV+Pv7ZyhTrVo1EhIS2L59O1FRUcTFxT3Q++/cuTN1\n6tRh2LBhHDt2jIMHDzJ+/HhsbW0zzPWVn4rRx08Ikdn1+OuEXDrOmDGw7Ztm2Noarab/+z9rR5b/\nXF1dadWqFQsXLqRdu3bUr1+fKVOm8Pzzz7Ny5Urc3d1ZsmQJ69atw8vLCz8/P+bPn5/hGJUrV8bP\nz4+pU6fi4eGR1iFhxIgRaYu3tzdubm706XP/0d0aNWrEwoULmT9/Pl5eXnzxxRfMnTs3Q5k2bdow\natQoBg4ciLu7O7Nnz36g929jY8PatWtJTEzk8ccfZ9iwYUydOhWlFI6Ojg90zIeWkzk5Csoi80EV\nXnKeciY/54O6nXRbt/rsX9qh2XcatHZw0HrDhnyp2iLkM5UzD3OeQkJCNKCDg4MtF5DO+XxQ0s1c\niGIoOTWZfisGcXDBeDjVB1dXWL8eTM+ZimJq7dq1uLi4UKtWLc6ePcv48eNp3LgxzZo1s0o8kqCE\nKGa01ryw6jU2+42Gv7tQpozxjJPpUR9RjMXExDB58mQuXLhAmTJl8PHxYcGCBVa7ByUJSohi5rMD\ngQROHgDnfPDwgG3bINPjMaKYGjp0KEOHDrV2GGkkQQlRjMTGwvK3BsE5GypV0uzapahd29pRCWGe\n9OITophY/etWOnVJYs8eGypXht27JTmJgk1aUEIUAz8c28mzvV3R5+3x9IRdu4rWPE6iaJIEJUQR\ntzv0V/o+44w+34rKnqkEBdnw2GPWjkqI+5NLfEIUYSHn/qJT1zuknm9FJc9kft4tyUkUHpKghCii\n4uOhY7dYks89TsXKd9iz25YaNawdlRA5JwlKiCIoMRH69oXrfzShvMcdfg6yk+QkCh1JUEIUMbHx\niTTvEsqWLVC+POzeaScdIkShJAlKiCIk6U4KXp0Pc/LnOriWvMP27eDlZe2ohHgwFk1QSqmySqm1\nSqnbSqlzSqnn71G2mVLqZ6VUrFLqslJqnCVjEaK4SUnRNOnxCxf2/QsH5yR2bLOjSRNrRyXEg7N0\nN/NFQBLgATQBNimljmmtT6YvpJQqD2wB3gBWA/ZAMZhQWoi8oTV49w/mj21tsHVIYttmexlbTxR6\nFmtBKaVcAF9gutY6Vmu9F1gPDDFTfDywVWu9TGudqLWO0Vr/YalYhChOtIax42P5ZW1LbGzvsHG9\nLW3bWjsqIR6eJVtQtYFkrfXpdNuOAe3MlG0F/KaU2g/UBH4BXtVan89cUCk1EhgJ4OHhQVBQkAVD\nfnixsbEFLqaCSM5TzkRHR5OSkpKrcxUY+ChfflmDEiVS+c9/TuJgH01xONXymcqZwnyeLJmgXIFb\nmbbdBNzMlPUEmgGdgd+A2cAKwDtzQa31YmAxQIsWLbSPj4/lIraAoKAgClpMBZGcp5wpXbo00dHR\nOT5Xkz74ky+/rIFSsGyZDc89V3xuOslnKmcK83myZIKKBUpm2lYSiDFTNh5Yq7U+DKCU8gOilFKl\ntNY3LRiTEEXWgq/OMWeK8XDT/IWJPPecg5UjEsKyLNmL7zRgq5SqlW5bY+CkmbLHAZ1uXZspI4TI\nxooNkYwf+QjoEoz/dzSvj5XkJIoeiyUorfVtYA3wjlLKRSnlDfQClpop/jXQRynVRCllB0wH9krr\nSYj727n/BoOfdYUUBwaMuMbc90pbOyQh8oSlH9QdAzgBVzDuKY3WWp9USj2plIq9W0hrvROYAmwy\nla0JZPvMlBDC8Ndf0LenE6kJrnToeZVln5fDSrNxC5HnLPoclNb6OtDbzPY9GJ0o0m/7BPjEkvUL\nUZRdugRdusDN64607ZDI5tXu2MhYMKIIk4+3EIXA9RupNGwTTlgYtGwJG9c5YG9v7aiEyFuSoIQo\n4OLjoXHbs1w760n5R6PYtAnczD28IUQRIwlKiAIsORladv2T8BM1cCl3g8O7y+Hubu2ohMgfkqCE\nKKC0ho79/+LknlrYucSyf1dJqlWTHhGi+JAEJUQBNWUK/LyuJjb2CWzfbE+jhiWsHZIQ+UoSlBAF\n0Lx5mg8+AFtbzerV0O5J6REhih9JUEIUMJfjujBxonEp7+uvFX16Olo5IiGsQxKUEAVI5M3mRP41\nC4Ap711l8GArBySEFUmCEqKA2LLrFqEnZoK25YWxkbw3RbrrieJNEpQQBUDwrwn07KEg2Rm3Siv5\ncuEj1g5JCKuTBCWElZ07B890tyc5zg3XSjuoXn62jK8nBBYei08IkTtXrmg6d4FLl2xo106Tmjqb\nW7dSrB2WEAWCJCghrCQmBpq2jSDidGUaNkrlhx9s6NUrKUu59evXExISQsOGDalfvz6PPfYYJUrI\nM1Gi6JMEJYQVJCbC450uEBFaBTePK2zd4k6pUubLnjlzBj8/P1xdXUlJSSEpKQlPT08aNmzI448/\nToMGDahfvz7Vq1eXxCWKFElQQuSzlBTweSacU4eq4FDqBof3lKVixexvOo0ePZp3332X69evp20L\nCwsjLCyMH3/8EWdn5wyJq1GjRjz++OMMGDCAGjVq5MdbEiJPSCcJIfKR1tB7yEUObvOkhFMMQTsc\nqVPr3t8THR0deffdd3FxccmyLzk5mVu3bnH79m3u3LlDWFgYP/zwA9OmTePAgQN59TaEyBeSoITI\nR9OmwcYVlbGxS+SH9ZpWLZxy9LqXXnoJV1fX+xcE7O3t6dq1K88/L5NUi8JNEpQQ+eS92XHMmgUl\nSsC67x3o3qlkjl9rZ2fHhx9+aLYVlVnJkiVZtmwZSvqqi0JOEpQQ+eCjz28xbbIzAF99BT175v4Y\ngwcPpmzZsvcs4+DgQK1atR4kRCEKHElQQuSx79bEMXaUkZzG/ecsQ4c+2HFKlCjB3Llz79mKSkxM\n5MiRI9SpU4e9e/c+WEVCFBCSoITIQz/tusPAASUg1ZbnRv+F/4xqD3W8fv36UbFixXuWSUpKIioq\nii5dujB9+nRSUuTBX1E4SYISIo+EhEC3HndIveOAj28oKxbVfOhj2tjYsGDBgiytKEfHrFNyxMfH\nM3/+fJ544gnCw8Mfum4h8ptFE5RSqqxSaq1S6rZS6pxS6p7diJRS9kqpP5RS8r9HFCl//gldu8Kd\nOGcadwhlx8o6Fhtfr3v37lSvXj1t3dnZmVdffRVXV1dsbDL+l46LiyMkJAQvLy/WrVtnmQCEyCeW\nbkEtApIAD2AQ8IlSqv49yr8JXLVwDEJYVUQEdOh0hytXoHNn+OXHOlhygAelFP7+/jg7O+Pk5MTA\ngQOZO3cuJ06coGHDhjg7O2con5KSQkxMDIMGDeKll14iPj7ecsEIkYcslqCUUi6ALzBdax2rtd4L\nrAeGZFO+OjAYeN9SMQhhbVeuQAvvaMLP21Gv8S3WrAEHB8vX07FjR+rXr0/FihX53//+B0DVqlUJ\nDg5m7NixODllfb4qLi6O5cuX06BBA37//XfLByWEhSmttWUOpFRTYJ/W2jndtolAO611lk61SqmN\nwJfADSBQa+2ZzXFHAiMBPDw8mn/77bcWiddSYmNjc/wAZXFWHM7TrVu2jBpXi0tnPXCq+BdLF4VT\nrkzujvH666+TkpKSlnTu5e7QR+a6noeEhPD2228THx9PcnJyhn1KKezt7RkzZgw9e/YstM9LFYfP\nlCUUxPPUvn37I1rrFvctqLW2yAI8CURm2vYyEGSmbB9gs+l3HyA8J3U0b95cFzS7du2ydgiFQlE/\nTzdval2v8S0NWjt6nNVnzsc+0HHatWunGzdubJGYrl69qjt06KCdnZ01kGVxdnbW3bt31zdu3LBI\nffmtqH+mLKUgnicgWOfgb74l70HFApkfjS8JxKTfYLoUOBv4PwvWLYTV3L4NXbol8scxN2zLXeDg\nz67UqHL/ER/yWvny5dmxYwfvvfdetpf8duzYQe3atdm/f78VIhTi3iyZoE4Dtkqp9I+xNwZOZipX\nC6gG7FFKRQJrgIpKqUilVDULxiNEnktIgN694Zf9DpRyj+GnHdC4djlrh5VGKcXrr7/OgQMHqFKl\nSpbu6ImJiVy9epVOnToxY8YMeWZKFCgWS1Ba69sYyeYdpZSLUsob6AUszVT0BFAFaGJaXgIum36/\nYKl4hMhrSUnQu28SO3ZAhQrwyx432japYu2wzGrcuDF//PEHvr6+WXr5gfHM1Jw5c2jTpg0XL160\nQoRCZGXpbuZjACfgCrACGK21PqmUelIpFQugtU7WWkfeXYDrQKppXb6+iUIhORkGPp/M1s32KOcb\nbNycQJ061o7q3lxcXAgMDOSLL77I9pmpo0eP4uXlxfr1660UpRD/sGiC0lpf11r31lq7aK0f1Vov\nN23fo7U2241Eax2ks+nBJ0RBlJICw19IZc33tuBwkw8CjtKyWdaRHAqqgQMHcvz4cerXr5+lNXV3\nfqmBAwfyyiuvkJCQYKUohZChjooUHx8fXnvtNWuHUaSlpMALL2iWBdqAXSwTF+1iUv+O1g4r16pX\nr86RI0cYM2ZMth0oli5dSsOGDTl16pQVIhRCEhRXr15lzJgxVKtWDQcHBzw8POjYsSPbt2/P0etD\nQkJQShEVFZXHkf4jICDA7HMNa9as4f335bnnvJKSAsOGwdKlCuxiGT73O+a82NvaYT0wOzs75syZ\nw4YNGyhTpgx2dnYZ9sfHx3PmzBmaN2/OF198cfcRESHyTbFPUL6+vhw6dIgvv/yS06dPs3HjRrp1\n68a1a9fyPZakpKSHen3ZsmVxc3OzUDQiveRkGDoUli0DV1fNm5/8xFdjX7B2WBbRsWNHQkND8fb2\nznLJT2tNXFwc48aNo1evXty8edNKUYpiKScPSxWUxdIP6t64cUMDevv27dmWWbp0qW7RooV2dXXV\n7u7uul+/fjo8PFxrrXVYWFiWhx+HDRumtTYeuHz11VczHGvYsGG6e/fuaevt2rXTo0aN0hMmTNDl\ny5fXLVq00FprPW/ePN2wYUPt7OysK1WqpF988cW0hyl37dqVpc7//Oc/ZuusWrWqnjlzph45cqR2\nc3PTlStX1rNnz84QU2hoqG7btq12cHDQtWvX1ps2bdIuLi7666+/fqBzmp2C+LBgTt25o/XAgVqD\n1q6uqXrv3ryry5IP6uZWamqqnjt3rnZycjL7YK+Dg4P28PDQBw4csEp8mRXmz1R+KojnCSs8qFvo\nuLq64urqyvr167O9GZyUlISfnx/Hjh1j48aNREVFMXDgQACqVKmCn58fACdPnuTSpUssXLgwVzEE\nBgaitWbPnj188803gDGlgr+/PydPnmT58uUcOnSIsWPHAtCmTZu0gUIvXbrEpUuXmDhxYrbHX7Bg\nAQ0bNuTo0aNMnjyZSZMmceDAAQBSU1Pp06cPtra2HDx4kICAAPz8/EhMTMzVeyjKkpNhyBBYsQKw\nj6Hj9Ll4e1s7qryhlGLChAns27ePypUrm31m6vLly3To0IGZM2eSmppqpUhFsZGTLFZQlrwY6mj1\n6tW6TJky2sHBQbdq1UpPmDBBHzx4MNvyf/zxhwb0hQsXtNZaL1iwQAP66tWrGcrltAXVsGHD+8a4\nefNmbW9vr1NSUrTWWn/99dfaxcUlSzlzLagBAwZkKFOzZk09c+ZMrbXWW7Zs0SVKlEhrEWqt9b59\n+zQgLSitdUKC1n36GC0nHG7qxyYO0dHx0XlapzVbUOnFxMToAQMG3HOYpFatWumIiAirxVgYP1PW\nUBDPE9KCyhlfX18iIiLYsGED3bp1Y//+/bRq1YpZs2YBcPToUXr16kXVqlVxc3OjRQtjfMPz589b\npP7mzZtn2bZz5046d+6Mp6cnbm5u9O3bl6SkJCIjI3N9/EaNGmVYr1SpEleuXAHg1KlTVKpUicqV\nK6ftb9myZZbnY4qj27fhmWdg7VpQjjd5ZPQwfn77A0o5lrJ2aPnC1dWVFStWsHjxYlxcXMw+MxUc\nHEzdunXZtGmTlaIURZ38JcKYjbRz5868/fbb7N+/nxdffJEZM2Zw8+ZNunbtirOzM0uXLuXw4cNs\n2bIFuH+HBhsbmyy9nu7cuZOlXOaZUc+dO0f37t2pV68eq1at4siRI3z11Vc5qtOczD2zlFJyaeY+\noqONyQa3bQM7txuUGtWb3dM+pJJbJWuHlu8GDRrE8ePHqVevXrbPTPXv359XX31VLg0Li5MEZYaX\nlxfJycmEhIQQFRXFrFmzaNu2LXXr1k1rfdxla2sLkGUMM3d3dy5dupRh27Fjx+5bd3BwMElJSSxY\nsIDWrVtTu3ZtIiIiMpSxt7e3yJhpdevWJSIiIsPxg4ODi3UCu3oVOnSAffvA0xO270rkp0nzqF2u\ntrVDs5oaNWrw66+/MnLkSLPPTMXHx7N48WK2bdtmhehEUVasE9S1a9fo0KEDgYGBHD9+nLCwMFat\nWsXs2bPp2LEjXl5eODg48NFHH/H333+zadMmpk+fnuEYHh4eKKXYtGkTV69eJTY2FoAOHTqwefNm\n1q9fT2hoKOPHj+fChfsPNVirVi1SU1Px9/cnLCyMFStW4O/vn6FMtWrVSEhIYPv27URFRREXF/dA\n779z587UqVOHYcOGcezYMQ4ePMj48eOxtbUttHMEPYzwcGjbFn79Fcp7RrP75xTaNX+EZhWbWTs0\nq7Ozs2PBggWsW7eO0qVLZ2iZ29nZ0aZNG7p3727FCEVRVKwTlKurK61atWLhwoW0a9eO+vXrM2XK\nFJ5//nlWrlyJu7s7S5YsYd26dXh5eeHn58f8+fMzHMPd3R0/Pz+mTp2Kh4dH2kgOI0aMSFu8vb1x\nc3OjT58+942pUaNGLFy4kPnz5+Pl5cUXX3zB3LlzM5Rp06YNo0aNYuDAgbi7uzN79uwHev82Njas\nXbuWxMREHn/8cYYNG8bUqVNRSmXpwVXUhYbCk0/CqVNQ6tFzRD1Xlwtqr7XDKnC6dOlCaGgorVq1\nSrvk5+LiwqpVq+TepbC8nPSkKCiLTFiY90JCQjSgg4ODLXrcgnye9u7VumxZo7eeR52/NZPK6Pn7\n51slloLSi+9+UlJS9Icffqjt7Oz0tm3brBJDQf5MFSQF8TyRw158ttZOkMK61q5di4uLC7Vq1eLs\n2bOMHz+exo0b06xZ8bistXYtPP+8Ma9T7danOd2+KZN8XuON1m9YO7QCzcbGhkmTJjFu3DgcHBys\nHY4ooqRNXszFxMTw2muv4eXlxaBBg6hXrx5bt24tFvegPvoIfH2N5DTohVjOP9WcYS3780GnD6wd\nWqEhyUnkJWlBFXNDhw5l6NCh1g4jX6Wmwr//DXdv3b37LkyZ4sqkK/uoV75esUjOQhQGkqBEsRIX\nBy+8AN99B7a28OYHoTzSfi9KvUgjj0b3P4AQIt/IJT5RbISHGz31vvsO3Nzgv0v/5uM7TzDvwDwS\nkmViPmupVq1alp6qQoC0oEQxcfAg9OkDkZHw2GPwSeBFhu37F672rmwZvAVH2+LVrT6/DR8+nKio\nKDZu3Jhl3+HDh7OMqCIEFIMWVGRkJN26dWPZsmUyFEsxtXQp+PgYyal9e9i0M4rXgjsQnxzP1sFb\nebTUo9YOsVhzd3fPMoySNTzsfGzC8op8gvr444/ZsWMHo0aNwt3dnTfeeCPLEESiaEpJgcmTjYkG\nExNhzBjYuhUOXN/IhZsX2DhwI/Ur1Ld2mMVe5kt8SikWL15M//79cXFxoUaNGgQGBmZ4zcWLF3nn\nnXcoU6YMZcqUoXv37vz5559p+8+cOUOvXr145JFHcHFxoVmzZllab9WqVWPGjBmMGDGC0qVLM2jQ\noLx9oyLXinSCSk5OZtGiRSQnJxMbG0tMTAyLFi1iyZIl1g5N5LHLl6FLF6OnXokS8PHHsGgR2NnB\n8CbDCX0tFO9Hi+jETkXAO++8Q69evTh27BjPPfccI0aMSJtBIC4ujvbt22Nvb8/u3bs5cOAAFStW\npFOnTmnDfsXGxtKtWze2b9/OsWPH8PX1pW/fvpw6dSpDPfPnz6du3boEBwenzWAgCo4inaA2bdqU\nZQRxGxsbBg8ebKWIRH74+Wdo2hR27gQPD9ixA14ZlcprP77G/gv7AahSqoqVoxT3MmTIEAYPHkzN\nmjWZOXMmtra2/PzzzwB8++23aK2ZPHkyjRo1om7dunz22WfExsamtZIaN27MqFGjaNiwITVr1mTq\n1Kk0a9aM1atXZ6inXbt2TJo0iZo1a1KrVq18f5/i3op0gpo9ezYxMTEZtj355JN4enpaKSKRl1JT\n4YMPjPtMly79M/Crjw9M2j6JRYcX8fO5n60dpsiB9POY2dra4u7unjaTwJEjRwgLC+Ppp59OmxW7\nVKlS3LhxgzNnzgBw+/ZtJk2ahJeXF2XKlMHV1ZXg4OAs87jdnd9NFEwW7cWnlCoLfAl0AaKAf2ut\nl5sp9yYwDKhqKvex1nqOJWP5+++/OXr0aIZtbm5u95weXRRe16/DsGFw9zbDW2/BzJnGs05z989l\n3oF5vNbyNSZ7T7ZuoCJH7jWPWWpqKk2aNOGNN97giSeeyFCubNmyAEycOJEtW7Ywd+5catWqhbOz\nM0OHDs3SEUJ6DxZslu5mvghIAjyAJsAmpdQxrfXJTOUUMBQ4DjwGbFNKXdBaf2upQP73v/9lmTPJ\n2dmZzp07W6oKUUDs3Gkkp/BwKFMGvvkGevQw9n1z7Bve3P4mz9Z/Fv+n/GWUiCKgWbNmrFixglKl\nSlGzZk2zZfbu3cvQoUPx9fUFICEhgTNnzlC7dvGd16swsliCUkq5AL5AA611LLBXKbUeGAK8lb6s\n1jr9/BChSqkfAG/AIgkqMTGRL7/8MsP9JycnJ8aNGydTAhQhCQkwZQosWGCsP/EEfPstVKtmrGut\n2fzXZjpU78A3vb+hhE0Jq8Uq4NatW4SEhGTYVrp06VwfZ9CgQcydO5epU6fi5ubGo48+yoULF/jh\nhx8YNWoUtWrVonbt2qxdu5ZevXphZ2eHn58fCQnyMHZhY8kWVG0gWWt9Ot22Y0C7e71IGV9pnwQ+\ny2b/SGAkGJMDBgUF3TeQHTt2kJycnGFbcnIy9erVy9HrcyM2NtbixyyKLH2e/vrLhffe8+LsWRds\nbDRDh55l8ODznD2rOXvWSE5KKV4q+xJJpZM4sPeAxerOS9HR0aSkpBS5z1RkZCR79uyhadOmGba3\nbds2rXWT/j2fPHmS8uXLp61nLvP+++/z8ccf07t3b27fvk25cuVo0qQJv//+OxcvXqR///7MmTMH\nb29vXF1d6devH15eXkRGRqYdw1y9RVGh/huVkzk5crJgJJnITNteBoLu8zo/jETmcL86cjofVJMm\nTTSQtiildK9evXI6VUmuFMS5VgoiS52n5GStP/xQazs7Y/6mWrW0/uWXjGX+uPqHbvd1O33h5gWL\n1JmfCst8UAWB/N/LmYJ4nrDCfFCxQMlM20oCMWbKAqCUeg3jXtSTWmuLDPNw4sQJQkNDM2xzdnaW\nzhFFwPHj8PLLcOiQsT56NMyZA+nvc1+8dZGugV1JSE4gMVlGDhGiMLPkDZnTgK1SKv3DBI2BzB0k\nAFBKjcC4N9VRax1uqSD8/f2z9NQpX7483t7yUGZhFR9vTI/RvLmRnCpXNnrrffxxxuR0I/4GTy17\nihvxN9gyaAuPlX3MekELIR6axRKU1vo2sAZ4RynlopTyBnoBSzOXVUoNAmYBnbXWf1sqhtu3b7N8\n+fIMvfecnZ2ZMGGC9N4qpH76CRo2NJ5vSkmBV1+F33+H7t0zlou/E88z3z7D6WunWTdgHU0rNjV/\nQCFEoWHpLm1jACfgCrACGK21PqmUelIpFZuu3LtAOeCwUirWtHz6sJUvX748Sy+91NTUYjchX1Fw\n6ZLRdbxTJzhzBurXh337jFlwS2a+kAzEJMUQmxTL0j5L6VC9Q/4HLISwOIs+B6W1vg70NrN9D+Ca\nbr26Jes1HZM5c+Zw+/bttG02Njb4+vpSqlQpS1cn8khCgtFtfNYsiI0FBweYPh3efBPs7bOW11qT\nqlOp4FKBwy8fxtZGZpARoqgoMg8FBQcHExERkWGbo6Mj48ePt1JEIje0hu+/By8v49mm2Fjo1QtO\nnICpU80nJ4D/BP0H3+98SUpJkuQkRBFTZBLUvHnziI+Pz7CtatWqNGvWzEoRiZwKDjbGz+vXD8LC\noEEDY4DXdesgm4ECAFh0aBEzf55Jeefy2NnYZV9QCFEoFYkEdePGDX744Ye0sboAXF1dpWt5AXf8\nuDHLbcuWsHs3lCtn9Mz79Vfo2PHer111chVjN4/lmTrP8GmPT6UTjBBFUJG4JhIQEJClc4TWmgED\nBlgpInEvp07BjBmwcqWx7uQEY8caA7yWKXP/1+8M28ngtYPxftSbb32/lUt7QhRRhf5/ttaa+fPn\np01UBsbw/EOGDCkQ00iLf/z+O3z4IQQGGlNj2NsbD9u+9RY88kjOj+Ni50Jrz9asfW4tTnZOeRew\nEMKqCn2C2r17N9HR0Rm22dnZMW7cOCtFJNLTGvbuhSlTGnDANByera0xIsTUqVAlF/MGxiTG4Obg\nxhOeT7Br2C65rCdEEVfo70HNmzeP2NjYDNu8vLyoW7eulSISYDxUu2YNtGljTBx44EB5HB1hzBgI\nDYVPP81dcroce5mmn8yIIesAAA6sSURBVDVl9j5jIHxJTkIUfYUqQcXHx7Nt27a0zhCXL19mx44d\nGcq4uroyadIka4QngCtXjMt4tWqBry8cPAhly8LQoWc5fx4WLYIaNXJ3zFuJt+i2rBsRMRG0rdo2\nbwIXQhQ4heoS37Vr1+jWrRsVKlRg3LhxREVFZSljY2ND795ZnhUWeejuZbxPPoHVq+HuNFzVqsH4\n8TBiBBw+fBZ392q5PnZiciJ9V/bl+OXjrB+4nlaerSwauxCi4CpUCcrW1hYbGxsiIyN55513SEpK\nyjDunr29PSNHjsQ+u6c6hUVdvAjLl8OSJXDSNCSwUsZstqNHQ9euUOIh5gjUWjP8h+H8FPYTS3ov\n4elaT1smcCFEoVDoEpSDgwPJyclZHsoF477EkCFDrBBZ8REba9xbWrrUGMjVmNILPDzgpZeMzg9V\nq1qmLqUUTz32FC0qtmBoYxlPUYjiptAlqBL3+EpeokQJnnjiCfr168f48eOzzN4pHkxsLGzZYgxF\ntH493O3Rb29vtJYGDzZGF7dkwzX8VjieJT0Z1mSY5Q4qhChUClUnCVtb23v23oqLiyMhIYHly5fT\nrFkzPv/883yMrmi5ft24dNerF7i7Q//+8O23RnLy9jZ64f1/e/cfXFV95nH8/eSGREJ+CGIR5IdI\nYV2pJUgKSyklitXQahU7Si21ZbsV1wIdpkut1nXGarvd6XRKO9aRUtktgsViS3fBiFVrg9KOsrCb\nqKwIZRHFEeVXIAmBEPLsH+deSWKSe0MunHNzP6+Z7+Sek++9eXLm5Dz53vO9z/fdd4OkNXNmepPT\nsv9exugHR/PynpfT96IiknEybgTV+p5TZ8455xwmTZrELbfcchai6h1aWqCmJhgpPf10sLRF60M9\neTLceGPQujsLrzvWvrGWuU/O5aqLr9KaTiJZLuMSVPvVctsrKCjgpptu4pFHHiE3N6N+vbNuz56g\nBt4zz8Af/gDvvXfqe7FYsBbTzJlwww0wZMiZj+fPb/2ZWb+dxYTBE/jdzb8jL6bJLiLZLKOu4LFY\njBOJOcwdKCgo4O677+aee+7RBzk7sHt3kJASbefOtt8fOhQqKoI2fTqce+5ZjK12N9etuo5hxcOo\n/FIlhXmFyZ8kIr1aRiUoM6OgoKDNooQJBQUFLF26lNmzZ4cQWfQcPgxbtsCmTafaO++07VNUBJ/6\nFFx5JcyYEazFFFZeH1YyjAUTFzCndA7n9zs/nCBEJFIyKkEBlJSUfChBFRYWsm7dOsrLy8MJKmQH\nD8KrrwZt8+YgGW3bdmoKeEJJCUydCtOmQXk5lJYGdfHCdODoAeqb6hlx7gi+d8X3wg1GRCIl4xJU\n//79P1g5Nzc3l/79+1NVVcWll14acmRn3tGjQR27RDJKtHYLCQPBrLrS0mCtpYkTgzZmDOREaN5m\nQ1MD1666lvcb3uf1ea/rnpOItJFxCWrgwIEA5OfnM2LECKqqqhg8eHDIUaXP8ePBqrLbt8OOHUFL\nPN6zp+PnFBTA2LFw2WVw+eVBMvr4xyE//+zG3h0nTp5g1m9nsemdTTxx0xNKTiLyIRmXoC644AJy\ncnKYNGkSlZWVFBZmzs109+DtuLfeatt27z71eO/eD781l5CbC6NGBYmodRs5smclhc42d+e2dbdR\nuaOSJZ9bwo1/e2PYIYlIBGVcgpoyZQqFhYUsWbIkMtPIGxth//4gubRu773Xdvvdd09VYehMTk5Q\nKmjMmKCNHh20MWOC/RH5lXvkwU0PsrxmOfdNu4/by24POxwRiaiMu9wtWLAg7a/Z0gINDXDkCNTV\nnWq1tcGI58CB4Gv7xwcPwr59U0ny0aw2ioqCRDN8eNBaPx4+PPi8UW9IQl2ZUzqHHMth3ifmhR2K\niERYWi+FZjYAWAZcDewH7nb3X3fQz4B/Bb4e3/UIcJd7Z29uBZqb4c03gxFLoh09mtp2Q0OQdFon\nocTjdusddlOMvDwYODBYtjzRBg1qu53YV1LSk5+V2Z7f9TyTLpxEcX4x8yfODzscEYm4dP+v/hDQ\nBAwCSoFKM6tx963t+s0FbgDGAQ48C+wClnT14jU1wf2WM6Ffv2B0U1wcfC0qCpLJeecFC+4lviZa\nYvu1116gouLToX1+KFNsPriZ7774XeZ9Yh6LKxaHHY6IZABLMmhJ/YXM+gGHgI+5+/b4vhXAO+5+\nV7u+fwF+5e5L49v/ANzm7l2uRpeTM97z8taTk9NELHacnJxT7dR2U3z72AePE9u5uQ3EYo3EYkeJ\nxRrIzU08bsSs5bR+79raWs49myUXMlBdUR3V46rpe6wvpf9TSu7JXv4eZg9UV1fT3NxMWVlZ2KFE\nnv72UhPF47Rhw4Yt7p70JE/nlWIM0JxITnE1wLQO+o6Nf691v7EdvaiZzSUYcdGnTx8uuaSix4G2\ntASti6pJKTt58iS1tbU9f6Fe6ni/4/z1sr8Sa4ox4sUR1B/v0fupvV5zczPurnMqBfrbS00mH6d0\nJqhC4Ei7fYeBok76Hm7Xr9DMrP19qPgoaylAWVmZb968OX0Rp0FVVVXWVrBIxt2ZvGwy/Q/15ydj\nf8KXf/TlsEOKvPLycmpra6murg47lMjT315qonicUq2Vms4EVQ8Ut9tXDNSl0LcYqE82SUIyi5mx\nYuYKjhw/Qt32jk4DEZHOpbPwzXYg18xGt9o3Dmg/QYL4vnEp9JMMdKz5GL/c8kvcndHnjWbCkAlh\nhyQiGShtCcrdG4A1wP1m1s/MpgDXAys66P4o8C0zu9DMhgD/BPwqXbFIeE62nGT2mtnMfXIuL+15\nKexwRCSDpbt06DeAvsD7wCrgDnffamZTzaz13fFfAOuAV4HXgMr4Pslg7s78p+az5vU1LL5mMZOH\nTQ47JBHJYGmd7+vuBwk+39R+/4sEEyMS2w7cGW/SSzzwwgMs2bKE70z5Dgv/bmHY4YhIhovQ4guS\nyXYe3Mn3X/g+Xx33VX44/YdhhyMivYA+MSlpMWrAKDZ+bSPjLxif8hRSEZGuaAQlPbLhzQ2s3roa\ngIkXTqRPrE/IEYlIb6ERlJy2mr01fP7xzzOseBgzL5mp5CQiaaURlJyWXYd2UfFYBUV5RTw1+ykl\nJxFJO42gpNv2NezjmpXXcKz5GBv/fiPDS4aHHZKI9EJKUNJtv9n6G94+8jbP3focYz/SYY1fEZEe\nU4KSbps/cT4zPjqDUQNGhR2KiPRiugclKWnxFhY+vZDqvUGVbSUnETnTlKAkKXdn0TOL+NnLP+PZ\nnc+GHY6IZAklKEnqx3/5MYtfWsyCiQtY9MlFYYcjIllCCUq6tLx6OXc+dyc3j72Zn1b8VFUiROSs\nUYKSTrk7T/zvE0wfOZ1Hb3iUHNPpIiJnj2bxSafMjDWz1tB0son83PywwxGRLKN/ieVDtu3fxozH\nZrCvYR95sTwK8wqTP0lEJM00gpI29hzZw9UrrqbpZBN1TXWc3+/8sEMSkSylBCUfONR4iIqVFdQe\nq2XDnA1c3P/isEMSkSymBCUANJ5o5LpV17Hj4A7Wz17P+MHjww5JRLKc7kEJAAcaD7D/6H5WzlzJ\nlSOvDDscERGNoLKdu+M4Q4uH8sodr5AXyws7JBERQCOorHfvn+5lzn/MobmlWclJRCJFCSqL/XzT\nz/nBiz8gP5ZPzGJhhyMi0oYSVJZavXU131z/Ta7/m+t5+NqHVcJIRCInLQnKzAaY2e/NrMHMdpvZ\nl7ro+20ze83M6sxsl5l9Ox0xSOqe3/U8t/7+VqYMn8KqL6wiN0e3IkUketJ1ZXoIaAIGAaVApZnV\nuPvWDvoa8BXgFWAU8IyZve3uj6cpFknCMMqGlLH2i2vp26dv2OGIiHSoxwnKzPoBXwA+5u71wEYz\nWwvcCtzVvr+7/6jV5htm9p/AFEAJ6gxrPNFI3z59uWLkFWy8aKPe1hORSEvHCGoM0Ozu21vtqwGm\nJXuiBVfIqcAvuugzF5gb36w3szd6EOuZMBDYH3YQGUDHKXUDzUzHKjmdU6mJ4nEakUqndCSoQuBI\nu32HgaIUnnsfwX2wf++sg7svBZaebnBnmpltdveysOOIOh2n1OlYpUbHKTWZfJySTpIwsyoz807a\nRqAeKG73tGKgLsnrzie4F/U5dz9+ur+AiIj0TklHUO5e3tX34/egcs1stLvviO8eB3Q0QSLxnK8R\n3J/6tLvvST1cERHJFj2eZu7uDcAa4H4z62dmU4DrgRUd9Tez2cC/AJ9x9//r6c+PgMi+/RgxOk6p\n07FKjY5TajL2OJm79/xFzAYA/wZ8BjgA3OXuv45/byqw3t0L49u7gKFA67f1Vrr7P/Y4EBER6TXS\nkqBERETSTaWOREQkkpSgREQkkpSg0szMRpvZMTNbGXYsUWNm+Wa2LF6vsc7Mqs1sRthxRUV3alpm\nK51D3ZfJ1yQlqPR7CPivsIOIqFzgbYIqIyXAPwOrzeyiEGOKktY1LWcDD5vZ2HBDihydQ92Xsdck\nJag0MrMvArXAH8OOJYrcvcHd73P3N929xd2fBHYBE8KOLWytalre6+717r4RSNS0lDidQ92T6dck\nJag0MbNi4H7gW2HHkinMbBBBLcdOP9SdRTqraakRVBd0DnWuN1yTlKDS5wFgmSpjpMbM+gCPAcvd\nfVvY8URAT2paZiWdQ0ll/DVJCSoFyeoRmlkpcBWwOOxYw5RC3cZEvxyCSiNNwPzQAo6W06ppma10\nDnWtt1yTtJRqClKoR7gQuAh4K77GUiEQM7NL3f3yMx5gRCQ7TvDBEivLCCYCfNbdT5zpuDLEdrpZ\n0zJb6RxKSTm94JqkShJpYGYFtP3vdxHByXGHu+8LJaiIMrMlBKsuXxVf4FLizOxxwIGvExyjp4BP\ndrIyddbSOZRcb7kmaQSVBu5+FDia2DazeuBYJp0IZ4OZjQBuJ6jDuLfVir63u/tjoQUWHd8gqGn5\nPkFNyzuUnNrSOZSa3nJN0ghKREQiSZMkREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQk\nkpSgREQkkpSgREQkkv4fpnIt6Q3iZsAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4BQZddYRwF9",
        "colab_type": "text"
      },
      "source": [
        "## Xavier and He Initialization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MvKMdcVXRwGA",
        "colab_type": "code",
        "outputId": "d25a5707-f20d-4303-9b9c-38e6894a0716",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 499
        }
      },
      "source": [
        "[name for name in dir(keras.initializers) if not name.startswith(\"_\")]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Constant',\n",
              " 'GlorotNormal',\n",
              " 'GlorotUniform',\n",
              " 'Identity',\n",
              " 'Initializer',\n",
              " 'Ones',\n",
              " 'Orthogonal',\n",
              " 'RandomNormal',\n",
              " 'RandomUniform',\n",
              " 'TruncatedNormal',\n",
              " 'VarianceScaling',\n",
              " 'Zeros',\n",
              " 'constant',\n",
              " 'deserialize',\n",
              " 'get',\n",
              " 'glorot_normal',\n",
              " 'glorot_uniform',\n",
              " 'he_normal',\n",
              " 'he_uniform',\n",
              " 'identity',\n",
              " 'lecun_normal',\n",
              " 'lecun_uniform',\n",
              " 'ones',\n",
              " 'orthogonal',\n",
              " 'serialize',\n",
              " 'zeros']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OY3FKgaBRwGH",
        "colab_type": "code",
        "outputId": "23dc892b-4423-4497-8a49-3c6bc0beeab9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.core.Dense at 0x7f6ec2b81550>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tt3E5LbbRwGU",
        "colab_type": "code",
        "outputId": "89815a0b-ce60-48e6-dccf-d85dd26287c2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',\n",
        "                                          distribution='uniform')\n",
        "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=init)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.core.Dense at 0x7f6ec33de908>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dr4Cj_peRwGf",
        "colab_type": "text"
      },
      "source": [
        "## Nonsaturating Activation Functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y1ujjy4tRwGj",
        "colab_type": "text"
      },
      "source": [
        "### Leaky ReLU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XY99CGxHRwGn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def leaky_relu(z, alpha=0.01):\n",
        "    return np.maximum(alpha*z, z)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ngDoDCzTRwGw",
        "colab_type": "code",
        "outputId": "a9aefa51-7a66-4bfc-84b5-cde0f70a93fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        }
      },
      "source": [
        "plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\n",
        "plt.plot([-5, 5], [0, 0], 'k-')\n",
        "plt.plot([0, 0], [-0.5, 4.2], 'k-')\n",
        "plt.grid(True)\n",
        "props = dict(facecolor='black', shrink=0.1)\n",
        "plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\n",
        "plt.title(\"Leaky ReLU activation function\", fontsize=14)\n",
        "plt.axis([-5, 5, -0.5, 4.2])\n",
        "\n",
        "save_fig(\"leaky_relu_plot\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving figure leaky_relu_plot\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8FPX9x/HXh3AlHCJyVMGCeKCg\nVSCetBiPUm9RUEG04sGhVeuB1oMKiGfFWjwBiyJyi1yitD9Fo+JVQVG8gFJQQUUEEgghAZLv74/v\noiHk2E2ymdnN+/l47IM9JjvvHTb7zsx8d8acc4iIiIRNraADiIiIlEQFJSIioaSCEhGRUFJBiYhI\nKKmgREQklFRQIiISSiooKZWZZZrZ40HnSAZmlmFmzsyaVcO8VpvZ4GqYz6Fm9p6Z5ZnZ6njPL4o8\nzsx6BZ1Dqo4KKkGZ2Xgzmxd0jlhFSs9FLtvNbKWZ3W9m9WJ8nn5mllPOfPYo1/J+riqUUhDvAvsC\nG6pwPsPM7LMSHjoaeLKq5lOGe4Bc4NDIPKtFGe/9fYGXqiuHxF/toANIjfQscAdQF//B9mzk/tsD\nSxRnzrntwA/VNK/11TEf4CBgjnNudTXNr0zOuWpZvlJ9tAaVpMxsLzMba2Y/mtkWM3vTzNKLPL6P\nmU0xszVmts3MPjezy8t5zlPMLMvMBplZNzPbYWa/KjbNvWb2aTnxcp1zPzjnvnHOvQi8CnQv9jyt\nzGyqmW2KXF42s4NjXAwVYmYPmNmyyHJZbWZ/M7P6xaY5w8w+iEyzwcxeMrP6ZpYJtAEe2rWmGJn+\n5018ZtY48nNnF3vO7pFl2qK8HGbWDxgKdCyyRtov8thua3Bm9mszmxV5H2wxs5lm1rrI48PM7DMz\n6x1Zo91iZrPL2hwZeV1HAndF5j3MzNpGrqcXn3bXprci0/Q0s1fNLNfMvjCz3xf7mUPNbK6ZZZtZ\nTmRT4hFmNgy4DDizyOvOKD6fyO0jzOy1yPLbGFnz2qvI4+PNbJ6Z/dnM1kbeZ8+aWVppr1uqlwoq\nCZmZAS8DrYCzgE7AW8DrZrZvZLL6wEeRxzsCo4AxZnZKKc/ZC5gFDHDOjXbOvQWsBP5YZJpakdvj\nYsh6JNAV2FHkvjTgDSAPOBE4HvgeeK2aPjy2AlcAhwHXAL2BO4vkOw2Yiy/WLsBJwJv436fzgTXA\n3fhNTvtSjHNuM35TVN9iD/UFXnXO/RhFjmnAw8CyIvOZVnxekf+TOUDLSM6TgP2A2ZH3yS5tgYuA\n8/B/LHQC7i1l+RCZ37JIhn2BkWVMW5J7gUfxJfchMNXMGkYy7wcsBBzwe6Az8ASQEpnPdOC1Iq/7\n3RJedwPg30AOcEzkdZ0APFNs0t8BhwOn8svr/3OMr0XixTmnSwJegPHAvFIeOxn/i5la7P4lwK1l\nPOdU4J9FbmcCjwMDgGyge7HpBwNfFrl9OpAP7FPGPDKB7ZF8+fgPoQKgZ5FprgBWAFbkvhT8/psL\nI7f7ATnlzOfxEu4v8+dKea5BwH+L3H4HmFrG9KuBwcXuy4i81maR2+fg9980itxOBTYDF8eQYxjw\nWVnzx3/AFwBtizzeDigETi3yPHnAXkWmubPovErJ8xkwrMjttpHXmF5sOgf0KjbNwCKPt4rc99vI\n7XuBr4G6sbz3i82nf+Q926iE/4ODijzPt0BKkWmeBl6ryO+kLlV/0RpUcuoCpAHrI5tHcswPDDgc\nOBDAzFLM7E4z+zSyiSoH/9f/r4s9Vw/8X6+nOef+r9hjzwHtzOyEyO0rgNnOufIGAkwDjsKvGU0H\nnnZ+U1/R/AcAW4pkzwb23pU/nsysl5ktNLMfIvN+hN2XSydgQSVnMx9fUOdFbp8DGDA7hhzROAz4\nzhXZT+Sc+x/wHdChyHRfO+eyi9z+DmgR47xiUXQz8HeRf3fNrxOw0Pn9dhV1GPCpc25LkfvexRdz\n0df9hXOuoFiWeL5uiYEGSSSnWsA6/OaL4jZH/h0M3IzfnLEUv0ZzH3v+cn4CHAFcaWbvu8ifmeB3\nxpvZXOAKM1uG/5A9m/JlO+f+C2BmlwCfm1k/59z4IvmX4DdpFbcxiucH/zr3KuH+JviyK5GZHYdf\nkxwO3Ahk4V9XrJuwyuSc22Fm0/Gb9SZE/p3lnMutxhxFT2Wwo4THYv0DtjDy78+bDs2sTinT/jw/\n55yLbG2srj+Yq/p1S5yooJLTR/h9DoWRv5ZL8lvgJefc8/DzfqtD8B+ERa0CrsNvMhtrZgOKlhR+\nk8gM4H/4UWqvxRI08kF9H3C/mU2PfEB/BPQBfnLOFc8TrWXAGWZmxfJ2jjxWmq7AWufciF13mFmb\nYtN8DJyCf+0l2Y7fJFmeicBbZtYBOA2/PzCWHNHM50tgPzNru2stysza4fdDfRFFxljsGj1YdL/b\nURV4no+BS8ysbilrUdG+7ivMrFGRtagT8OXzZQUySQD0l0Jia2xmRxW7tMWXxDvAHDM73cwOMLPj\nzWy4me1aq1oOnGJmvzWzQ/H7mg4oaSaRkjsJ/yE6ptjO9Vfx+4aGAuOdc4UlPEV5JuP/cr02cnsS\nfg1wjpmdGMnfzcwett1H8tUq4fUfHnnsKfy+lsfM7Egza29mN+KL76EysiwHWplZXzNrZ2ZXR36m\nqHuBC8zsHjPrYGYdzezGIgM4VgO/Mz8SsdSRcM65d/H7WiYDP7H7ZsNocqwG2phZZ/OjA0v6Ltlr\n+M1pk8ws3fwIu0n4PwJeL2M5xMw5tw14H/hLZJmcQMXW+J4EGgLTzexoMzvIzPqY2a6yWw0cHvk/\nbVbKWtok/CbUCeZH83UDxgAzd629S/ipoBLb7/B/bRa9jIysMZyB/wB6Gr/GMB1ozy/b++8B/oPf\nF/IWfsTYpNJm5Jxbid/JfDpFSioyr2eBOvzyfaaYRP5Kfhy4NfIXby7QDb9W9gLwFX5/197ApiI/\nmlrC68+MPOf/Is9xMPB/kdfaG7jAOTe/jCwv4QvsH/gP9t8DdxWb5hX8vqPTI/N8E1/gu8r5LmB/\n/CjH8r6TNAk/km1q0X0h0eQAXgRewRfbevYssF3/P+dGHn8jcvkB6FFszbKqXBH590N8IQyJ9Qmc\nc2vx/3d18Xk/xq/F74xM8jR+LWgR/nV1LeE5coE/AI3x//dzgPeK5JMEYPF5j0pNYmZP4UdG/b7c\niUVEoqR9UFJh5r/02AH/3acLA44jIklGBSWVMQf/JchxzrmXgw4jIslFm/hERCSUNEhCRERCKW6b\n+Jo1a+batm0br6evlK1bt9KgQYOgYyQkLbvYLVu2jIKCAjp06FD+xLIbvd8qrrRlt2oVbNwI9erB\nYYdBSjTf2Ktiixcv/sk517y86eJWUG3btmXRokXxevpKyczMJCMjI+gYCUnLLnYZGRlkZWWF9vch\nzPR+q7iSlt3DD8PgwdCgAXzwAXTsGEw2M/s6mum0iU9EpAZ49VW49VZ/fcKE4MopFiooEZEk97//\nwUUXQWEh/PWvcP75QSeKjgpKRCSJbd0KPXrApk1w1lkwbFjQiaKnghIRSVLOweWXw9Kl0L49TJwI\ntRLoUz+BooqISCwefBBeeAEaNYLZs2Gvkk5AE2IxFZSZHWxmeWY2MV6BRESk8j74oCl33OGvT5oE\nhx4abJ6KiHUN6gn8UYpFRCSkVqyAe+45DOdg+HA4O5rTiIZQ1AVlZr3xJ7Or7KmuRUQkTrZs8YMi\ncnLq0KMHDIn5hCfhEdUXdc2sMXA3cDJwVRnTDQAGALRs2ZLMzMwqiFj1cnJyQpst7LTsYpeVlUVB\nQYGWWwXo/RabwkIYOrQjX3zRnP3330L//kt4662C8n8wpKI9ksQI/BGr1+x+MtXdOefGAmMB0tPT\nXVi/Aa5vp1ecll3smjRpQlZWlpZbBej9FpsRI2DhQj8Y4r77vuCMM35X/g+FWLkFFTnN8qlAp/jH\nERGRinjpJRg6FMxgyhRITd0WdKRKi2YNKgNoC3wTWXtqCKSYWQfnXOf4RRMRkWh89RVccon/3tN9\n98Hpp0MybBmNpqDGAlOL3B6ML6yr4xFIRESil53tB0Vs3gy9esFttwWdqOqUW1DOuVwgd9dtM8sB\n8pxz6+MZTEREylZY6Necli2DI46AZ5/1m/iSRcyn23DODYtDDhERidHw4TBvHuy9tz9SRMOGQSeq\nWjrUkYhIApo1C+6+2x9bb+pUaNcu6ERVTwUlIpJgPv8c/vhHf/3BB6F792DzxIsKSkQkgWzatOtI\nEdCnD9x8c9CJ4kcFJSKSIAoKoG9f+O9/4aij4J//TK5BEcWpoEREEsRf/wrz58M++/h9UGlpQSeK\nLxWUiEgCmD4d7r8fUlL89bZtg04UfyooEZGQ+/RTf2ZcgIcfhpNPDjZPdVFBiYiE2MaNflBEbq4f\nuXf99UEnqj4qKBGRkNq5E3r3hlWroEsXGD06uQdFFKeCEhEJqdtvh1dfhebN/aCI1NSgE1UvFZSI\nSAhNngwjR0Lt2jBjBuy/f9CJqp8KSkQkZD7+GK6KnLv8H/+Abt2CzRMUFZSISIisX+8HRWzbBldc\nAddcE3Si4KigRERCYscOuOgi+OYbOPZYeOKJmjUoojgVlIhISNxyC7zxBvzqV/Dii1C/ftCJgqWC\nEhEJgQkTYNQoqFPHl1OrVkEnCp4KSkQkYIsWwYAB/vrjj8MJJwSbJyxUUCIiAVq3Ds47D/LzYeDA\nX4pKVFAiIoHZvh0uuADWrIGuXeHRR4NOFC4qKBGRgNx0E7z9Nuy3n/8ybt26QScKFxWUiEgAxo3z\nw8jr1oWZM/3IPdmdCkpEpJq9//4vX8AdPdp/50n2pIISEalG338P55/v9z9de+0v53mSPamgRESq\nSX4+9OzpS6pbN/j734NOFG4qKBGRanL99fDee9C6Nbzwgv9SrpROBSUiUg3GjIGxY/3hi2bPhhYt\ngk4UfiooEZE4e+cduO46f33sWH92XCmfCkpEJI7WrvX7nXbsgBtugEsvDTpR4lBBiYjESV6eH7G3\nbh2cfDI89FDQiRKLCkpEJA6cg6uvhv/8B9q0gWnT/OnbJXoqKBGROHjiCRg/HlJT/aCIZs2CTpR4\nVFAiIlXszTfhxhv99WeegaOOCjZPolJBiYhUoW++8Uco37nTnyG3d++gEyUuFZSISBXZts2f22n9\neujeHe6/P+hEiU0FJSJSBZzzJxv86CNo1w6mTIGUlKBTJTYVlIhIFRg1CiZOhAYN/KCIpk2DTpT4\nVFAiIpW0YAEMHuyvjx8PRxwRaJykoYISEamEVavgoougoADuuAN69Qo6UfJQQYmIVFBurh8UsWED\nnHEG3H130ImSS1QFZWYTzex7M9tsZsvN7Kp4BxMRCTPn4Mor4ZNP4OCDYdIkDYqoatGuQd0PtHXO\nNQbOAe4xMx2PV0RqrJEjYepUaNjQD4po0iToRMknqoJyzn3unMvfdTNyOTBuqUREQuzf/4bbbvPX\nn38eOnQINk+yivrQhWb2JNAPSAU+Bl4pYZoBwACAli1bkpmZWSUhq1pOTk5os4Wdll3ssrKyKCgo\n0HKrgDC+39aurc+gQV0oLKzDZZetpkmT1YQsIhDOZRcrc85FP7FZCnA8kAE86JzbUdq06enpbtGi\nRZUOGA+ZmZlkZGQEHSMhadnFLiMjg6ysLJYsWRJ0lIQTtvdbTg4cfzx89hmccw7MmgW1QjrULGzL\nrigzW+ycSy9vupgWrXOuwDm3EGgNXF3RcCIiicY56NfPl9Ohh/pNe2Etp2RR0cVbG+2DEpEa5P77\n4cUXoXFjPyiiceOgEyW/cgvKzFqYWW8za2hmKWb2B6APsCD+8UREgvfyyzBkCJjB5MnQvn3QiWqG\naAZJOPzmvNH4QvsauME5NzeewUREwmD5cujb12/iGzECzjwz6EQ1R7kF5ZxbD5xYDVlEREJl82bo\n0QOys+H88/2hjKT6aBefiEgJCgvhj3+EL7+Ejh39QWA1KKJ6aXGLiJRgxAiYM8cfIWL2bGjUKOhE\nNY8KSkSkmLlzYdgwPyhiyhQ46KCgE9VMKigRkSK+/BIuucRfv/9+OO20YPPUZCooEZGIrCw491zY\nsgUuvBBuvTXoRDWbCkpEBD8o4pJLYMUK+M1v4Jln/CY+CY4KSkQEGDrUfyG3aVM/KKJBg6ATiQpK\nRGq8F1+Ee+7xw8inTYMDDgg6kYAKSkRquM8+g8su89cfeghOPTXYPPILFZSI1FibNvkjRWzd6g9n\ndOONQSeSolRQIlIjFRRAnz6wciV06gRjx2pQRNiooESkRrrzTn/q9mbN/IkH09KCTiTFqaBEpMaZ\nNg0efBBSUuCFF6BNm6ATSUlUUCJSo3zyCVx+ub/+yCMQ0rOiCyooEalBNmzwgyK2bfMj9669NuhE\nUhYVlIjUCDt3wkUXwerVcPTRMHq0BkWEnQpKRGqEv/wFFiyAFi1g5kyoXz/oRFIeFZSIJL2JE+Hv\nf4fatf1RI1q3DjqRREMFJSJJ7aOPoH9/f/3RR+G3vw02j0RPBSUiSevHH/2giLw8uOoqGDQo6EQS\nCxWUiCSlHTv8OZ2+/RaOOw4ef1yDIhKNCkpEktLNN8Obb8K++/r9TvXqBZ1IYqWCEpGkM348PPYY\n1Knjy2m//YJOJBWhghKRpPKf//yyr+nJJ+H444PNIxWnghKRpPHDD3D++ZCfD1df7QdGSOJSQYlI\nUti+HXr1grVr/VDyf/wj6ERSWSooEUkKN9wA77wDrVrBjBlQt27QiaSyVFAikvCefhqeesqP1Js1\nC1q2DDqRVAUVlIgktHffhT/9yV8fPdofCFaSgwpKRBLWd99Bz57+S7nXXw/9+gWdSKqSCkpEElJ+\nvi+nH36AE0+EkSODTiRVTQUlIgnHOb9Z7/334de/9qdtr1Mn6FRS1VRQIpJwRo+GceP8OZ1mzYLm\nzYNOJPGgghKRhPL2235/E8A//wmdOwebR+JHBSUiCWPNGv9l3J074aaboG/foBNJPKmgRCQh5OXB\neef5czydcgo8+GDQiSTeVFAiEnrO+QPALloEbdvCtGn+9O2S3FRQIhJ6jz0Gzz0HaWkwezbss0/Q\niaQ6qKBEJNQyM/3+JoBnn4Ujjww0jlSjcgvKzOqZ2Tgz+9rMtpjZEjM7vTrCiUjN9sMP9bjgAigo\ngL/8xZ/CXWqOaNagagPfAicCewFDgOlm1jZ+sUSkpsvNhbvuOpyffoI//AHuvTfoRFLdyt3N6Jzb\nCgwrctc8M1sFdAFWxyeWiNRkzkH//rBiRSMOPBCmTIGUlKBTSXWLeRyMmbUEDgE+L+GxAcAAgJYt\nW5KZmVnZfHGRk5MT2mxhp2UXu6ysLAoKCrTcYjB9emsmTz6I+vV3cuedH/PJJ1uDjpRwkuF31Zxz\n0U9sVgeYD6x0zg0sa9r09HS3aNGiSsaLj8zMTDIyMoKOkZC07GKXkZFBVlYWS5YsCTpKQnjtNb9J\nr7AQhg//jLvuOjzoSAkpzL+rZrbYOZde3nRRj+Izs1rA88B24NpKZBMRKdH//gcXXeTLacgQ6Nbt\np6AjSYCiKigzM2Ac0BLo6ZzbEddUIlLjbN0KPXrAxo1w1lkwfHjQiSRo0e6Dego4DDjVObctjnlE\npAZyDq64ApYuhUMOgYkToZa+pVnjRfM9qDbAQOAo4Aczy4lcdJhGEakSf/sbTJ8OjRr5I0XstVfQ\niSQMohlm/jVg1ZBFRGqgf/0Lbr/dX584EQ47LNg8Eh5aiRaRwPz3v9Cnj9/EN3w4nHNO0IkkTFRQ\nIhKILVv8oIisLP/vkCFBJ5KwUUGJSLUrLITLLoPPP/eb9J57ToMiZE96S4hItbvvPpg1yw+GmD0b\nGjcOOpGEkQpKRKrVvHlw111gBpMn+2HlIiXROSlFpNosWwZ9+/pBEffeC2ecEXQiCTOtQYlItcjO\nhnPPhc2boVevX4aWi5RGBSUicVdYCJde6tegDj/cnxnX9O1KKYcKSkTibvhweOkl2HtvPyiiYcOg\nE0kiUEGJSFzNng133+2HkU+dCgceGHQiSRQqKBGJmy++8Jv2AB54ALp3DzaPJBYVlIjERVaWHxSR\nkwO9e8PgwUEnkkSjghKRKldQABdf7I+1d9RRMG6cBkVI7FRQIlLl7roL5s+HffbxR4xISws6kSQi\nFZSIVKkXXvCHMkpJgWnToG3boBNJolJBiUiV+fRT6NfPXx85Ek45JdA4kuBUUCJSJTZu9KfNyM31\nI/f+/OegE0miU0GJSKXt3OlH6q1aBV26wJgxGhQhlaeCEpFKu+MOePVVaN4cZs6E1NSgE0kyUEGJ\nSKVMmQIPPQS1a8OMGfDrXwedSJKFCkpEKmzJErjySn/9H/+Abt2CzSPJRQUlIhXy009+UMS2bXD5\n5XDNNUEnkmSjghKRmO3cCRdeCF9/DcccA08+qUERUvVUUCISs1tugTfegJYt/aCI+vWDTiTJSAUl\nIjF5/nm/v6lOHXjxRWjVKuhEkqxUUCIStUWLoH9/f/2xx6Br12DzSHJTQYlIVNatg/POg/x8GDAA\nBg4MOpEkOxWUiJRrxw644AJYswZOOAEefTToRFITqKBEpFw33ghvvw377ee/jFuvXtCJpCZQQYlI\nmZ55Bp54AurW9SP29t036ERSU6igRKRUH3wAV1/trz/1FBx7bLB5pGZRQYlIib7/Hs4/H7Zvhz/9\nCa64IuhEUtOooERkD9u3Q69e8N13/vh6jzwSdCKpiVRQIrKH66+Hd9+F1q39Kdzr1Ak6kdREKigR\n2c2YMf5Srx7MmgUtWgSdSGoqFZSI/Oydd+C66/z1sWMhPT3YPFKzqaBEBIC1a6FnT/+l3BtugD/+\nMehEUtOpoESEvDw/Ym/dOjjpJH+GXJGgRVVQZnatmS0ys3wzGx/nTCJSjZzzw8j/8x9o0wamTfOn\nbxcJWrRvw++Ae4A/AKnxiyMi1e3JJ/3RIlJT/aCI5s2DTiTiRVVQzrmZAGaWDrSOayIRqTZvveX3\nNwGMGwedOgWbR6Qo7YMSqaG+/dZ/GXfnThg8GPr0CTqRyO6qdEuzmQ0ABgC0bNmSzMzMqnz6KpOT\nkxPabGGnZRe7rKwsCgoKQrXc8vNrcf31nVi/vhHp6Rs57bSlZGa6oGPtQe+3ikuGZVelBeWcGwuM\nBUhPT3cZGRlV+fRVJjMzk7BmCzstu9g1adKErKys0Cw35/wQ8uXLoV07+Pe/m9K06YlBxyqR3m8V\nlwzLTpv4RGqYUaNg4kRIS4PZs6Fp06ATiZQsqjUoM6sdmTYFSDGz+sBO59zOeIYTkar1+ut+fxPA\n+PFwxBGBxhEpU7RrUEOAbcBtwCWR60PiFUpEqt7q1XDhhVBQALff7k/hLhJm0Q4zHwYMi2sSEYmb\n3Fzo0QM2bIDTT4cRI4JOJFI+7YMSSXLOwZVXwiefwMEHw+TJkJISdCqR8qmgRJLcww/D1KnQsKEf\nFNGkSdCJRKKjghJJYv/3f/CXv/jrEyZAhw7B5hGJhQpKJEmtXAm9e0NhIdx1F5x3XtCJRGKjghJJ\nQjk5flDEpk1w9tkwdGjQiURip4ISSTLOweWXw2efQfv2/ku5tfSbLglIb1uRJPPAAzBjBjRuDHPm\n+H9FEpEKSiSJvPIK3HknmMGkSX4NSiRRqaCqQUZGBtdee23QMSTJrVgBF1/sN/HdfTecdVbQiUQq\nRwUF9OvXj7P02ywJbMsWOPdcyM72o/XuuCPoRCKVp4ISSXCFhf70GV9+6b/n9NxzGhQhyUFv43Jk\nZ2czYMAAWrRoQaNGjTjxxBNZtGjRz49v2LCBPn360Lp1a1JTU+nYsSPPPvtsmc+5YMECmjRpwujR\no+MdX2qAe+755QgRc+ZAo0ZBJxKpGiqoMjjnOPPMM1m7di3z5s3j448/plu3bpx88sl8//33AOTl\n5dG5c2fmzZvH559/zp///GcGDhzIggULSnzOGTNmcN555zF27FgGDRpUnS9HktDcuf47TmYwZQoc\ndFDQiUSqTpWeUTfZvPHGGyxZsoT169eTmpoKwIgRI3jppZd4/vnnufXWW2nVqhW33HLLzz8zYMAA\nXn/9daZMmcIpp5yy2/ONHTuWW265hRkzZtC9e/dqfS2SfL76Ci65xF+/7z447bRg84hUNRVUGRYv\nXkxubi7Nmzff7f68vDxWrlwJQEFBAQ888ADTpk1j7dq15Ofns3379j1OtTx79mzGjBnDW2+9xfHH\nH19dL0GSVHa2HxSxZYs/r9Ou4+2JJBMVVBkKCwtp2bIlb7/99h6PNY58+3HkyJE8/PDDjBo1iiOO\nOIKGDRtyxx138OOPP+42/ZFHHsnSpUsZN24cxx13HGZWLa9Bkk9hIfTtC8uX+zPiPvus38QnkmxU\nUGXo3Lkz69ato1atWrRr167EaRYuXMjZZ5/NpZdeCvj9VsuXL6dJsXMaHHDAATz22GNkZGQwYMAA\nxo4dq5KSChk6FF5+GZo29YMjGjQIOpFIfGiQRMTmzZtZsmTJbpeDDjqIrl27cu655zJ//nxWrVrF\ne++9x9ChQ39eqzrkkENYsGABCxcu5KuvvuLaa69l1apVJc6jXbt2vPHGG/zrX/9i4MCBOOeq8yVK\nEpg504/aq1ULpk2DUv5uEkkKKqiIt99+m06dOu12ueWWW3jllVc4+eST6d+/P+3bt+fCCy9k2bJl\n7LfffgAMGTKEY445htNPP51u3brRoEED+vbtW+p8DjzwQDIzM5k/f75KSmLy2Wf++04Af/sbnHpq\nsHlE4k2b+IDx48czfvz4Uh8fNWoUo0aNKvGxvffem5kzZ5b5/JmZmbvdPvDAA/n2229jjSk12KZN\n/vQZW7f6wxnddFPQiUTiT2tQIiFXUAB9+vgTEHbqBE8/rUERUjOooERCbsgQ+Pe/oVkzmDUL0tKC\nTiRSPVRQIiE2fbo/v1NKir/epk3QiUSqT1IX1M6dOxkzZgwbNmwIOopIzD75xJ8ZF+Dvf4eTTgo2\nj0h1S9qC+vbbbznmmGO47rrruOCCCzRaThLKhg1+UERuLlx2GVx3XdCJRKpfUhbUnDlz6NixI59+\n+ik7duzggw8+YOTIkUHHEonKzp3QuzesXg3p6TB6tAZFSM2UVAWVn5/PoEGDuPjii9myZQsFBQUA\n5ObmMnTo0N1OkyESVrfdBq/vsgn3AAAKYElEQVS9Bi1a+C/m1q8fdCKRYCRNQa1YsYIjjzySCRMm\nkJubu8fjzjlWrFgRQDKR6E2aBA8/DLVrw4wZsP/+QScSCU5SfFF34sSJDBo0iNzc3D32NdWpU4fG\njRsza9Ysfve73wWUUKR8H30EV13lrz/6KOjtKjVdQhfU1q1b6d+/P3PmzClxrSktLY1jjz2WF154\ngX322SeAhCLRWb8ezjsP8vLgyitB57IUSeBNfEuXLqVDhw7MmjWrxHJKTU1l+PDhLFiwQOUkobZj\nB1x4IXzzDRx3HDzxhAZFiEACrkE553jqqacYPHgw27Zt2+PxevXq0bRpU+bOnUt6enoACUViM3gw\nZGbCr34FL74I9eoFnUgkHBKqoLKysrjkkkt44403SiyntLQ0fv/73zNhwoSfTygoEmbjx/v9TXXq\n+BF7kYPkiwgJVFAffPAB55xzDtnZ2eTn5+/xeFpaGo888gj9+/fXiQAlIXz44S/7mp54Ao4/Ptg8\nImET+oIqLCzkgQce4J577ilxral+/frsu+++zJs3jw4dOgSQUCR269b5QRH5+b6k+vcPOpFI+IS6\noH788Ud69erF4sWLS92k17NnT8aMGUNqamoACUVit3079OoFa9dC165QyqnGRGq8QEfxbdiwgY8+\n+qjEx15//XUOPfRQ3n///T1G6dWqVYsGDRowbtw4JkyYoHKShHLDDbBwIbRq5b+MW7du0IlEwinQ\ngrrmmmvo2rUrK1eu/Pm+nTt3ctttt3HWWWexadMmduzYsdvPpKWlceihh/Lpp5/Su3fv6o4sUin/\n/Cc89ZQfqTdzph+5JyIlC6ygli9fzty5c9m+fTtnn30227dvZ82aNRx77LE89thjJW7SS01N5cor\nr+Tjjz+mXbt2AaQWqbj33oM//clff+opOOaYYPOIhF1U+6DMrCkwDugO/ATc7pybXJkZ33bbbezY\nsYPCwkJWr15Njx49WLhwIbm5uT8f5PXnkLVrk5aWxuTJkznzzDMrM1uRQOzYUYuePf3+p+uu++U8\nTyJSumgHSTwBbAdaAkcBL5vZJ865zysy0y+++IL58+f/XETbtm3j9ddfL3X4eMeOHZk1axatWrWq\nyOxEApWXB6tWNWDbNjjxRH8wWBEpn5V3Ij8zawBsAg53zi2P3Pc8sNY5d1tpP9eoUSPXpUuXEh9b\nunQpGzduLDdcrVq1aN26NW3btq3S7zZlZWXRpEmTKnu+mkTLbnfO+fM3lXbZvh3WrFkCQL16R9Gl\ni/9SrkRH77eKC/Oye/PNNxc758o91E80a1CHADt3lVPEJ8CJxSc0swHAAPBHEc/KytrjybZt28am\nTZvKnWlKSgpt27alYcOGZGdnRxEzegUFBSVmk/Il27JzDgoKrMIX56L7wyklpZCDDspm61ad2TkW\nyfZ+q07JsOyiKaiGwOZi92UDjYpP6JwbC4wFSE9PdyWdILB79+7lnpepTZs2LFq0iGbNmkURL3aZ\nmZlkZGTE5bmTXdiWXX4+ZGWVfcnOLv2xEsbixCQlBfbaC5o0Kf0yY0YGZlksWfJx1bzoGiRs77dE\nEuZlF+0WsWgKKgcofmC7xsCWGDOxePFiFi5cuMc5m4r78ccfWbp0KSeddFKss5AEk5dXfsGUVTZ5\neZWbf0oK7L23L5LyiqakS4MG5R95fMECn1VEYhNNQS0HapvZwc65Xas+RwIxD5C4+eabyYviE2Xb\ntm307NmTZcuW0bx581hnI9XEudgLpnjRlDAuJia1a/9SMEUv0ZZNWppObSESVuUWlHNuq5nNBO42\ns6vwo/jOBU6IZUbvv/8+H374YblrT7ts2bKFG2+8kYkTJ8YyG4mBc5CbG92msF2Xb7/tTEHBL7eL\nfY86ZnXqlFww0ZZNaqoKRiRZRTvM/BrgGeBHYANwdaxDzG+66aYSTyxYr1496tWrR35+PnXr1uWQ\nQw7h6KOPpkuXLtrEVw7nYOvW2Pa5FL/s3BnrXHff2lu3bvkFU1bR1K+vghGRkkVVUM65jUCPis7k\n3Xff5b333qNRo0YUFBRQUFBAu3bt6Ny5M0cffTS/+c1v6NixIy1atKjoLBKSc5CTU/Ed/FlZUOw7\nzTGrXz+2fS4rVy7mlFO6/Fw29etXzbIQESmuWo5mvs8++3DfffdxxBFHcPjhh9OmTZukOGdTYWH5\nBVNe0RQWVi5DWlrs+12KTh/r2VszM7fQvn3lMouIRKNaCqp9+/bcfvvt1TGrmBQWwpYtFd/Bn51d\n+YJp0CD2/S5Fp9GRsEUkWYX6fFDlKSiAzZtj3+/yww/HkZfnfzbKMRulatiwYvtedt2vowqIiJQs\n0IIqKNizWGIpms3Fvz4ctV92nDRqFNsmseK3ayd0xYuIhFfcPl7XrYO77iq7YLbE/FXfPe21V+z7\nXr766n3+8IfjaNxYBSMiElZx+3heswZGjCh7GrPdyyXWomnUyB8JIFbZ2Xk0bVqx1yUiItUjbgXV\nogVcc035BVMr0HP6iohIWMWtoPbfH4YOjdezi4hIstP6i4iIhJIKSkREQkkFJSIioaSCEhGRUFJB\niYhIKKmgREQklFRQIiISSiooEREJJRWUiIiEkgpKRERCyVxlT4hU2hObrQe+jsuTV14z4KegQyQo\nLbuK0XKrGC23igvzsmvjnGte3kRxK6gwM7NFzrn0oHMkIi27itFyqxgtt4pLhmWnTXwiIhJKKigR\nEQmlmlpQY4MOkMC07CpGy61itNwqLuGXXY3cByUiIuFXU9egREQk5FRQIiISSiooEREJJRUUYGYH\nm1memU0MOkvYmVk9MxtnZl+b2RYzW2JmpwedK6zMrKmZzTKzrZFldnHQmcJO77GqkQyfayoo7wng\nw6BDJIjawLfAicBewBBgupm1DTBTmD0BbAdaAn2Bp8ysY7CRQk/vsaqR8J9rNb6gzKw3kAUsCDpL\nInDObXXODXPOrXbOFTrn5gGrgC5BZwsbM2sA9AT+6pzLcc4tBOYClwabLNz0Hqu8ZPlcq9EFZWaN\ngbuBm4LOkqjMrCVwCPB50FlC6BBgp3NueZH7PgG0BhUDvcdik0yfazW6oIARwDjn3JqggyQiM6sD\nTAKec859FXSeEGoIbC52XzbQKIAsCUnvsQpJms+1pC0oM8s0M1fKZaGZHQWcCjwSdNYwKW+5FZmu\nFvA8fv/KtYEFDrccoHGx+xoDWwLIknD0Hotdsn2u1Q46QLw45zLKetzMbgDaAt+YGfi/dlPMrINz\nrnPcA4ZUecsNwPwCG4ff8X+Gc25HvHMlqOVAbTM72Dm3InLfkWhTVbn0HquwDJLoc63GHurIzNLY\n/a/bwfj/2Kudc+sDCZUgzGw0cBRwqnMuJ+g8YWZmUwEHXIVfZq8AJzjnVFJl0HusYpLtcy1p16DK\n45zLBXJ33TazHCAvEf8Tq5OZtQEGAvnAD5G/0gAGOucmBRYsvK4BngF+BDbgPyhUTmXQe6ziku1z\nrcauQYmISLgl7SAJERFJbCooEREJJRWUiIiEkgpKRERCSQUlIiKhpIISEZFQUkGJiEgoqaBERCSU\n/h9r5scSI6iwhAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tdh3Jj9kRwG2",
        "colab_type": "code",
        "outputId": "031918ca-d22d-4719-ba05-27f4404ebb08",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 276
        }
      },
      "source": [
        "[m for m in dir(keras.activations) if not m.startswith(\"_\")]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['deserialize',\n",
              " 'elu',\n",
              " 'exponential',\n",
              " 'get',\n",
              " 'hard_sigmoid',\n",
              " 'linear',\n",
              " 'relu',\n",
              " 'selu',\n",
              " 'serialize',\n",
              " 'sigmoid',\n",
              " 'softmax',\n",
              " 'softplus',\n",
              " 'softsign',\n",
              " 'tanh']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kiyo1-9FRwG_",
        "colab_type": "code",
        "outputId": "d00b4a0f-cd92-4b7c-8ec1-5c5e96b3e093",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "[m for m in dir(keras.layers) if \"relu\" in m.lower()]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rVq1D-FKRwHG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "leaky_relu = keras.layers.LeakyReLU(alpha=0.2)\n",
        "layer = keras.layers.Dense(10, activation=leaky_relu)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t47Em-KjRwHM",
        "colab_type": "code",
        "outputId": "9a4044e0-9c96-4fc3-fcff-bb449814b353",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "layer.activation"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.advanced_activations.LeakyReLU at 0x7f6eaaffc518>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q4CNTjQ4RwHT",
        "colab_type": "text"
      },
      "source": [
        "Let's train a neural network on Fashion MNIST using the Leaky ReLU:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sVePHdBORwHV",
        "colab_type": "code",
        "outputId": "f6c0babc-0bac-4126-8a38-7a186efb4616",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 165
        }
      },
      "source": [
        "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
        "X_train_full = X_train_full / 255.0\n",
        "X_test = X_test / 255.0\n",
        "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
        "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "8192/5148 [===============================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dyPZ7t7-RwHb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=leaky_relu),\n",
        "    keras.layers.Dense(100, activation=leaky_relu),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRGUzDiqRwHm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0i5oUkUhRwHv",
        "colab_type": "code",
        "outputId": "78f0063b-a358-4518-edbc-0f385db2b56d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        }
      },
      "source": [
        "history = model.fit(X_train, y_train, epochs=10,\n",
        "                    validation_data=(X_valid, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/10\n",
            "55000/55000 [==============================] - 5s 94us/sample - loss: 1.3638 - accuracy: 0.6008 - val_loss: 0.9190 - val_accuracy: 0.7156\n",
            "Epoch 2/10\n",
            "55000/55000 [==============================] - 5s 86us/sample - loss: 0.8236 - accuracy: 0.7333 - val_loss: 0.7321 - val_accuracy: 0.7606\n",
            "Epoch 3/10\n",
            "55000/55000 [==============================] - 5s 87us/sample - loss: 0.7021 - accuracy: 0.7700 - val_loss: 0.6501 - val_accuracy: 0.7900\n",
            "Epoch 4/10\n",
            "55000/55000 [==============================] - 5s 86us/sample - loss: 0.6375 - accuracy: 0.7905 - val_loss: 0.5995 - val_accuracy: 0.8100\n",
            "Epoch 5/10\n",
            "55000/55000 [==============================] - 5s 87us/sample - loss: 0.5946 - accuracy: 0.8044 - val_loss: 0.5660 - val_accuracy: 0.8192\n",
            "Epoch 6/10\n",
            "55000/55000 [==============================] - 5s 88us/sample - loss: 0.5641 - accuracy: 0.8142 - val_loss: 0.5410 - val_accuracy: 0.8244\n",
            "Epoch 7/10\n",
            "55000/55000 [==============================] - 5s 91us/sample - loss: 0.5408 - accuracy: 0.8198 - val_loss: 0.5206 - val_accuracy: 0.8346\n",
            "Epoch 8/10\n",
            "55000/55000 [==============================] - 5s 90us/sample - loss: 0.5226 - accuracy: 0.8251 - val_loss: 0.5044 - val_accuracy: 0.8372\n",
            "Epoch 9/10\n",
            "55000/55000 [==============================] - 5s 90us/sample - loss: 0.5078 - accuracy: 0.8292 - val_loss: 0.4915 - val_accuracy: 0.8394\n",
            "Epoch 10/10\n",
            "55000/55000 [==============================] - 6s 108us/sample - loss: 0.4956 - accuracy: 0.8319 - val_loss: 0.4841 - val_accuracy: 0.8414\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "diZXbkDTRwH5",
        "colab_type": "text"
      },
      "source": [
        "### ELU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-Vy6oyFRwH6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def elu(z, alpha=1):\n",
        "    return np.where(z < 0, alpha * (np.exp(z) - 1), z)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2G6IibtXRwH_",
        "colab_type": "code",
        "outputId": "1ca2881f-a6fc-4b72-bec6-98d9a42388bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        }
      },
      "source": [
        "plt.plot(z, elu(z), \"b-\", linewidth=2)\n",
        "plt.plot([-5, 5], [0, 0], 'k-')\n",
        "plt.plot([-5, 5], [-1, -1], 'k--')\n",
        "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
        "plt.grid(True)\n",
        "plt.title(r\"ELU activation function ($\\alpha=1$)\", fontsize=14)\n",
        "plt.axis([-5, 5, -2.2, 3.2])\n",
        "\n",
        "save_fig(\"elu_plot\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving figure elu_plot\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8FeW9x/HPLwnKKiBorCJgXVDr\nwhWq1bqkalUWt7q2asUNKtqWqq0b9Gql2ipWqApKixcFF1CwKgh41XvABaWgIFAFRECQfTlAgARI\nnvvHc4CQ9SSZZOac832/XueVyTxzZn5nGM43sz1jzjlERESiJivsAkRERMqjgBIRkUhSQImISCQp\noEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSnDzIab2bg0Wk6WmT1rZuvMzJlZXl0vs5Ja\n6uUzJ5bV0sxWmdnh9bG86jKzV83szrDrEDD1JJGezGw4cH05TZ86536UaG/tnOtewftjwBzn3O2l\nxvcAnnLONQ204OSW3Ry/zcZTaTmVLL87MBbIA74B1jvnttflMhPLjVHqc9fXZ04s6zH8tndDXS+r\nnGWfCdwFdAIOBm5wzg0vNc3xwGTgMOfcxvquUfbICbsAqVPvAteVGlfnX4B1pb6+LOrxS+kIYIVz\n7uN6Wl6F6uszm1lj4GbgwvpYXjmaAnOAFxKvMpxzs83sG+Ba4Ol6rE1K0SG+9FbonFtZ6rW+rhdq\nZheY2QdmtsHM1pvZJDM7pkS7mdmdZrbAzArNbJmZPZJoGw6cBdyWOOzlzKz9rjYzG2dmPROHiLJL\nLfclM3szmTqSWU6J+exrZgMTyywws0/M7PQS7TEzG2xmD5vZWjNbbWYDzKzC/1+J5T8BtE0se3GJ\neT1Vetpd9SSzrJqs3+p+5pp+bqAr4ICPylknnczsPTPbZmZfm9mZZnalmZWZtqacc2875+5zzr0G\nFFcy6ZvAz4NartSMAkrqQhNgIHAy/vDVRuAtM9sn0f4w0A94BPgBcAWwNNH2W2Aq8D/A9xKvXW27\nvAo0B366a4SZNQUuBkYmWUcyy9nlUeAq4Ebgv4DZwEQz+16Jaa4BdgKnAbcDfRLvqchvgT8ByxLL\n/mEl05ZW1bJqu34huc+cTC2lnQHMcKXOLZjZD4EPgP8DTgA+AR4E7k98FkpNf5+Z5VfxOqOSOqoy\nDTjZzBrVYh5SSzrEl94uMLP8UuOeds7dXZcLdc6NKfm7md0AbML/h58J/A7o45x7LjHJ1/gvTZxz\nG81sO7DVObeygvlvMLO38V+OExOjL8F/Ub5ZYroK63DOfVjVchLvaQLcCtzsnBufGPcr4GzgNqBv\nYtL/OOf+mBieb2a3AOcAL1fwGTaa2WagqLLlV6DCZSWCutrr18xq8pmr/bmBdsDycsY/DrzlnOuf\nWN5LwFvAFOfc++VM/wwwuoJl7PJdFe2VWQ40wJ+nWliL+UgtKKDS2xSgZ6lx9XES/HDgIeAU4AD8\nnnoW0BZ/Dmxf4L1aLmYk8LyZNXbObcWH1RjnXEGSdSTrcPwX1e7DTM65IjObChxbYrovSr1vOXBg\nNZZTHZUt61hqv36T/cxV1VKeRsCqkiPM7CD8ntVPSozejv+3KrP3lKhnPVCXh6u3JX5qDypECqj0\nttU593UN37sJfxittBb4Q2WVGYc/dNUL/1fsTuA/wD6Vvamaxifme7GZvQecC5xfz3WUPEy1o5y2\nmhxCLwas1LgGpX4Palk1Ufqy3+rWshZoWWrcrvOT00uM6wDMc859WN5MzOw+4L7KS6WLc+6DKqap\nyP6Jn2tq+H4JgAJKKjIP6GpmVup8wUmJtnKZWSvgaKC3c+7/EuNOYs+29iVQiD8MtKCC2WwHsito\nA8A5V2hmr+L3nFoDK4FYNepIajn4wzvbgR8nhjF/ccapwEtVvLcm1uDPC5V0IrA4yfcHsX7r8jN/\nDvQoNa4FPtiKEstqhj/3VNmhz7o+xHcc8J1zblWVU0qdUUClt30Th09KKnLO7fqrcD8z61iqPe6c\nWwwMwZ/0ftLM/gEU4K/A+jlwUSXL3ID/K/kWM1sKHAI8ht97wTm32cwGAY+YWSH+MGQroJNzbkhi\nHovx56vaA/n4+4PKu+JqJP5Q1mHAy6WmqbSOZJfjnNtiZkOAv5rZWmAR/hxPLjC4kvVQU+8DA83s\nIvwfAr2AQ0kyoGq6fkvNoy4/86TEfFs559Ylxs3E7zXea2Yv4v+dVgBHmNmRzrkyQVvTQ3yJc3RH\nJH7Nwl9F2RH/b/9tiUnPSNQqIdJVfOntXPx/9JKvz0u0n5H4veRrAIBz7hvgTOBI4B38VU1XA1c4\n5yZUtMDEF/xV+Cux5uDvI+mH/6t+l3uBvybGfwmMAdqUaB+A/wv+P/g9iorOGX2A/yv5WPa+ei/Z\nOpJdzt3AKPyVbzMT87zAObeigulr47kSr4+AzcDr1ZxHEOu3Tj6zc242e7alXeMW4feYbgVm4T/z\nufh/t6DvEevMnm29Ef5Kwc/xV1QCYGYNgUuBfwS8bKkm9SQhIvXKzC4ABgHHOueKwq6nNDO7DbjY\nOXde2LVkOu1BiUi9cs5NxO/Rtqlq2pDsAH4ddhGiPSgREYko7UGJiEgkKaBERCSSQr/MvHXr1q59\n+/Zhl1HGli1baNKkSdhlpBSts+TNmzePoqIijj22dMcMUpFU276WLIG1ayE7Gzp0gEYh9EkR1XU2\nY8aMtc65A6qaLvSAat++PdOnT696wnoWi8XIy8sLu4yUonWWvLy8POLxeCS3/ahKpe3rj3+Ehx6C\nhg3h3Xfhxz8Op46orjMzW5LMdDrEJyISoKef9uGUnQ2jR4cXTulAASUiEpBXX4VfJy5QHzoULgzr\nsYxpQgElIhKA99+Ha68F5+Dhh+HGG8OuKPUFGlBmNtLMVpjZJjObb2Y3Bzl/EZEo+vxzuOQS2L4d\nfvMbuOeesCtKD0HvQT0CtHfO7YfvULS/mXUKeBkiIpGxcCF06QKbN8NVV8ETT4CVfmCK1EigAeWc\nm+uc29UZp0u8Dg9yGSIiUbFqFZx/vv957rnw/POQpRMngQn8MnMzG4x/3ksjfC/Bb5czTU8ST3rN\nzc0lFosFXUat5efnR7KuKNM6S148HqeoqEjrqxqitn1t3ZpNnz4dWbiwGUceuZnf/W4mU6dGq+/b\nqK2z6qqTvvhKPNwsD/irc670Uzd369y5s4vivSBRvX8gyrTOkrfrPqiZM2eGXUrKiNL2VVgI3brB\ne+/B4YfDRx9Bbm7YVZUVpXVWkpnNcM51rmq6OtkZdc4VJR7V3Ab/jBcRkbRQXAzXX+/DKTcXJk2K\nZjilg7o+WpqDzkGJSJpwDvr0gVGjoFkzmDDB70FJ3QgsoMzsQDO72syamlm2mZ2Pfzz4e0EtQ0Qk\nTH/5Czz5JOyzD/zrX/Bf/xV2RektyIskHP5w3jP44FsC9HHOvRngMkREQjFsGNx3n7+EfORIOPvs\nsCtKf4EFlHNuDXBWUPMTEYmKN9+Enj398FNPwRVXhFtPptAV+yIilfjoI38DbnEx9OsHvXuHXVHm\nUECJiFRg7lzo3h0KCuCWW+DBB8OuKLMooEREyrF0KVxwAcTjcPHFMHiwujCqbwooEZFS1q3zXRgt\nWwannw4vvww5oT/eNfMooEREStiyxR/W+/JLOO44f4FEGI9rFwWUiMhuO3b4CyI++QTatoWJE6Fl\ny7CrylwKKBERfC8Rt9wC48dDq1a+C6NDDgm7qsymgBIRAe691z8uo3FjH1JHHx12RaKAEpGM98QT\n8Ne/+gshxoyBU04JuyIBBZSIZLgXX4Q77vDDzz3nLy2XaFBAiUjGeucd6NHDDw8YANddF2o5UooC\nSkQy0r//DT/7GezcCXfe6V8SLQooEck48+dD167+nqdrr4VHHw27IimPAkpEMsqKFb6XiLVr/fmm\n556DLH0TRpL+WUQkY2zc6ENp8WI4+WR49VVo0CDsqqQiCigRyQgFBb7T1y++gA4d/L1OTZuGXZVU\nRgElImmvqAiuuQYmT4aDD/a9RLRuHXZVUhUFlIikNefgtttg7Fho3tz3r9euXdhVSTIUUCKS1v70\nJ3j2Wdh3X3jrLTj++LArkmQpoEQkbT37LDzwgL9K75VX4Iwzwq5IqkMBJSJpaexY6N3bDw8ZApdc\nEm49Un0KKBFJO5Mnwy9+AcXF/hBfz55hVyQ1oYASkbQyaxZcdBEUFvo9qL59w65IakoBJSJpY9Ei\nfyPupk1w+eXw97+DWdhVSU0poEQkLaxZ47swWrkSfvITGDkSsrPDrkpqQwElIikvP993/rpgAXTs\nCK+/7i8rl9SmgBKRlLZ9O1x2GUyfDocdBhMm+BtyJfUpoEQkZRUXww03+AcPHnCA/3nQQWFXJUFR\nQIlISnIO7roLXnrJd/o6YQIccUTYVUmQFFAikpIGDIAnnvCPyxg7Fjp1CrsiCZoCSkRSzvPPwx/+\n4IdfeAF++tNw65G6oYASkZQyfjzcdJMfHjQIrr463Hqk7gQWUGa2r5kNM7MlZrbZzGaaWZeg5i8i\n8skncMUV/vlO994Lv/lN2BVJXQpyDyoHWAqcBTQH+gKjzax9gMsQkQy1ZEljunWDbdvgxhvhz38O\nuyKpazlBzcg5twV4oMSocWa2COgELA5qOSKSeZYtgz/84QTWr4fu3f1jNNSFUfqrs3NQZpYLHAXM\nratliEj627DB96+3enVDTjsNRo2CnMD+tJYoq5N/ZjNrALwIPO+c+6qc9p5AT4Dc3FxisVhdlFEr\n+fn5kawryrTOkhePxykqKtL6qkJhYRZ33XUic+c259BDN3P33bOYNm1n2GWljFT/Pxl4QJlZFjAC\n2A7cXt40zrmhwFCAzp07u7y8vKDLqLVYLEYU64oyrbPktWjRgng8rvVViZ07fRdGc+ZAmzYwYMAc\nLrro9LDLSimp/n8y0IAyMwOGAblAV+fcjiDnLyKZwTn41a/gzTehZUuYNAlWry4MuyypZ0GfgxoC\nHANc6JzbFvC8RSRD9OsHw4ZBo0b+vqdjjw27IglDkPdBtQN6AR2BlWaWn3hdE9QyRCT9Pfmkv4Q8\nOxtGj4ZTTw27IglLkJeZLwF04aeI1NioUfDb3/rhf/7TX1IumUtdHYlIJLz7Llx3nT//9Je/QI8e\nYVckYVNAiUjoPvsMLr0UduyAPn32dAQrmU0BJSKhWrgQunTxj23/+c/h8cfVS4R4CigRCc2qVXDe\nebB6tX9kxvDhkKVvJUnQpiAiodi0ye85ffONf9jgmDGwzz5hVyVRooASkXpXWOjPOX3+uX9M+9tv\nQ7NmYVclUaOAEpF6VVTkr9Z7/3046CB45x048MCwq5IoUkCJSL1xzt/n9OqrsN9+MGECHHZY2FVJ\nVCmgRKTePPwwPP20P9f0xhvQsWPYFUmUKaBEpF7885/Qt6+/hPyllyCFO9mWeqKAEpE698Yb0KuX\nHx482D9GQ6QqCigRqVMffghXXw3FxfDf/+0foyGSDAWUiNSZOXPgwguhoAB69vQBJZIsBZSI1Ikl\nS+D88yEe9/c8DR6sLoykehRQIhK4tWt9OC1fDmee6S+KyM4OuypJNQooEQnUli3+OU7z5sHxx/sL\nJBo2DLsqSUUKKBEJzI4dcMUV8Omn0K4dTJwILVqEXZWkKgWUiASiuBhuusn3DtG6te/C6OCDw65K\nUpkCSkQCcc89MGIENGkC48fDUUeFXZGkOgWUiNTa3/4Gjz0GOTn+sRknnxx2RZIOFFAiUisvvgh3\n3umHhw/3V++JBEEBJSI1NmkS9Ojhhx9/HK65JtRyJM0ooESkRqZN833q7dwJv/893HFH2BVJulFA\niUi1zZsH3br5e55++Uv4y1/CrkjSkQJKRKpl+XJ/nmntWujSxT9GI0vfJFIHtFmJSNLicbjgAt/P\n3imn+CfjNmgQdlWSrhRQIpKUbdvgootg9mw4+mh/r1OTJmFXJelMASUiVSoq8lfoffABHHKIv3qv\nVauwq5J0p4ASkUo5B717w+uv+371Jk6Etm3DrkoygQJKRCr14IMwdKjvkfytt+C448KuSDKFAkpE\nKjRkiA+orCwYNQpOPz3siiSTKKBEpFyvvQa33eaHn33WXyAhUp8UUCJSRizmL4pwDvr3h5tvDrsi\nyUSBBpSZ3W5m082s0MyGBzlvEakfM2fCxRfD9u1w++1w331hVySZKifg+S0H+gPnA40CnreI1LFv\nvvG9Q2zaBFdeCQMHglnYVUmmCjSgnHNjAcysM9AmyHmLSN1avdp3YbRyJZx9NrzwAmRnh12VZLKg\n96CSYmY9gZ4Aubm5xGKxMMqoVH5+fiTrijKts+TF43GKioois762bs3mjjtO5Ouv9+PIIzdzxx0z\nmTq1KOyy9qLtq/pSfZ2FElDOuaHAUIDOnTu7vLy8MMqoVCwWI4p1RZnWWfJatGhBPB6PxPravh26\nd/c9lB9+OHzwQTNyc88Iu6wytH1VX6qvM13FJ5LBiov9Awf/93/hwAN9F0a5uWFXJeIpoEQylHP+\nIYMvvwxNm8KECX4PSiQqAj3EZ2Y5iXlmA9lm1hDY6ZzbGeRyRKT2Hn0UBg3yj8v417/gpJPCrkhk\nb0HvQfUFtgH3ANcmhvsGvAwRqaX/+R+45x5/CfnIkXDOOWFXJFJW0JeZPwA8EOQ8RSRY48bBLbf4\n4UGD/P1OIlGkc1AiGWTqVB9IRUVw//3w61+HXZFIxRRQIhniP/+Bbt38k3FvugkeeijsikQqp4AS\nyQBLl/peIjZs8L2SP/OMujCS6FNAiaS59evhggtg2TL/PKdXXoGcUG7RF6keBZRIGtu6FS680B/e\n+8EP4M03oZG6cZYUoYASSVM7d8JVV8HHH8Ohh8LEidCyZdhViSRPASWShpyDnj39JeX77++7MGqj\n5wtIilFAiaSh++/3N+M2agTjx8Mxx4RdkUj1KaBE0sygQfDII/5ZTq+9Bj/6UdgVidSMAkokjbzy\nCvTp44efew66dg23HpHaUECJpIl334Vf/tIPP/ronmGRVKWAEkkDM2bApZfCjh3+ERp33RV2RSK1\np4ASSXELFkCXLpCfD9dcA489pl4iJD0ooERS2MqVvgujNWvgvPP8eacs/a+WNKFNWSRFbdzouzBa\ntAh++EMYMwb22SfsqkSCo4ASSUEFBXDJJTBrFhx5pL/XqWnTsKsSCZYCSiTFFBXBdddBLAbf+x68\n8w4ccEDYVYkETwElkkKcg9/8xt+Au99+vn+99u3DrkqkbiigRFLIn/8MgwfDvvv6nslPOCHsikTq\njgJKJEX84x/Qr5+/Su+ll+Css8KuSKRuKaBEUsC//gW/+pUfHjwYfvazcOsRqQ8KKJGImzIFrr4a\niovhwQehV6+wKxKpHwookQibPRsuuggKC/0eVL9+YVckUn8UUCIRtXix7yVi40Z/SO+pp9SFkWQW\nBZRIBK1d68NpxQp/McSLL/rnO4lkEgWUSMTk50O3bjB/Ppx4IrzxBjRsGHZVIvVPASUSITt2wOWX\nw7Rp/gbcCROgefOwqxIJhwJKJCKKi+HGG2HSJN910Tvv+K6MRDKVAkokIu6+G0aOhCZN4O23fSew\nIplMASUSAQMG+FeDBvD669C5c9gViYRPASUSshEj4Pe/98PPPw8//Wm49YhEhQJKJEQTJvjzTgBP\nPAE//3m49YhESaABZWb7m9nrZrbFzJaY2S+CnL9IOtm6NZvLL4edO/35pz59wq5IJFpyAp7f08B2\nIBfoCIw3s1nOubkBL0ckpW3dCt9805SiIrj+enjkkbArEokec84FMyOzJsAG4Djn3PzEuBHAd865\neyp6X7NmzVynTp0CqSFI8XicFi1ahF1GStE6S05BAUybNhPnYP/9O3LccerCKBnavqovquts8uTJ\nM5xzVV4KFOQe1FHAzl3hlDALKPPUGjPrCfQEaNCgAfF4PMAyglFUVBTJuqJM66xqO3dmsWBBU5yD\nrCzHIYdsZOPGYP5ITHfavqov1ddZkAHVFNhUatxGoFnpCZ1zQ4GhAJ07d3bTp08PsIxgxGIx8vLy\nwi4jpWidVS4e9/3qbd8OTZvm0b79Rr744vOwy0oZ2r6qL6rrzJI8ZBBkQOUD+5Uatx+wOcBliKSk\njRuha1f44gvo0AFatYItW7TnJFKZIK/imw/kmFnJ+99PBHSBhGS0DRv8vU1Tp0Lbtr4LowYNwq5K\nJPoCCyjn3BZgLPAnM2tiZj8GLgZGBLUMkVSzdi2ccw78+99w2GEwebIPKRGpWtA36vYGGgGrgZeB\nW3WJuWSq1avh7LPh8899v3qTJ/seykUkOYHeB+WcWw9cEuQ8RVLRwoXQpQssWABHHw3vv6+eyUWq\nS10diQRs2jQ49VQfTh07QiymcBKpCQWUSIDeegvy8mDNGjjvPJgyBXJzw65KJDUpoEQC4Bz8/e9w\nySWwbRvccAOMGwfNytwFKCLJUkCJ1NK2bdCjB/z2t/6puH/8IwwbpkvJRWor6M5iRTLKkiXws5/B\nZ59B48Y+mK6+OuyqRNKDAkqkhiZNgmuv9fc6ff/7/km4J5wQdlUi6UOH+ESqqbAQ7rwTLrjAh9P5\n5/sbcRVOIsHSHpRINcyb5596+/nnkJ0NDz0Ef/iDHxaRYCmgRJJQVARPPQX33ecfNvj978NLL8Ep\np4RdmUj6UkCJVOHLL+Gmm3xnrwDXXefDar/SffeLSKB0DkqkAoWF0L+/7w1i6lQ4+GB44w144QWF\nk0h90B6USDnGj4c+feDrr/3vN98Mjz0GEXx6tkjaUkCJlLBgAfzudz6gAI45xh/OO/vscOsSyUQ6\nxCcCrFwJt90Gxx7rw6lZM/jb32DWLIWTSFi0ByUZLR6HAQPgiSf81XlZWb4fvYcfhoMOCrs6kcym\ngJKMtHYtDBoETz4JGzf6cRdfDH/+M/zgB+HWJiKeAkoyynff+b2lIUP8HhPAT37ig+nUU8OtTUT2\npoCSjDBtGgwcCK++Cjt3+nFdu8L998Npp4Vbm4iUTwElaWvLFnjtNXjmGfjkEz8uOxuuvBLuvhtO\nOinc+kSkcgooSSvOwaef+sdejBoFmzf78S1bQs+e/kq9Qw8Nt0YRSY4CStLCt9/C6NHw3HO+a6Jd\nTjsNbrzRP6OpSZPw6hOR6lNAScpavNgfwnv1VX+OaZcDD4Trr/fBdPTRoZUnIrWkgJKU4Zy/cXbC\nBBg7FqZP39PWuDF06wa/+IX/qceti6Q+BZRE2oYN8O67PpQmToQVK/a0NWkC3bvDFVdAly4+pEQk\nfSigJFI2boQPP4TJk2HKFL+XVFS0p/3gg/2TbLt18z8VSiLpSwEloXEOlizx54+mTvWhNGsWFBfv\nmSYnB846y+8hdekCxx8PZuHVLCL1RwEl9cI534vDF1/4vaJp0/xrzZq9p2vQAH70Ix9KZ54JP/6x\n77hVRDKPAkoCt2kTzJkDs2fv/dqwoey0rVvDD38IJ58MZ5zhuxvSYTsRAQWU1FBhIXzzjX9+0q7X\ntGknsnYtLF1a/nv2398fouvUyQfSySdD+/Y6ZCci5VNASbk2bfI3vy5dWvbn4sV+uOS5Iq8lAPvs\n45+rdPzxe14nnADf+57CSESSp4DKIEVFsG4drFq192v1av9z5UpYtsyHz6ZNlc8rKwu+/3048sg9\nr23bvuCyy06gXTvdhyQitaeASkGFhb6PuQ0b/Gv9+sp/btjgL0ZYs6a8vZ7yNWoEbdv6fuvK+3nY\nYX5PqaRYbD1HHBH85xWRzBRIQJnZ7UAP4HjgZedcjyDmm4qcgx07YNs2/yooqHp461bIz/ehs3nz\nnuGKxu3YUfP6WraE3Nw9r4MO2vv3Qw7xAbT//jocJyLhCmoPajnQHzgfaFSdNxYWwvz5/vBT6Vdx\ncfnjq2qrqn3Hjj2v7dv3/rlr+LvvjmXQoKqn2zW8K3AKCpLfS6mpnBx/6XXLlj5IWrbce7i8ca1a\n+T7qSu/1iIhEVSAB5ZwbC2BmnYE21XnvnDnz6NAhr9TYK4HewFagaznv6pF4rQUuL6f9VuAqYClw\nXTntdwIXAvOAXuW09wXOBWYCfcppfxg4DfgYuK9Ma3b2QBo37khW1rsUFPQnK4vdr+xsOP74Zzng\ngA6sW/cW8+Y9TnY2e71uvXUE7dodyowZo5g4cUiZ9rFjX6N169YMHz6c4cOHs337nvNJAG+//TaN\nGzdm8ODB/P3vo8vUF4vFABgwYADjxo3bq61Ro0ZMmDABgIceeoj33ntvr/ZWrVoxZswYAO69916m\nTp26uy0ej3PccccxcuRIAPr06cPMmTP3ev9RRx3F0KFDAejZsyfz58/fq71jx44MHDgQgGuvvZZl\ny5bt1X7qqafyyCOPAHDZZZexbt26vdrPOecc+vXrB0CXLl3Ytm3bXu3du3fnrrvuAiAvL6/Murny\nyivp3bs3W7dupWvXsttejx496NGjB2vXruXyy8tue7feeitXXXUVS5cu5brrym57d955JxdeeCFb\nt27l66+/LlND3759Offcc5k5cyZ9+pTd9h5++GFOO+00Pv74Y+67r+y2N3DgQDp27Mi7775L//79\ny7Q/++yzdOjQgbfeeovHH3+8TPuIESM49NBDGTVqFEOGDCnT/tpre297pZXc9kaPDnbbKy4uZsqU\nKUDZbQ+gTZs22vZKbXvxeJwWLVoAe7a9efPm0atX2e+9+tz2khXKOSgz6wn09L81YZ99ihOHkxxm\n0KxZAS1bbga2sGzZzsR79hxyOuCAfA48cB1FReuYP3/H7vFmDoC2beMcfPAKCgtXMWvWdsxciWng\nmGPW0K7dYrZsWcbUqQW723fVcPrpSzj00M/YvPkbJk3akmhzu39eeumXdOjQgEWL5jJmzKbd47Oy\n/M9f/3o6RxwRZ8aMWYwYES/z+W+++VPatl3Bxx/PJh4v296mzVRatVpITs5ciovjFBfvfVjvo48+\nonnz5nz11Vflvn/KlCk0bNiQ+fPnl9u+60ti4cKFZdq3bdu2u33RokVl2ouLi3e3f/vtt3u1FxUV\nsWrVqt3ty5YtK/P+5cuX725fvnx5mfZly5btbl+1alWZ9m+//XZ3+5o1a9hU6mqORYsW7W5fv349\nhYWFe7UvXLhwd3t562b+/PlLNR+DAAAF6UlEQVTEYjEKCgrKbf/qq6+IxWJs3Lix3Pa5c+cSi8VY\nvXp1ue2zZ8+mWbNmbN68GedcmWlmzZpFTk4OX3/9dbnv/+yzz9i+fTtz5swpt3369OnE43FmzZpV\nbvunn37KihUrmD27/G1v6tSpLFy4kLlz55bbHua217hx4wq3PYAGDRpo2yu17RUVFe0e3rXtlbfu\noH63vWSZcy7piaucmVl/oE11zkF17tzZTS/ZLXVExGKxcv/KkYppnSUvLy+PeDxe5q98qZi2r+qL\n6jozsxnOuc5VTZeVxIxiZuYqeH0YTLkiIiJ7q/IQn3Murx7qEBER2UtQl5nnJOaVDWSbWUNgp3Nu\nZxDzFxGRzFPlIb4k9QW2AfcA1yaG+wY0bxERyUBBXWb+APBAEPMSERGB4PagREREAqWAEhGRSFJA\niYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSS\nFFAiIhJJCigREYkkBZSIiESSAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBER\niSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJK\nREQiqdYBZWb7mtkwM1tiZpvNbKaZdQmiOBERyVxB7EHlAEuBs4DmQF9gtJm1D2DeIiKSoXJqOwPn\n3BbggRKjxpnZIqATsLi28xcRkcxU64AqzcxygaOAuZVM0xPoCZCbm0ssFgu6jFrLz8+PZF1RpnWW\nvHg8TlFRkdZXNWj7qr5UX2fmnAtuZmYNgAnAQudcr2Te07lzZzd9+vTAaghKLBYjLy8v7DJSitZZ\n8vLy8ojH48ycOTPsUlKGtq/qi+o6M7MZzrnOVU1X5TkoM4uZmavg9WGJ6bKAEcB24PZaVS8iIhmv\nykN8zrm8qqYxMwOGAblAV+fcjtqXJiIimSyoc1BDgGOAc51z2wKap4iIZLAg7oNqB/QCOgIrzSw/\n8bqm1tWJiEjGCuIy8yWABVCLiIjIburqSEREIkkBJSIikRTofVA1KsBsDbAk1CLK1xpYG3YRKUbr\nrHq0vqpH66v6orrO2jnnDqhqotADKqrMbHoyN5LJHlpn1aP1VT1aX9WX6utMh/hERCSSFFAiIhJJ\nCqiKDQ27gBSkdVY9Wl/Vo/VVfSm9znQOSkREIkl7UCIiEkkKKBERiSQFlIiIRJICKklmdqSZFZjZ\nyLBriSoz29fMhpnZEjPbbGYzzaxL2HVFjZntb2avm9mWxLr6Rdg1RZW2qdpJ9e8tBVTyngb+HXYR\nEZcDLAXOApoDfYHRZtY+xJqi6Gn8gz1zgWuAIWb2g3BLiixtU7WT0t9bCqgkmNnVQBx4L+xaosw5\nt8U594BzbrFzrtg5Nw5YBHQKu7aoMLMmwGVAP+dcvnPuQ+BN4LpwK4smbVM1lw7fWwqoKpjZfsCf\ngDvCriXVmFkucBQwN+xaIuQoYKdzbn6JcbMA7UElQdtUctLle0sBVbWHgGHOuWVhF5JKzKwB8CLw\nvHPuq7DriZCmwKZS4zYCzUKoJaVom6qWtPjeyuiAMrOYmbkKXh+aWUfgXOCJsGuNgqrWV4npsoAR\n+PMst4dWcDTlA/uVGrcfsDmEWlKGtqnkpdP3Vq2fqJvKnHN5lbWbWR+gPfCtmYH/6zfbzI51zp1U\n5wVGTFXrC8D8ihqGvwCgq3NuR13XlWLmAzlmdqRzbkFi3InokFWFtE1VWx5p8r2lro4qYWaN2fuv\n3bvw//C3OufWhFJUxJnZM0BH4FznXH7Y9USRmb0COOBm/Lp6GzjNOaeQKoe2qepJp++tjN6Dqopz\nbiuwddfvZpYPFKTaP3J9MbN2QC+gEFiZ+OsNoJdz7sXQCoue3sBzwGpgHf6LQ+FUDm1T1ZdO31va\ngxIRkUjK6IskREQkuhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgk/T/X\nSE/diHEg1QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Q_RlKxRRwII",
        "colab_type": "text"
      },
      "source": [
        "Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDNKuv_cRwIJ",
        "colab_type": "code",
        "outputId": "78485c13-abce-45d0-9605-75e9421658fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "keras.layers.Dense(10, activation=\"elu\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.core.Dense at 0x7f6eaafa1e80>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k_CLlxLHRwIR",
        "colab_type": "text"
      },
      "source": [
        "### SELU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDZlAeELRwIS",
        "colab_type": "text"
      },
      "source": [
        "This activation function was proposed in this [great paper](https://arxiv.org/pdf/1706.02515.pdf) by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017. During training, a neural network composed exclusively of a stack of dense layers using the SELU activation function and LeCun initialization will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out. Unfortunately, the self-normalizing property of the SELU activation function is easily broken: you cannot use ℓ<sub>1</sub> or ℓ<sub>2</sub> regularization, regular dropout, max-norm, skip connections or other non-sequential topologies (so recurrent neural networks won't self-normalize). However, in practice it works quite well with sequential CNNs. If you break self-normalization, SELU will not necessarily outperform other activation functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SLWxayPtRwIT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.special import erfc\n",
        "\n",
        "# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
        "# (see equation 14 in the paper):\n",
        "alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\n",
        "scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ulgT1PsGRwIW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
        "    return scale * elu(z, alpha)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0htg9GzaRwIh",
        "colab_type": "code",
        "outputId": "7e6cb086-e784-401d-dd64-773679137ab4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        }
      },
      "source": [
        "plt.plot(z, selu(z), \"b-\", linewidth=2)\n",
        "plt.plot([-5, 5], [0, 0], 'k-')\n",
        "plt.plot([-5, 5], [-1.758, -1.758], 'k--')\n",
        "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
        "plt.grid(True)\n",
        "plt.title(\"SELU activation function\", fontsize=14)\n",
        "plt.axis([-5, 5, -2.2, 3.2])\n",
        "\n",
        "save_fig(\"selu_plot\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving figure selu_plot\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8FPW5x/HPEwIIiEZBchTUeMN7\nRYhtxfaYVqyKl9qD1VpAsVoQqxaRVkUUDnCwtVTRFrAIgoJWqeIN0VZpo7WIFSTeRcWCICoXXTHh\nEhJ+54/fxixLLptkNjO7+b5fr3llmZnMPDtM9rsz++yMOecQERGJmpywCxAREamJAkpERCJJASUi\nIpGkgBIRkUhSQImISCQpoEREJJIUUCJ1MLOVZjaiGdYzxszebIb15JjZn8xso5k5MytK9zrrqWeW\nmc0PswaJLgWUpMTM9jGzKfEX7G1m9pmZLTSzUxPmKY6/6CUPDybM48zsvFrWMcjMSmuZVuvvBaGO\ngDgBmBLgegriz6UwadJE4OSg1lOHvsAlwNnAvsCiZlgnZlYUf96dkyb9EhjQHDVI5skNuwDJGI8A\n7YFLgQ+ALvgX1E5J880ERiaN25L26tLEObe+mdZTCtQYzgE7FPjEOdcswVQf59yXYdcg0aUjKKmX\nmeUB3wWud84tdM6tcs694pyb6Jx7MGn2zc65T5OGtL8ImdnpZvZPM/vCzD43s7+a2ZFJ8+xnZvfH\nT29tNrMSM/uemQ0CRgNHJxz1DYr/zten+MzsATN7JGmZOWa22syGp1jHf+I/X4mvpzj+ezsdwcWX\ne1N82dvM7A0z+2HC9KojsX5m9mz8+bydeERbwzaaBdwOHBD/3ZXx8cVm9sfkeRNPvcXnmWJmE8xs\ng5mtM7OJZpaTME+b+PRV8Zo/NLOrzawA+Ed8tvXxdc+qZT1tzWxS/Ah9q5ktNrPvJEyvOhI7xcxe\njj/vJWbWs7bnLZlLASWpqHp3f46Z7RZ2MbXoAEwCvgkUAV8CT5pZGwAz6wA8DxQA5wLHAmPjv/sQ\n8HtgOf60177xccnmAGea2Z4J406Oz//nVOqIjwc4Pf57/1PL8/kl8CvgunitjwLzzKxH0nz/B9wJ\nHAe8AjxoZrvXscyxwJr4uk+oZb7a9AcqgN7AlcAw4IKE6fcCFwHDgSPxR9sxYDXQLz7P0fF1/7KW\nddwaX+bPgOOBN4BnzGzfpPluAa4HegIbgfvNzBr4fCTqnHMaNNQ74F9gPge2Ai/hPzP5VtI8xUA5\n1YFWNVyRMI8DzqtlHYOA0lqm1fp7tczfAagEvhP/98+Br4DOtcw/BnizhvErgRHxx7nAZ8ClCdOn\nA39rQB0F8edSWNf6gY+Bm2vYvnOSljMkYXrX+Ljv1FHPCGBlDcv9Y9K4WcD8pHleSprnWWB6/PFh\n8XWfXst6i+LTO9e2nvi2KgcuSpjeClgBjE9azmkJ85wUH9ct7L8TDcEOOoKSlDjnHgH2w3+4/jT+\nXfRiM0v+vOkhoEfScH+66zOzQ+Kn4FaY2SZ8kOQAB8RnOR543Tm3obHrcM5V4J9f//g62+KDe04D\n6kjlueyB39b/Spr0InBU0rjXEx6vjf/skuq6Guj1pH+vTVjX8cAOqk/lNcYhQGsSnrdzrhL/hijM\n5y0hUZOEpMw5txX/rvlZYKyZTQfGmNlE51x5fLYvnXMfNHIVm4B2ZtbaObe9amT8MzDwp8tqMx9/\n6moI/uijAngbaFPH7zTGHOAlM+sKfCu+/HnNWEfy7Qe+3k7OORc/y9XQN547gOTTY61rmG970r9d\nI9bVWLU+74RpesOdZfQfKk3xNv5NTlCfSy3H75PHJ43vmTB9F2bWCTgCmOCce8459w7QkZ3fgC0D\nvlFDm3OVcvzppDo55/6N72K8EH8k9bjzHXip1lEV5LWuyzm3CX9UcFLSpO/gt3nQ1uM/F0p0XAOX\nUYL/v/teLdPrfd74U3nlJDxvM2sFnEh6nrdEnI6gpF7xF96/APfgT618BRQCvwYWxl9Qq7Q3s/9K\nWkS5c+7zhH8X1PBh/4fOubfM7G/A9HhX3AqgO3AHMNc591EtJX4BbAB+bmar8Z/F/A5/9FLlAfyH\n6o+b2fX4o5tjgK+cc//Af9Z0YLwb7KP4+G21rO9+4DL850CJTQ6p1LEO33Z/WryLbqurucvxd/ij\n1PeBpfjvCn2X6rAO0t+BSWZ2Dv5NwBBgf/w2SYlz7j0zm4v/v/sl8CrQDShwzs0GVuGPdM40syeB\nLVXBnrCMMjObCvzWzDbgOx6vAfIJ8LtokkHC/hBMQ/QHoC0wAd8l9gWwGXgfuA3YO2G+YvyLUPLw\nYsI8NU13wFnx6Xn4QPogvp73gN8Cu9dT4/eBN/FNHG8Cp+EbNAYlzNMN/xlSLL7sZUBRwnN8OP78\nXNXvkdAkkbCcg+PzfAbkNqKOy/AhWAkUx8eNYecmiRzgJnwHXDm+m+3chOkF1NxsUWczCTU3SbQG\nJuPDdQPwv9TcJFFfI0VbfBfex8A2/BuMKxOm3wR8gj+lOKuOZUyKb9ttwGISmj6oodmitm2hIfMH\ni/8Hi4iIRIo+gxIRkUhSQImISCQpoEREJJIUUCIiEkmht5l37tzZFRQUhF3GLsrKyujQoUPYZWQU\nbbPULV++nMrKSo46KvkCCVKbqO5f5eXwzjtQUQGdOkGUXs6ius2WLl26wTm3T33zhR5QBQUFLFmy\nJOwydlFcXExRUVHYZWQUbbPUFRUVEYvFIrnvR1UU969Nm+Ckk3w4fe978Mwz0Cboa5c0QRS3GYCZ\nrUplPp3iExFphIoKuOACePNNOOIIeOSRaIVTNlBAiYg0kHNw9dX+iKlzZ3jqKdhrr7Cryj4KKBGR\nBpo0CaZOhbZt4fHH4eCDw64oOymgREQa4LHH4Npr/eN774XevcOtJ5sFGlBmNsfMPjGzTWb2npld\nFuTyRUTCtHQp9O/vT/GNH+8/g5L0CfoI6hb81Yv3AM4BxptZr4DXISLS7FavhrPPhs2b4eKLYWTy\nrTolcIEGlHPuLVd9i4Kqq1QfEuQ6RESa26ZNcOaZ8MknUFQE06aBJd/iUQIX+PegzGwKMAhoh7+d\nwYIa5hkMDAbIz8+nuLg46DKarLS0NJJ1RZm2WepisRiVlZXaXg0Q1v5VWWmMHHkMb7zRif3338zw\n4a+yaFFF/b8YAZn+N5mW220k3AWzCPitS7h9d7LCwkIXxS8rRvULblGmbZa6qi/qlpSUhF1Kxghj\n/3IOrrwSpkzx7eSLF8MhGXROKKp/k2a21DlXWN98aenic85VOudexN8gbmg61iEikm533OHDqU0b\n372XSeGUDdLdZp6LPoMSkQz0+OMwfLh/PGuWv6SRNK/AAsrMupjZT8xsdzNrZWanARcCC4Nah4hI\nc1i6FH76U3+Kb9w4uPDCsCtqmYJsknD403l34YNvFTDMOfdEgOsQEUmr5HbyG28Mu6KWK7CAcs6t\nB04OankiIs3tq6/grLN8O/nJJ6udPGy61JGICP7q5D/5Cbz+OnTvDvPm6erkYVNAiUiL5xwMGwYL\nFvibDi5YAHvvHXZVooASkRbvzjth8mS1k0eNAkpEWrQnn4RrrvGPZ86E73wn3HqkmgJKRFqsV1/1\nnzs5B2PH+tZyiQ4FlIi0SGvWVLeTX3QRjBoVdkWSTAElIi1OVTv52rXw3/+tdvKoUkCJSItSUeGv\nDPHaa3DYYb6dvG3bsKuSmiigRKRFGT4cnnqqup28U6ewK5LaKKBEpMW48074wx+q28kPPTTsiqQu\nCigRaRHmz69uJ7/nHrWTZwIFlIhkvWXLfDv5jh0wZgz07x92RZIKBZSIZLU1a3zHXlkZDBgAN98c\ndkWSKgWUiGSt0lL/XaeqdvLp09VOnkkUUCKSlSorfTt5SYnayTOVAkpEstLw4b4xYu+9q9vKJbMo\noEQk6/zhD76lvKqd/LDDwq5IGkMBJSJZ5amn/L2dAGbMgO9+N9x6pPEUUCKSNUpK4IILfDv56NG+\na08ylwJKRLLCxx9Xt5P37+8DSjKbAkpEMl5VO/nHH/srRMyYoXbybKCAEpGMVlnpbzS4bJm/tt5j\nj6mdPFsooEQko117rb9tu9rJs48CSkQy1uTJcMcd0Lo1PPoodO8edkUSJAWUiGSkBQvg6qv94xkz\n/KWMJLsooEQk47z2WnU7+c03w8CBYVck6aCAEpGMsnatbycvLfXNEWPGhF2RpIsCSkQyRlmZbydf\nswZOOknt5NlOASUiGaGqnfzVV+GQQ3w7+W67hV2VpJMCSkQywl13HcITT8Bee/kGic6dw65I0k0B\nJSKRN2UKPPzw/monb2EUUCISaU8/DVdd5R9Pnw4nnxxuPdJ8AgsoM2trZjPMbJWZfWVmJWZ2RlDL\nF5GW57XX4PzzfTv5wIErueiisCuS5hTkEVQusBo4GdgTGAXMNbOCANchIi1EYjv5hRfCJZesDLsk\naWaBBZRzrsw5N8Y5t9I5t8M5Nx/4D9ArqHWISMuQ3E5+zz1qJ2+JctO1YDPLB7oDb9UwbTAwGCA/\nP5/i4uJ0ldFopaWlkawryrTNUheLxaisrNT2qkFlJYwefQyvvtqZ/fbbwogRr7J48XbtX42Q6dvM\nnHPBL9SsNfA0sMI5N6SueQsLC92SJUsCr6GpiouLKSoqCruMjKJtlrqioiJisRglJSVhlxI5114L\nt93m28lfegkOP9yP1/7VcFHdZma21DlXWN98gXfxmVkOMBsoB64Mevkikr2mTvXh1Lo1zJtXHU7S\nMgV6is/MDJgB5AN9nXPbg1y+iGSvZ56pbie/+26I4Bt/aWZBfwY1FTgS6OOc2xLwskUkS73xhm8n\nr6yEG2+Eiy8OuyKJgiC/B3UgMAToAXxqZqXxoX9Q6xCR7PPJJ3DmmfDVV/4WGmPHhl2RREVgR1DO\nuVWAGkFFJGVV7eSrV0Pv3jBrFuTo+jYSp11BREJRWQkDBsDSpXDwwbo6uexKASUiobjuOh9KeXnw\n1FOwzz5hVyRRo4ASkWZ3113w+99Dbi488ggccUTYFUkUKaBEpFn99a9wZfwbktOmwfe/H249El0K\nKBFpNm+8AT/+sf/8aeRIuOSSsCuSKFNAiUiz+PRTf3XyqnbycePCrkiiTgElImm3eTOccw589BF8\n+9swc6bayaV+2kVEJK127PDt5K+8AgcdBI8/Du3ahV2VZAIFlIik1XXXwaOPwp57+nbyLl3Crkgy\nhQJKRNJm2jSYONG3k8+bB0ceGXZFkkkUUCKSFn/7G1xxhX/8pz+pnVwaTgElIoF7883qdvIbboCf\n/SzsiiQTKaBEJFCffuqvTr5pkw+p8ePDrkgylQJKRAKT3E5+771qJ5fG064jIoHYsQMuusi3kxcU\nqJ1cmk4BJSKBuOEGf+FXtZNLUBRQItJkd98Nt95afXXyo44KuyLJBgooEWmSZ5+FoUP947vuglNO\nCbceyR4KKBFptLfegvPO8+3k110Hl14adkWSTRRQItIon31W3U5+3nkwYULYFUm2UUCJSINVtZOv\nWgXf+hbcd5/aySV42qVEpEGq2sn//W+1k0t6KaBEpEFGjvSdenvs4dvJ8/PDrkiylQJKRFI2fTr8\n9rdqJ5fmoYASkZQ89xxcfrl/PHUq9OkTbj2S/RRQIlKvt9+ubif/9a/hssvCrkhaAgWUiNSpqp38\nyy+hXz+45ZawK5KWQgElIrXasgV++ENYuRK++U21k0vz0q4mIjWqaid/+WU48EB44glo3z7sqqQl\nUUCJSI1uvBEefljt5BIeBZSI7OKee+A3v4FWrXxIHX102BVJS6SAEpGdLFwIQ4b4x1OnwqmnhluP\ntFwKKBH52ttv+069igr41a/g5z8PuyJpyQINKDO70syWmNk2M5sV5LJFJL3WratuJ/+f//Gn+ETC\nlBvw8tYC44HTAF0+UiRDJLaTn3ACzJ6tdnIJX6AB5ZybB2BmhUC3IJctIumxYwcMGgSLF8MBB6id\nXKIj6COolJjZYGAwQH5+PsXFxWGUUafS0tJI1hVl2mapi8ViVFZWRmJ73X33QcydeyAdOlQwZswy\n3n23jHffDbuqXWn/arhM32ahBJRzbhowDaCwsNAVFRWFUUadiouLiWJdUaZtlrq8vDxisVjo22vm\nTHjgAd9OPm9eLj/4wQmh1lMX7V8Nl+nbTGeZRVqov/8dBg/2jydPhh/8INx6RJIpoERaoHfeqW4n\nHzGi+ntPIlES6Ck+M8uNL7MV0MrMdgMqnHMVQa5HRBqvqp08FoMf/cjfgFAkioI+ghoFbAGuBwbE\nH48KeB0i0khbt8K558J//gOFhTBnjtrJJbqCbjMfA4wJcpkiEoyqdvKXXoL991c7uUSf3juJtBA3\n3wwPPQQdO/qrk++7b9gVidRNASXSAsycCf/3f76d/C9/gWOPDbsikfopoESy3D/+Ud1O/sc/wmmn\nhVuPSKoUUCJZ7N13/YVfKypg+HC4/PKwKxJJnQJKJEutX1/dTn7uuXDrrWFXJNIwCiiRLFTVTv7h\nh9Crl28nb9Uq7KpEGkYBJZJlduyASy6BRYt8O/mTT0KHDmFXJdJwCiiRLDN6NDz4oG8nnz9f7eSS\nuRRQIlnk3nth/Hh/Om/uXPjGN8KuSKTxFFAiWaK4GH7+c//4D3+A008PtRyRJlNAiWSB5ct9O/n2\n7XDNNTB0aNgViTSdAkokw23Y4NvJv/gCzjkHfve7sCsSCYYCSiSDVbWTr1gBPXtW3x1XJBsooEQy\nlHPws5/Bv/4F3bqpnVyyjwJKJEONHg1//jPsvru/Ovl++4VdkUiwFFAiGei++2DcOH+zwYceUju5\nZCcFlEiGef55uOwy//jOO6Fv33DrEUkXBZRIBlm+HH70I99OPmwY/OIXYVckkj4KKJEMkdxOPnFi\n2BWJpJcCSiQDbNvmj5xWrIDjj4f771c7uWQ/BZRIxFW1k7/4InTt6tvJd9897KpE0k8BJRJxY8b4\nL+BWtZN37Rp2RSLNQwElEmGzZ8PYsdXt5McdF3ZFIs1HASUSUS+8AJde6h/fcYfayaXlUUCJRND7\n71e3k199NVx5ZdgViTQ/BZRIxGzc6I+WPv8czj4bbrst7IpEwqGAEomQbdv81ck/+MC3k+vq5NKS\nKaBEIsI5fwkjtZOLeAookYgYOxbmzPG3zJg/X+3kIgookQiYM8d/3yknBx58EHr0CLsikfApoERC\n9s9/VreTT5oEZ50Vbj0iUaGAEgnR++/7pojycrjqKj+IiBdoQJnZ3mb2qJmVmdkqM/tpkMsXySYV\nFcaZZ/p28jPPhNtvD7sikWjJDXh5k4FyIB/oATxlZq85594KeD0iGc05WLmyA2Vl/vOmBx9UO7lI\nMnPOBbMgsw7AF8Axzrn34uNmAx87566v7fc6duzoevXqFUgNQYrFYuTl5YVdRkbRNkvdyy+XsHUr\ntGnTg549oW3bsCuKPu1fDRfVbfb8888vdc4V1jdfkEdQ3YGKqnCKew04OXlGMxsMDAZo3bo1sVgs\nwDKCUVlZGcm6okzbLDXbtuWwdat/3LVrGVu2bGfLlnBrygTavxou07dZkAG1O7ApadyXQMfkGZ1z\n04BpAIWFhW7JkiUBlhGM4uJiioqKwi4jo2ib1c856NMH3n23iLy8cj78cFHYJWUM7V8NF9VtZmYp\nzRdkk0QpsEfSuD2ArwJch0hGmz8f/v53yM2Frl112CRSlyAD6j0g18wOSxh3HKAGCRGgshKuj38a\ne+CBkJsbzOe/ItkqsIByzpUB84CxZtbBzE4CfgjMDmodIpnsvvvg7behoAD22y/sakSiL+gv6l4B\ntAPWAX8GhqrFXATKyuDmm/3j8eP9JY1EpG6B/pk45z53zp3rnOvgnDvAOfdAkMsXyVS33gpr1kDP\nnnDhhWFXI5IZ9D5OJM1WrfIBBXDnnTp6EkmV/lRE0uxXv4KtW/2R00knhV2NSOZQQImkUXEx/OUv\n0L599VGUiKRGASWSJpWV8Mtf+sfXXw/duoVbj0imUUCJpMndd8Prr/vvPI0YEXY1IplHASWSBp9+\nCjfc4B9PnAjt2oVbj0gmUkCJpMFVV0EsBmecAf36hV2NSGZSQIkE7LHH4OGHoUMHmDoVUrwupogk\nUUCJBOjLL+EXv/CPJ0zwnz+JSOMooEQCdP31sHYtfPvb1UElIo2jgBIJyAsvwF13QevWMH26buEu\n0lQKKJEAbNoEF1/sH99wAxx9dLj1iGQDBZRIAK66Clau9BeDvfHGsKsRyQ4KKJEmeughf6+ndu3g\n/vuhTZuwKxLJDgookSb46CO4/HL/+Lbb4Igjwq1HJJsooEQaqbISLrrIfyH37LNhyJCwKxLJLgoo\nkUaaMAGefx7y82HGDH0hVyRoCiiRRnj6aRg92ofSfffBPvuEXZFI9skNuwCRTPPhh9C/PzgH48bB\nD34QdkUi2UlHUCINsHmzv/jrF1/4z51Gjgy7IpHspYASSZFzMHQolJTAoYf6U3s5+gsSSRv9eYmk\n6PbbfSi1bw/z5kFeXtgViWQ3BZRICh55pPquuDNnwrHHhluPSEuggBKpx+LFMGCAP8X3m9/A+eeH\nXZFIy6CAEqnDihVwzjmwdSsMHgy//nXYFYm0HAookVqsWwd9+8L69XD66TB5sr6MK9KcFFAiNfj8\nc//9pvfeg+OO8xeEzdW3BkWalQJKJMmmTXDGGfDaa9C9O/z1r7DHHmFXJdLyKKBEEpSVwVlnwb//\nDQcdBAsX+mvtiUjzU0CJxJWVwQ9/CP/8J3Tt6sOpW7ewqxJpuXRWXQR/y4yzzoJ//Qu6dPHhdNBB\nYVcl0rLpCEpavPXr4Xvf8+G0//7+COrww8OuSkR0BCUt2po1cOqp8O67/vp6CxfCAQeEXZWIQEBH\nUGZ2pZktMbNtZjYriGWKpNvrr0Pv3j6cjj3WHzkpnESiI6hTfGuB8cA9AS1PJK0WLICTToLVq31I\nFRfDf/1X2FWJSKJAAso5N8859xiwMYjliaTT5Mn+Xk6lpXDhhf603t57h12ViCQL5TMoMxsMDAbI\nz8+nuLg4jDLqVFpaGsm6oizq26y83Jg8+VCeeKIrABddtJJBg1ayeHHz1xKLxaisrIz09oqaqO9f\nUZTp2yyUgHLOTQOmARQWFrqioqIwyqhTcXExUawryqK8zVatgh//GF55Bdq0genTYeDAAqAglHry\n8vKIxWKR3V5RFOX9K6oyfZvVe4rPzIrNzNUyvNgcRYo0xTPPQM+ePpwOPNC3kw8cGHZVIlKfeo+g\nnHNFzVCHSODKy+Hmm+HWW/29nM44A2bPhk6dwq5MRFIRyCk+M8uNL6sV0MrMdgMqnHMVQSxfpKHe\negv69/cXfM3Jgf/9X7jxRv9YRDJDUH+uo4AtwPXAgPjjUQEtWyRlO3bApEnQq5cPp4MP9t9vuukm\nhZNIpgnkCMo5NwYYE8SyRBrrzTdhyBBYtMj/+9JL4fbboWPHcOsSkcbRe0rJeFu3wqhRcPzxPpz2\n3Rcee8x36imcRDKXrsUnGcs5mD8fhg+HDz7w4y6/HG65BfLywq1NRJpOASUZ6Y034Jpr/FUgAI46\nCqZN85cvEpHsoFN8klHWrIHBg6FHDx9Oe+0Fd9wBJSUKJ5FsoyMoyQiffeZP3d11F2zbBq1awVVX\nwejR+l6TSLZSQEmkffKJ78SbPBk2b/bjzj/ff6/piCPCrU1E0ksBJZH0/vvwu9/Bvff6K0KAvwL5\nuHFw3HHh1iYizUMBJZGxYwc8+yxMmQJPPum79MygXz+47jo44YSwKxSR5qSAktBt3AgzZ/rPl1as\n8ONat4aLL4YRI+Dww8OtT0TCoYCSUFRWwvPPw6xZMHeub3wAf8v1IUP8VSDy80MtUURCpoCSZuMc\nLFsG998PDz4Ia9f68Wb+SuNXXOF/tmoVbp0iEg0KKEkr5/yXah97DB54AJYvr5528MHw05/CJZf4\nxyIiiRRQErjycn/67okn/PDRR9XT9tkHLrjA3wrjW9/yR08iIjVRQEmTOeevhffEE/vxxz/6TrxN\nm6qnd+niW8T79YM+fXwDhIhIfRRQ0iirV8MLL8Bzz/lLDq1eDdD96+nHHAPnnOOD6Zvf1L2YRKTh\nFFBSr/Jyf627RYv88NJL/pp4iTp1gmOOWceFF3bh1FP1mZKINJ0CSnaydau/8d+rr/ph2TJ/Z9qq\nNvAqeXlw4olwyil++MY34IUX3qaoqEs4hYtI1lFAtVDl5f5yQu+8A2+/XT288w5UVOw6/xFHQO/e\nfjjxRP9vnbYTkXRSQGWxigr/2dCHH1YPy5f7IPrgA/9l2WQ5OXDkkdCzZ/XQo4duACgizU8BlaGc\n851yH3/sPw/6+GM/JAbSqlU1hxD49u5DDvFhdNRR/ueRR/rmhg4dmve5iIjURAEVMVu2wPr1sG6d\n/5k4rF27cyCVldW/vK5dfcNC1XDooT6QDj8c2rVL//MREWksBVTAnPMNBV9+CbFYasPGjdUhlEro\nVGnfHrp18yFUNXTrVh1GBQWw225pe6oiImnVYgJqxw4fHFu3+iHxcU3jli3LZ/lyf0RTWuqDI/Fn\nbY/LympuMkhVmzb+agtVQ5cu1Y/33XfnMNpzT12JQUSyV+gB9ckncNNN/kV9+/bqn4mPGztu27bq\n0Km66V3qjmz0c8rN9U0FNQ177VXzuKow6thRoSMiAhEIqLVrlzN+fFHS2POBK4DNQN8afmtQfNgA\nnFfD9KHABcBqYODXY818l1rHjtey555nk5OznHXrhpCTw07D0UePom3bY+jY8VNefnkYrVr5K2zn\n5Pif/ftPoGfP3qxcuYiZM0d+Pb1quOOOSfTo0YPnnnuO8ePHs3179Sk8gD/96U8cfvjhPPnkk9x6\n6+93qX727Nnsv//+PPTQQ0ydOnWX6Q8//DCdO3dm1qxZzJo1a5fpCxYsoH379kyZMoW5c+fuMr24\nuBiAiRMnMn/+/J2mtWvXjqeffhqAcePGsXDhwp2md+rUiUceeQSAG264gZdeeunrabFYjGOOOYY5\nc+YAMGzYMEpKSnb6/e7duzNt2jQABg8ezHvvvbfT9B49ejBp0iQABgwYwJqkbwSfeOKJ3HLLLQD0\n69ePjRs37jT9lFNO4aabbgICTsdMAAAFTElEQVTgjDPOYMuWLTtNP+ussxgxYgQARUVFu2yb888/\nnyuuuILNmzfTt++u+96gQYMYNGgQGzZs4Lzzdt33hg4dygUXXMDq1asZOHDgLtOvvfZazj77bDZv\n3swHH3ywSw2jRo2iT58+lJSUMGzYsF1+f8KECfTu3ZtFixYxcuTIXaZPmrTzvpcscd/7/e8za9/b\nsWMHL7zwArDrvgfQrVs37XtJ+14sFiMv3oJbte8tX76cIUOG7PL7zbnvpSr0gGrTBvbbz4dH1dCr\nF3z/+/603J137jzNDE47Dfr29afTRo/eeVpODgwYAOeeCxs2wNVX+3GJRyXXXusvwbN8ub/3ULJR\noyA3913y8vKo4f+JPn3894EWLYKHH07fthERacnMORdqAYWFhW7JkiWh1lCT4uLiGt/lSO20zVJX\nVFRELBbb5V2+1E77V8NFdZuZ2VLnXGF98+laACIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikdTk\ngDKztmY2w8xWmdlXZlZiZmcEUZyIiLRcQRxB5eK/EXsysCcwCphrZgUBLFtERFqoJn9R1zlXBoxJ\nGDXfzP4D9AJWNnX5IiLSMgV+JQkzywe6A2/VMc9gYDBAfn7+15c/iZLS0tJI1hVl2mapi8ViVFZW\nans1gPavhsv0bRbolSTMrDXwNLDCOVfDRYR2pStJZA9ts9TpShINp/2r4aK6zQK7koSZFZuZq2V4\nMWG+HGA2UA5c2aTqRUSkxav3FJ9zrqi+eczMgBlAPtDXObe96aWJiEhLFtRnUFPxN1Dq45zbUt/M\nIiIi9Qnie1AHAkOAHsCnZlYaH/o3uToREWmxgmgzXwXoHrAiIhIoXepIREQiSQElIiKRFPoddc1s\nPbAq1CJq1hnYEHYRGUbbrGG0vRpG26vhorrNDnTO7VPfTKEHVFSZ2ZJUvkgm1bTNGkbbq2G0vRou\n07eZTvGJiEgkKaBERCSSFFC1mxZ2ARlI26xhtL0aRtur4TJ6m+kzKBERiSQdQYmISCQpoEREJJIU\nUCIiEkkKqBSZ2WFmttXM5oRdS1SZWVszm2Fmq8zsKzMrMbMzwq4rasxsbzN71MzK4tvqp2HXFFXa\np5om01+3FFCpmwy8EnYREZcLrAZOBvYERgFzzawgxJqiaDL+xp75QH9gqpkdHW5JkaV9qmky+nVL\nAZUCM/sJEAMWhl1LlDnnypxzY5xzK51zO5xz84H/AL3Cri0qzKwD0A+4yTlX6px7EXgCGBhuZdGk\nfarxsuF1SwFVDzPbAxgLDA+7lkxjZvlAd+CtsGuJkO5AhXPuvYRxrwE6gkqB9qnUZMvrlgKqfuOA\nGc65NWEXkknMrDVwP3Cvc+7dsOuJkN2BTUnjvgQ6hlBLRtE+1SBZ8brVogPKzIrNzNUyvGhmPYA+\nwO1h1xoF9W2vhPlygNn4z1muDK3gaCoF9kgatwfwVQi1ZAztU6nLptetJt9RN5M554rqmm5mw4AC\n4CMzA//ut5WZHeWc65n2AiOmvu0FYH5DzcA3APR1zm1Pd10Z5j0g18wOc869Hx93HDplVSvtUw1W\nRJa8bulSR3Uws/bs/G53BP4/fqhzbn0oRUWcmd0F9AD6OOdKw64niszsQcABl+G31QKgt3NOIVUD\n7VMNk02vWy36CKo+zrnNwOaqf5tZKbA10/6Tm4uZHQgMAbYBn8bfvQEMcc7dH1ph0XMFcA+wDtiI\nf+FQONVA+1TDZdPrlo6gREQkklp0k4SIiESXAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIU\nUCIiEkkKKBERiaT/B9c9MCs0b4SXAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QlmoxejJRwIm",
        "colab_type": "text"
      },
      "source": [
        "By default, the SELU hyperparameters (`scale` and `alpha`) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kuuisLaYRwIp",
        "colab_type": "code",
        "outputId": "2b60fe06-0fd9-4356-f466-2fbc46a70f09",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        }
      },
      "source": [
        "np.random.seed(42)\n",
        "Z = np.random.normal(size=(500, 100)) # standardized inputs\n",
        "for layer in range(1000):\n",
        "    W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n",
        "    Z = selu(np.dot(Z, W))\n",
        "    means = np.mean(Z, axis=0).mean()\n",
        "    stds = np.std(Z, axis=0).mean()\n",
        "    if layer % 100 == 0:\n",
        "        print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Layer 0: mean -0.00, std deviation 1.00\n",
            "Layer 100: mean 0.02, std deviation 0.96\n",
            "Layer 200: mean 0.01, std deviation 0.90\n",
            "Layer 300: mean -0.02, std deviation 0.92\n",
            "Layer 400: mean 0.05, std deviation 0.89\n",
            "Layer 500: mean 0.01, std deviation 0.93\n",
            "Layer 600: mean 0.02, std deviation 0.92\n",
            "Layer 700: mean -0.02, std deviation 0.90\n",
            "Layer 800: mean 0.05, std deviation 0.83\n",
            "Layer 900: mean 0.02, std deviation 1.00\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWAgCMLPRwIu",
        "colab_type": "text"
      },
      "source": [
        "Using SELU is easy:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jqtQKnxyRwIv",
        "colab_type": "code",
        "outputId": "9690158f-74ff-4bab-dc9f-1104225adae0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "keras.layers.Dense(10, activation=\"selu\",\n",
        "                   kernel_initializer=\"lecun_normal\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.core.Dense at 0x7f6e9db63a20>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eKkTkd1aRwI2",
        "colab_type": "text"
      },
      "source": [
        "Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "34sKPe5dRwI3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DB6MePyCRwI5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
        "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
        "                             kernel_initializer=\"lecun_normal\"))\n",
        "for layer in range(99):\n",
        "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
        "                                 kernel_initializer=\"lecun_normal\"))\n",
        "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z6oTF9WvRwI8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FRyrUwGVRwI_",
        "colab_type": "text"
      },
      "source": [
        "Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tUTm_aQdRwJA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
        "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
        "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
        "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
        "X_test_scaled = (X_test - pixel_means) / pixel_stds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XBIMEKgPRwJD",
        "colab_type": "code",
        "outputId": "c6ba1b1f-a3c7-4cec-99a2-75a202ff4f81",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/5\n",
            "55000/55000 [==============================] - 81s 1ms/sample - loss: 1.4818 - accuracy: 0.3958 - val_loss: 1.5240 - val_accuracy: 0.2844\n",
            "Epoch 2/5\n",
            "55000/55000 [==============================] - 78s 1ms/sample - loss: 1.5991 - accuracy: 0.3461 - val_loss: 1.7466 - val_accuracy: 0.2812\n",
            "Epoch 3/5\n",
            "55000/55000 [==============================] - 82s 1ms/sample - loss: 1.1952 - accuracy: 0.5124 - val_loss: 1.0862 - val_accuracy: 0.5702\n",
            "Epoch 4/5\n",
            "55000/55000 [==============================] - 78s 1ms/sample - loss: 1.0782 - accuracy: 0.5695 - val_loss: 0.8320 - val_accuracy: 0.7062\n",
            "Epoch 5/5\n",
            "55000/55000 [==============================] - 78s 1ms/sample - loss: 0.7530 - accuracy: 0.7193 - val_loss: 0.7538 - val_accuracy: 0.7122\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dSIxTcz4RwJI",
        "colab_type": "text"
      },
      "source": [
        "Now look at what happens if we try to use the ReLU activation function instead:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b77nY1FmRwJJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gT1593N3RwJN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
        "model.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
        "for layer in range(99):\n",
        "    model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
        "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HtrS5AmxRwJW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4ejPG2WXRwJZ",
        "colab_type": "code",
        "outputId": "26aed068-b5c1-45c4-a503-a203c4bef11e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/5\n",
            "55000/55000 [==============================] - 45s 817us/sample - loss: 1.7793 - accuracy: 0.2765 - val_loss: 1.6866 - val_accuracy: 0.3002\n",
            "Epoch 2/5\n",
            "55000/55000 [==============================] - 46s 830us/sample - loss: 1.1687 - accuracy: 0.5045 - val_loss: 1.7109 - val_accuracy: 0.4454\n",
            "Epoch 3/5\n",
            "55000/55000 [==============================] - 45s 814us/sample - loss: 1.0323 - accuracy: 0.5653 - val_loss: 1.1129 - val_accuracy: 0.5040\n",
            "Epoch 4/5\n",
            "55000/55000 [==============================] - 46s 830us/sample - loss: 0.9160 - accuracy: 0.6231 - val_loss: 0.9224 - val_accuracy: 0.6530\n",
            "Epoch 5/5\n",
            "55000/55000 [==============================] - 45s 824us/sample - loss: 0.7820 - accuracy: 0.6891 - val_loss: 0.7141 - val_accuracy: 0.7244\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tMACazgARwJe",
        "colab_type": "text"
      },
      "source": [
        "Not great at all, we suffered from the vanishing/exploding gradients problem."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lJ67w7O8RwJf",
        "colab_type": "text"
      },
      "source": [
        "# Batch Normalization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnkvVI9JRwJg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Dense(300, activation=\"relu\"),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Dense(100, activation=\"relu\"),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j1oF8oyVRwJk",
        "colab_type": "code",
        "outputId": "64d7235a-fe4b-44d4-8981-b95210319a20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 425
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_5\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "flatten_5 (Flatten)          (None, 784)               0         \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2 (Batc (None, 784)               3136      \n",
            "_________________________________________________________________\n",
            "dense_412 (Dense)            (None, 300)               235500    \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2_1 (Ba (None, 300)               1200      \n",
            "_________________________________________________________________\n",
            "dense_413 (Dense)            (None, 100)               30100     \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2_2 (Ba (None, 100)               400       \n",
            "_________________________________________________________________\n",
            "dense_414 (Dense)            (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 271,346\n",
            "Trainable params: 268,978\n",
            "Non-trainable params: 2,368\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IQadeb46RwJp",
        "colab_type": "code",
        "outputId": "057c9d88-bee2-458f-8717-7aa7048d99c4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 91
        }
      },
      "source": [
        "bn1 = model.layers[1]\n",
        "[(var.name, var.trainable) for var in bn1.variables]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('batch_normalization_v2/gamma:0', True),\n",
              " ('batch_normalization_v2/beta:0', True),\n",
              " ('batch_normalization_v2/moving_mean:0', False),\n",
              " ('batch_normalization_v2/moving_variance:0', False)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a2eC5rbFRwJs",
        "colab_type": "code",
        "outputId": "6a3cd575-25f6-4e64-e5f6-5daf6161a51f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 56
        }
      },
      "source": [
        "bn1.updates"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ListWrapper([<tf.Operation 'batch_normalization_v2/cond_2/Identity' type=Identity>, <tf.Operation 'batch_normalization_v2/cond_3/Identity' type=Identity>])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3JqC_oUfRwJv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oflbQG79RwJx",
        "colab_type": "code",
        "outputId": "2c6ee884-c01c-4e99-d9eb-d39cbd524e0d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        }
      },
      "source": [
        "history = model.fit(X_train, y_train, epochs=10,\n",
        "                    validation_data=(X_valid, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/10\n",
            "55000/55000 [==============================] - 8s 147us/sample - loss: 0.8756 - accuracy: 0.7140 - val_loss: 0.5515 - val_accuracy: 0.8216\n",
            "Epoch 2/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.5765 - accuracy: 0.8033 - val_loss: 0.4742 - val_accuracy: 0.8432\n",
            "Epoch 3/10\n",
            "55000/55000 [==============================] - 8s 138us/sample - loss: 0.5146 - accuracy: 0.8214 - val_loss: 0.4382 - val_accuracy: 0.8528\n",
            "Epoch 4/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4820 - accuracy: 0.8325 - val_loss: 0.4169 - val_accuracy: 0.8598\n",
            "Epoch 5/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4589 - accuracy: 0.8403 - val_loss: 0.4003 - val_accuracy: 0.8654\n",
            "Epoch 6/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4427 - accuracy: 0.8457 - val_loss: 0.3883 - val_accuracy: 0.8700\n",
            "Epoch 7/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4219 - accuracy: 0.8522 - val_loss: 0.3794 - val_accuracy: 0.8722\n",
            "Epoch 8/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4150 - accuracy: 0.8549 - val_loss: 0.3696 - val_accuracy: 0.8758\n",
            "Epoch 9/10\n",
            "55000/55000 [==============================] - 8s 137us/sample - loss: 0.4013 - accuracy: 0.8589 - val_loss: 0.3631 - val_accuracy: 0.8736\n",
            "Epoch 10/10\n",
            "55000/55000 [==============================] - 9s 164us/sample - loss: 0.3931 - accuracy: 0.8616 - val_loss: 0.3581 - val_accuracy: 0.8766\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZUWkq21LRwJ2",
        "colab_type": "text"
      },
      "source": [
        "Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R6WFZsyURwJ3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Dense(300, use_bias=False),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Activation(\"relu\"),\n",
        "    keras.layers.Dense(100, use_bias=False),\n",
        "    keras.layers.Activation(\"relu\"),\n",
        "    keras.layers.BatchNormalization(),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s9ZaL1RhRwJ6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ETo3GG8RwJ8",
        "colab_type": "code",
        "outputId": "b9b039c6-81b5-43bc-804b-25f19f8abfec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        }
      },
      "source": [
        "history = model.fit(X_train, y_train, epochs=10,\n",
        "                    validation_data=(X_valid, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/10\n",
            "55000/55000 [==============================] - 8s 146us/sample - loss: 0.8618 - accuracy: 0.7092 - val_loss: 0.5658 - val_accuracy: 0.8086\n",
            "Epoch 2/10\n",
            "55000/55000 [==============================] - 7s 135us/sample - loss: 0.5809 - accuracy: 0.8008 - val_loss: 0.4837 - val_accuracy: 0.8350\n",
            "Epoch 3/10\n",
            "55000/55000 [==============================] - 8s 137us/sample - loss: 0.5161 - accuracy: 0.8202 - val_loss: 0.4465 - val_accuracy: 0.8480\n",
            "Epoch 4/10\n",
            "55000/55000 [==============================] - 8s 137us/sample - loss: 0.4851 - accuracy: 0.8301 - val_loss: 0.4257 - val_accuracy: 0.8552\n",
            "Epoch 5/10\n",
            "55000/55000 [==============================] - 9s 164us/sample - loss: 0.4581 - accuracy: 0.8401 - val_loss: 0.4112 - val_accuracy: 0.8604\n",
            "Epoch 6/10\n",
            "55000/55000 [==============================] - 8s 142us/sample - loss: 0.4411 - accuracy: 0.8458 - val_loss: 0.3982 - val_accuracy: 0.8612\n",
            "Epoch 7/10\n",
            "55000/55000 [==============================] - 7s 136us/sample - loss: 0.4299 - accuracy: 0.8481 - val_loss: 0.3926 - val_accuracy: 0.8656\n",
            "Epoch 8/10\n",
            "55000/55000 [==============================] - 8s 138us/sample - loss: 0.4129 - accuracy: 0.8559 - val_loss: 0.3840 - val_accuracy: 0.8664\n",
            "Epoch 9/10\n",
            "55000/55000 [==============================] - 8s 137us/sample - loss: 0.4011 - accuracy: 0.8593 - val_loss: 0.3772 - val_accuracy: 0.8694\n",
            "Epoch 10/10\n",
            "55000/55000 [==============================] - 7s 135us/sample - loss: 0.3932 - accuracy: 0.8623 - val_loss: 0.3714 - val_accuracy: 0.8698\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6X83yAMRwKC",
        "colab_type": "text"
      },
      "source": [
        "## Gradient Clipping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pTAp4KlARwKD",
        "colab_type": "text"
      },
      "source": [
        "All Keras optimizers accept `clipnorm` or `clipvalue` arguments:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DdKkR3I2RwKE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(clipvalue=1.0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pVYNcshyRwKS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(clipnorm=1.0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIax6cP_RwKb",
        "colab_type": "text"
      },
      "source": [
        "## Reusing Pretrained Layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gqshRNrRwKe",
        "colab_type": "text"
      },
      "source": [
        "### Reusing a Keras model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91nPWTNoRwKe",
        "colab_type": "text"
      },
      "source": [
        "Let's split the fashion MNIST training set in two:\n",
        "* `X_train_A`: all images of all items except for sandals and shirts (classes 5 and 6).\n",
        "* `X_train_B`: a much smaller training set of just the first 200 images of sandals or shirts.\n",
        "\n",
        "The validation set and the test set are also split this way, but without restricting the number of images.\n",
        "\n",
        "We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B, since classes in set A (sneakers, ankle boots, coats, t-shirts, etc.) are somewhat similar to classes in set B (sandals and shirts). However, since we are using `Dense` layers, only patterns that occur at the same location can be reused (in contrast, convolutional layers will transfer much better, since learned patterns can be detected anywhere on the image, as we will see in the CNN chapter)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zfh8RjuFRwKg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def split_dataset(X, y):\n",
        "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
        "    y_A = y[~y_5_or_6]\n",
        "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
        "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
        "    return ((X[~y_5_or_6], y_A),\n",
        "            (X[y_5_or_6], y_B))\n",
        "\n",
        "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
        "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
        "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
        "X_train_B = X_train_B[:200]\n",
        "y_train_B = y_train_B[:200]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "voHVEDC1RwKk",
        "colab_type": "code",
        "outputId": "b45cd38d-7f6f-4b0d-b500-235ca79f73c4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "X_train_A.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(43986, 28, 28)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h_z0tJKJRwKp",
        "colab_type": "code",
        "outputId": "90698949-d946-4df9-dca4-cd809eb3c3b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "X_train_B.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(200, 28, 28)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKcNJtH7RwKw",
        "colab_type": "code",
        "outputId": "e70e466e-7ea0-4f21-ca44-7d3b097091bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "y_train_A[:30]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,\n",
              "       1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0UlMABE9RwKz",
        "colab_type": "code",
        "outputId": "39072ffc-30de-40bc-d5c1-f9b5ffd5c0f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "y_train_B[:30]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,\n",
              "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RSKCqr_9RwK9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SpH81mcpRwLE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_A = keras.models.Sequential()\n",
        "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
        "for n_hidden in (300, 100, 50, 50, 50):\n",
        "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
        "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yvp7iuQvRwLI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_A.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gNnT9sSSRwLL",
        "colab_type": "code",
        "outputId": "5bd3d9ec-6d78-4cda-8414-72351ceb06ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
        "                    validation_data=(X_valid_A, y_valid_A))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 43986 samples, validate on 4014 samples\n",
            "Epoch 1/20\n",
            "43986/43986 [==============================] - 5s 115us/sample - loss: 0.5887 - accuracy: 0.8123 - val_loss: 0.3749 - val_accuracy: 0.8734\n",
            "Epoch 2/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.3516 - accuracy: 0.8793 - val_loss: 0.3223 - val_accuracy: 0.8874\n",
            "Epoch 3/20\n",
            "43986/43986 [==============================] - 5s 105us/sample - loss: 0.3160 - accuracy: 0.8895 - val_loss: 0.3009 - val_accuracy: 0.8954\n",
            "Epoch 4/20\n",
            "43986/43986 [==============================] - 5s 104us/sample - loss: 0.2963 - accuracy: 0.8977 - val_loss: 0.2849 - val_accuracy: 0.9038\n",
            "Epoch 5/20\n",
            "43986/43986 [==============================] - 5s 105us/sample - loss: 0.2825 - accuracy: 0.9035 - val_loss: 0.2767 - val_accuracy: 0.9078\n",
            "Epoch 6/20\n",
            "43986/43986 [==============================] - 5s 104us/sample - loss: 0.2719 - accuracy: 0.9069 - val_loss: 0.2673 - val_accuracy: 0.9096\n",
            "Epoch 7/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2639 - accuracy: 0.9090 - val_loss: 0.2657 - val_accuracy: 0.9101\n",
            "Epoch 8/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2570 - accuracy: 0.9122 - val_loss: 0.2592 - val_accuracy: 0.9113\n",
            "Epoch 9/20\n",
            "43986/43986 [==============================] - 5s 104us/sample - loss: 0.2514 - accuracy: 0.9138 - val_loss: 0.2567 - val_accuracy: 0.9133\n",
            "Epoch 10/20\n",
            "43986/43986 [==============================] - 6s 130us/sample - loss: 0.2465 - accuracy: 0.9167 - val_loss: 0.2561 - val_accuracy: 0.9106\n",
            "Epoch 11/20\n",
            "43986/43986 [==============================] - 5s 124us/sample - loss: 0.2418 - accuracy: 0.9178 - val_loss: 0.2485 - val_accuracy: 0.9180\n",
            "Epoch 12/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2379 - accuracy: 0.9190 - val_loss: 0.2459 - val_accuracy: 0.9175\n",
            "Epoch 13/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2343 - accuracy: 0.9200 - val_loss: 0.2425 - val_accuracy: 0.9183\n",
            "Epoch 14/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2313 - accuracy: 0.9213 - val_loss: 0.2410 - val_accuracy: 0.9190\n",
            "Epoch 15/20\n",
            "43986/43986 [==============================] - 5s 105us/sample - loss: 0.2281 - accuracy: 0.9223 - val_loss: 0.2385 - val_accuracy: 0.9173\n",
            "Epoch 16/20\n",
            "43986/43986 [==============================] - 5s 110us/sample - loss: 0.2252 - accuracy: 0.9224 - val_loss: 0.2360 - val_accuracy: 0.9208\n",
            "Epoch 17/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2229 - accuracy: 0.9232 - val_loss: 0.2419 - val_accuracy: 0.9160\n",
            "Epoch 18/20\n",
            "43986/43986 [==============================] - 5s 105us/sample - loss: 0.2195 - accuracy: 0.9251 - val_loss: 0.2358 - val_accuracy: 0.9170\n",
            "Epoch 19/20\n",
            "43986/43986 [==============================] - 5s 106us/sample - loss: 0.2177 - accuracy: 0.9251 - val_loss: 0.2332 - val_accuracy: 0.9210\n",
            "Epoch 20/20\n",
            "43986/43986 [==============================] - 5s 104us/sample - loss: 0.2154 - accuracy: 0.9261 - val_loss: 0.2372 - val_accuracy: 0.9158\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BDKcWVYHRwLO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_A.save(\"my_model_A.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jd2WXWnHRwLS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_B = keras.models.Sequential()\n",
        "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
        "for n_hidden in (300, 100, 50, 50, 50):\n",
        "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
        "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TiEZ1VC4RwLc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_B.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
        "              metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "36KtLaeVRwLe",
        "colab_type": "code",
        "outputId": "2ed65f69-0919-4438-bea4-f1e8ce183053",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
        "                      validation_data=(X_valid_B, y_valid_B))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 200 samples, validate on 986 samples\n",
            "Epoch 1/20\n",
            "200/200 [==============================] - 0s 2ms/sample - loss: 0.9537 - accuracy: 0.4800 - val_loss: 0.6472 - val_accuracy: 0.5710\n",
            "Epoch 2/20\n",
            "200/200 [==============================] - 0s 471us/sample - loss: 0.5805 - accuracy: 0.6850 - val_loss: 0.4863 - val_accuracy: 0.8428\n",
            "Epoch 3/20\n",
            "200/200 [==============================] - 0s 439us/sample - loss: 0.4561 - accuracy: 0.8750 - val_loss: 0.4116 - val_accuracy: 0.8905\n",
            "Epoch 4/20\n",
            "200/200 [==============================] - 0s 435us/sample - loss: 0.3885 - accuracy: 0.9100 - val_loss: 0.3650 - val_accuracy: 0.9148\n",
            "Epoch 5/20\n",
            "200/200 [==============================] - 0s 470us/sample - loss: 0.3426 - accuracy: 0.9250 - val_loss: 0.3308 - val_accuracy: 0.9270\n",
            "Epoch 6/20\n",
            "200/200 [==============================] - 0s 421us/sample - loss: 0.3084 - accuracy: 0.9300 - val_loss: 0.3044 - val_accuracy: 0.9371\n",
            "Epoch 7/20\n",
            "200/200 [==============================] - 0s 422us/sample - loss: 0.2810 - accuracy: 0.9400 - val_loss: 0.2806 - val_accuracy: 0.9432\n",
            "Epoch 8/20\n",
            "200/200 [==============================] - 0s 425us/sample - loss: 0.2572 - accuracy: 0.9500 - val_loss: 0.2607 - val_accuracy: 0.9462\n",
            "Epoch 9/20\n",
            "200/200 [==============================] - 0s 434us/sample - loss: 0.2372 - accuracy: 0.9600 - val_loss: 0.2439 - val_accuracy: 0.9513\n",
            "Epoch 10/20\n",
            "200/200 [==============================] - 0s 457us/sample - loss: 0.2202 - accuracy: 0.9600 - val_loss: 0.2290 - val_accuracy: 0.9523\n",
            "Epoch 11/20\n",
            "200/200 [==============================] - 0s 429us/sample - loss: 0.2047 - accuracy: 0.9650 - val_loss: 0.2161 - val_accuracy: 0.9564\n",
            "Epoch 12/20\n",
            "200/200 [==============================] - 0s 426us/sample - loss: 0.1917 - accuracy: 0.9700 - val_loss: 0.2046 - val_accuracy: 0.9584\n",
            "Epoch 13/20\n",
            "200/200 [==============================] - 0s 434us/sample - loss: 0.1798 - accuracy: 0.9750 - val_loss: 0.1944 - val_accuracy: 0.9604\n",
            "Epoch 14/20\n",
            "200/200 [==============================] - 0s 430us/sample - loss: 0.1690 - accuracy: 0.9750 - val_loss: 0.1860 - val_accuracy: 0.9604\n",
            "Epoch 15/20\n",
            "200/200 [==============================] - 0s 434us/sample - loss: 0.1594 - accuracy: 0.9850 - val_loss: 0.1774 - val_accuracy: 0.9635\n",
            "Epoch 16/20\n",
            "200/200 [==============================] - 0s 461us/sample - loss: 0.1508 - accuracy: 0.9850 - val_loss: 0.1691 - val_accuracy: 0.9675\n",
            "Epoch 17/20\n",
            "200/200 [==============================] - 0s 430us/sample - loss: 0.1426 - accuracy: 0.9900 - val_loss: 0.1621 - val_accuracy: 0.9686\n",
            "Epoch 18/20\n",
            "200/200 [==============================] - 0s 432us/sample - loss: 0.1355 - accuracy: 0.9900 - val_loss: 0.1558 - val_accuracy: 0.9706\n",
            "Epoch 19/20\n",
            "200/200 [==============================] - 0s 426us/sample - loss: 0.1288 - accuracy: 0.9900 - val_loss: 0.1505 - val_accuracy: 0.9706\n",
            "Epoch 20/20\n",
            "200/200 [==============================] - 0s 414us/sample - loss: 0.1230 - accuracy: 0.9900 - val_loss: 0.1454 - val_accuracy: 0.9716\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SOZMdRnjRwLh",
        "colab_type": "code",
        "outputId": "748891c6-6374-40c4-cc4b-f88a5c82a9e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 499
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_6\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "flatten_6 (Flatten)          (None, 784)               0         \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2_3 (Ba (None, 784)               3136      \n",
            "_________________________________________________________________\n",
            "dense_415 (Dense)            (None, 300)               235200    \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2_4 (Ba (None, 300)               1200      \n",
            "_________________________________________________________________\n",
            "activation (Activation)      (None, 300)               0         \n",
            "_________________________________________________________________\n",
            "dense_416 (Dense)            (None, 100)               30000     \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "batch_normalization_v2_5 (Ba (None, 100)               400       \n",
            "_________________________________________________________________\n",
            "dense_417 (Dense)            (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 270,946\n",
            "Trainable params: 268,578\n",
            "Non-trainable params: 2,368\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UlvmpwHhRwLo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_A = keras.models.load_model(\"my_model_A.h5\")\n",
        "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
        "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XINVSfhrRwLq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_A_clone = keras.models.clone_model(model_A)\n",
        "model_A_clone.set_weights(model_A.get_weights())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rza7plDPRwL4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for layer in model_B_on_A.layers[:-1]:\n",
        "    layer.trainable = False\n",
        "\n",
        "model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
        "                     metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCKwxo5cRwL6",
        "colab_type": "code",
        "outputId": "f0f28c2e-aa20-48e4-9d99-185d087f04ad",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 815
        }
      },
      "source": [
        "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
        "                           validation_data=(X_valid_B, y_valid_B))\n",
        "\n",
        "for layer in model_B_on_A.layers[:-1]:\n",
        "    layer.trainable = True\n",
        "\n",
        "model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
        "                     metrics=[\"accuracy\"])\n",
        "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
        "                           validation_data=(X_valid_B, y_valid_B))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 200 samples, validate on 986 samples\n",
            "Epoch 1/4\n",
            "200/200 [==============================] - 0s 2ms/sample - loss: 0.5837 - accuracy: 0.6650 - val_loss: 0.5838 - val_accuracy: 0.6329\n",
            "Epoch 2/4\n",
            "200/200 [==============================] - 0s 358us/sample - loss: 0.5473 - accuracy: 0.6850 - val_loss: 0.5469 - val_accuracy: 0.6785\n",
            "Epoch 3/4\n",
            "200/200 [==============================] - 0s 358us/sample - loss: 0.5107 - accuracy: 0.7300 - val_loss: 0.5130 - val_accuracy: 0.7170\n",
            "Epoch 4/4\n",
            "200/200 [==============================] - 0s 403us/sample - loss: 0.4773 - accuracy: 0.7550 - val_loss: 0.4849 - val_accuracy: 0.7404\n",
            "Train on 200 samples, validate on 986 samples\n",
            "Epoch 1/16\n",
            "200/200 [==============================] - 0s 2ms/sample - loss: 0.3986 - accuracy: 0.8050 - val_loss: 0.3414 - val_accuracy: 0.8682\n",
            "Epoch 2/16\n",
            "200/200 [==============================] - 0s 390us/sample - loss: 0.2792 - accuracy: 0.9300 - val_loss: 0.2620 - val_accuracy: 0.9320\n",
            "Epoch 3/16\n",
            "200/200 [==============================] - 0s 410us/sample - loss: 0.2125 - accuracy: 0.9650 - val_loss: 0.2146 - val_accuracy: 0.9544\n",
            "Epoch 4/16\n",
            "200/200 [==============================] - 0s 402us/sample - loss: 0.1717 - accuracy: 0.9800 - val_loss: 0.1823 - val_accuracy: 0.9635\n",
            "Epoch 5/16\n",
            "200/200 [==============================] - 0s 386us/sample - loss: 0.1434 - accuracy: 0.9800 - val_loss: 0.1584 - val_accuracy: 0.9736\n",
            "Epoch 6/16\n",
            "200/200 [==============================] - 0s 393us/sample - loss: 0.1230 - accuracy: 0.9850 - val_loss: 0.1406 - val_accuracy: 0.9807\n",
            "Epoch 7/16\n",
            "200/200 [==============================] - 0s 406us/sample - loss: 0.1074 - accuracy: 0.9900 - val_loss: 0.1270 - val_accuracy: 0.9828\n",
            "Epoch 8/16\n",
            "200/200 [==============================] - 0s 395us/sample - loss: 0.0953 - accuracy: 0.9950 - val_loss: 0.1158 - val_accuracy: 0.9848\n",
            "Epoch 9/16\n",
            "200/200 [==============================] - 0s 436us/sample - loss: 0.0854 - accuracy: 1.0000 - val_loss: 0.1076 - val_accuracy: 0.9878\n",
            "Epoch 10/16\n",
            "200/200 [==============================] - 0s 406us/sample - loss: 0.0780 - accuracy: 1.0000 - val_loss: 0.1006 - val_accuracy: 0.9888\n",
            "Epoch 11/16\n",
            "200/200 [==============================] - 0s 418us/sample - loss: 0.0717 - accuracy: 1.0000 - val_loss: 0.0944 - val_accuracy: 0.9899\n",
            "Epoch 12/16\n",
            "200/200 [==============================] - 0s 421us/sample - loss: 0.0661 - accuracy: 1.0000 - val_loss: 0.0890 - val_accuracy: 0.9899\n",
            "Epoch 13/16\n",
            "200/200 [==============================] - 0s 401us/sample - loss: 0.0613 - accuracy: 1.0000 - val_loss: 0.0846 - val_accuracy: 0.9899\n",
            "Epoch 14/16\n",
            "200/200 [==============================] - 0s 393us/sample - loss: 0.0573 - accuracy: 1.0000 - val_loss: 0.0806 - val_accuracy: 0.9899\n",
            "Epoch 15/16\n",
            "200/200 [==============================] - 0s 391us/sample - loss: 0.0537 - accuracy: 1.0000 - val_loss: 0.0769 - val_accuracy: 0.9899\n",
            "Epoch 16/16\n",
            "200/200 [==============================] - 0s 400us/sample - loss: 0.0505 - accuracy: 1.0000 - val_loss: 0.0739 - val_accuracy: 0.9899\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GV8wH3OgRwL-",
        "colab_type": "text"
      },
      "source": [
        "So, what's the final verdict?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyrAj0uoRwMC",
        "colab_type": "code",
        "outputId": "b45852cf-e776-480b-b23c-db1b8e2045f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "model_B.evaluate(X_test_B, y_test_B)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2000/2000 [==============================] - 0s 58us/sample - loss: 0.1431 - accuracy: 0.9705\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.14306601071357727, 0.9705]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vWEh4Hf0RwMG",
        "colab_type": "code",
        "outputId": "6d149a2c-02f3-4326-aad0-70ae3174538a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "model_B_on_A.evaluate(X_test_B, y_test_B)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2000/2000 [==============================] - 0s 63us/sample - loss: 0.0690 - accuracy: 0.9925\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.06896067446470261, 0.9925]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-uv5NxjNRwMJ",
        "colab_type": "text"
      },
      "source": [
        "Great! We got quite a bit of transfer: the error rate dropped by a factor of almost 4!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "46bekHiVRwML",
        "colab_type": "code",
        "outputId": "ba681ec1-cf6b-4fcf-9816-cf4011ec0f8d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "(100 - 97.05) / (100 - 99.25)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3.933333333333337"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b__kXivRRwMP",
        "colab_type": "text"
      },
      "source": [
        "# Faster Optimizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PE6y0IcmRwMP",
        "colab_type": "text"
      },
      "source": [
        "## Momentum optimization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1mN_KqEtRwMR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GnUkKl2wRwMS",
        "colab_type": "text"
      },
      "source": [
        "## Nesterov Accelerated Gradient"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BW462TbNRwMT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRXFp-G_RwMV",
        "colab_type": "text"
      },
      "source": [
        "## AdaGrad"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UUK_xa0wRwMV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.Adagrad(lr=0.001)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cFIa9gzKRwMX",
        "colab_type": "text"
      },
      "source": [
        "## RMSProp"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ieUnK1-cRwMX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZstXiw_6RwMZ",
        "colab_type": "text"
      },
      "source": [
        "## Adam Optimization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nvb2KavKRwMa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BChPWXJRwMc",
        "colab_type": "text"
      },
      "source": [
        "## Adamax Optimization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I2UrsBdSRwMc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djD6xv6tRwMe",
        "colab_type": "text"
      },
      "source": [
        "## Nadam Optimization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C-n6l-3WRwMe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.Nadam(lr=0.001, beta_1=0.9, beta_2=0.999)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e7F7t3PRRwMg",
        "colab_type": "text"
      },
      "source": [
        "## Learning Rate Scheduling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gMnPSzrRwMh",
        "colab_type": "text"
      },
      "source": [
        "### Power Scheduling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UJuergXaRwMh",
        "colab_type": "text"
      },
      "source": [
        "```lr = lr0 / (1 + steps / s)**c```\n",
        "* Keras uses `c=1` and `s = 1 / decay`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iy6uv8Z4RwMh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tfzyEP5XRwMj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oj6lWxRVRwMk",
        "colab_type": "code",
        "outputId": "5505a275-5d62-475d-96b7-8c9ed3cc2217",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "n_epochs = 25\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 6s 102us/sample - loss: 0.4869 - accuracy: 0.8294 - val_loss: 0.4033 - val_accuracy: 0.8594\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 6s 116us/sample - loss: 0.3800 - accuracy: 0.8652 - val_loss: 0.3881 - val_accuracy: 0.8622\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 6s 114us/sample - loss: 0.3470 - accuracy: 0.8771 - val_loss: 0.3601 - val_accuracy: 0.8768\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.3254 - accuracy: 0.8843 - val_loss: 0.3497 - val_accuracy: 0.8766\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 6s 108us/sample - loss: 0.3101 - accuracy: 0.8896 - val_loss: 0.3484 - val_accuracy: 0.8768\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 6s 117us/sample - loss: 0.2978 - accuracy: 0.8942 - val_loss: 0.3419 - val_accuracy: 0.8774\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.2864 - accuracy: 0.8982 - val_loss: 0.3343 - val_accuracy: 0.8796\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.2777 - accuracy: 0.9021 - val_loss: 0.3427 - val_accuracy: 0.8830\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 6s 102us/sample - loss: 0.2703 - accuracy: 0.9033 - val_loss: 0.3308 - val_accuracy: 0.8826\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 5s 100us/sample - loss: 0.2632 - accuracy: 0.9069 - val_loss: 0.3259 - val_accuracy: 0.8862\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 6s 111us/sample - loss: 0.2570 - accuracy: 0.9089 - val_loss: 0.3243 - val_accuracy: 0.8846\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 6s 103us/sample - loss: 0.2514 - accuracy: 0.9119 - val_loss: 0.3245 - val_accuracy: 0.8830\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.2461 - accuracy: 0.9133 - val_loss: 0.3225 - val_accuracy: 0.8850\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2418 - accuracy: 0.9153 - val_loss: 0.3235 - val_accuracy: 0.8884\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2373 - accuracy: 0.9161 - val_loss: 0.3274 - val_accuracy: 0.8886\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.2332 - accuracy: 0.9177 - val_loss: 0.3223 - val_accuracy: 0.8894\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2292 - accuracy: 0.9194 - val_loss: 0.3233 - val_accuracy: 0.8868\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2256 - accuracy: 0.9207 - val_loss: 0.3192 - val_accuracy: 0.8866\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 6s 102us/sample - loss: 0.2226 - accuracy: 0.9214 - val_loss: 0.3178 - val_accuracy: 0.8900\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 6s 117us/sample - loss: 0.2195 - accuracy: 0.9225 - val_loss: 0.3181 - val_accuracy: 0.8892\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 6s 109us/sample - loss: 0.2162 - accuracy: 0.9245 - val_loss: 0.3211 - val_accuracy: 0.8872\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 6s 104us/sample - loss: 0.2134 - accuracy: 0.9256 - val_loss: 0.3195 - val_accuracy: 0.8882\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 6s 103us/sample - loss: 0.2111 - accuracy: 0.9261 - val_loss: 0.3162 - val_accuracy: 0.8896\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2080 - accuracy: 0.9280 - val_loss: 0.3191 - val_accuracy: 0.8902\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2057 - accuracy: 0.9283 - val_loss: 0.3169 - val_accuracy: 0.8896\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y-qJMCZ-RwM1",
        "colab_type": "code",
        "outputId": "5c34ca95-c608-45b9-e819-8fe74c22fd1b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "source": [
        "learning_rate = 0.01\n",
        "decay = 1e-4\n",
        "batch_size = 32\n",
        "n_steps_per_epoch = len(X_train) // batch_size\n",
        "epochs = np.arange(n_epochs)\n",
        "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
        "\n",
        "plt.plot(epochs, lrs,  \"o-\")\n",
        "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.title(\"Power Scheduling\", fontsize=14)\n",
        "plt.grid(True)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNXdx/HPLxuEQNgNJLKpyCqL\nuIJW6m4fF9Rqn9a1Llj7+NjHtWpri7bWWrW1tq5Vi2vFVhAVlaqYKqiIimwii7JI2AQkEEhCCL/n\nj3uDwzBJJpiZyfJ9v17zytxzzr3zm2vMj3PvueeYuyMiIpJIaakOQEREmj4lGxERSTglGxERSTgl\nGxERSTglGxERSTglGxERSTglG5FGxMyWmtm1CTju982sTs9BmNmFZlZS3bZIJCUbaVTMbKyZefiq\nMLMvzOwuM8tJdWzxMLNLzGymmZWYWbGZzTaz36Y6rnoyDtgn1UFIw5SR6gBE9sAbwHlAJnAk8AiQ\nA1yeyqCqmFmWu2+LUX4RcC9wFfAmQfwDgcOTG2FiuHspUJrqOKRhUs9GGqNyd1/t7l+6+zPA08Co\nqkoz+46ZTTezMjNbY2Z/MrOssO5EM9tsZhnh9n5hL+nBiP1/a2ZvRGz3N7NJ4X5rzewfZtYlon6s\nmb1sZj83sxXAimriPhUY7+4Puftid5/v7v9096sjG5nZ98L4S81svZm9ZGYtI5q0NLOHzGyTma0w\ns+ui9m9rZg+HsW42s/+Y2UFRbc43s2VmttXMXgbyourHmNncqLIaL5PFuKw2xszmmtl/m9nnYSwv\nmFmniDYZ4X+fr8PXn8zsATMrrO5zpHFSspGmoJSgl4CZFQCvAjOBocDFwA+B28O2U4GWQNUf35HA\nuvAnEWWF4fG6Am8Dc4FDgGOB1sBEM4v8/+coYBBwInBMNXGuBg4xs2ovNZnZicCLwOvAMOC7wH/Y\n9f/Vq4A5wIHAHcAfzOzwcH8DJgEFwMnhOXgbmBJ+F8zsUGAs8DAwBHgJuLW6mL6lnsAPgNOB48N4\nbouovxa4ELgEOIzge/4oQbFIKrm7Xno1mhfBH8mXI7YPIUgW48Lt24BFQFpEmwuBcqBVuP0+cGP4\n/ing1wQJqyvQKmx7RFh/K/BmVAztAQcOiYjpK6BFLbF3Bd4L910Ufvb5QGZEm2nAszUcYynwj6iy\nRcAvw/dHAyVAdlSbT4Drw/fPAK9H1T8S/DnYuT0GmBvV5kKgpA7bY4AyoG1E2S+AxRHbq4AbIrYN\nWAAUpvp3Ta/6falnI43RieEN9jKCP95vA/8b1vUD3nf3HRHtpwJZwH7hdiHf9GSOIugJTQ/LhgPb\ngQ/C+mHAd8LPKwkvE30Z1u0b8Rlz3b28pqDdfZW7Hw4cANxD8If1IeADM2sVNhtKcD+nJrOjtlcC\ne0XE2wr4KirmgRHx9iM4b5Git+vLMncvjhWrmbUFuvDNuQ6yXcS2NB0aICCN0dvAaKACWOnuFXHu\nVzW0txC4wsz6AbnAR2HZd4G1wHv+zQ3+NILLUrGGG6+JeL8l3uDdfS7BZbn7zOwI4B3gbIIeUjyi\nv6/zzWW2tDCuI2PstyneGIEdBMkwUmYd9q9SU6zSjCjZSGO01d0XV1M3HzjbzNIiejdHANuAz8Pt\nqUAL4HpgqrtXhjek/0bwh/q1iON9TJAIltUhqdXFp+HP1uHPmQT3fP62h8f7mOBm/w53/6KaNvMJ\n7o9Eit7+CsgzMwt7GxDc36k37l5sZquBg4EpsPOe08EE97ekCdG/MKSpuR/IB+43s35m9l/A74G/\nuvtWAHcvIejNnAu8Fe73PrA3wR/dwojj3Qe0BcaZ2aFmto+ZHRuO9mpTl8DCUVY3m9kIM+thZocB\nTwBbgX+HzW4DzgpHxPU3swFmdlXEZbbavEFw32eimZ1kZr3M7HAzu8XMqno79wLHmtmNZtbbzC4l\nuIEfqRDoANxkZvua2cXA9+vyfeP0Z+B6MzvdzPoAdxPc29JCW02Mko00Ke5eBJxEcO/jE+Ax4B/A\nTVFNCwl69oXhfmUE923K2fUewkpgBMFlpdeAeQQJqDx81cXrwKHAc8BCYEJYfpy7Lww/7xWCP/wn\nEfRy/kNweW/HbkeLIeyFfI+gp/A3gpvtzwF9CO6X4O7vE4zSu5zg/s8ZBDfzI48zP6wfHbY5Dvhd\nHb9vPO4CngT+TpDwITgvZQn4LEkh+6aHLCKSemY2k+Dy5v/W2lgaDd2zEZGUMbMewAkEPbhM4FKC\n55UuTWVcUv+SehnNzDqY2QQz2xI+vRzz4S0L3BE+Pb0+fG8R9Q+b2QIz22FmF8bY/yozWx0+Yf2Y\nmbVI4NcSkT23g+BZow8Ihl8fBpzk7h+mNCqpd8m+Z3MfwaigPOAc4AEzGxCj3WiC6UcGE/wr5xTg\nsoj6WcBPCUbe7MLMTgBuIBjR04NgYsBb6u8riEh98WDKoSPcva27t3H3Q93937XvKY1N0u7ZWDAr\n79fAwKqboWb2JFDk7jdEtX0XGOvuD4fbFwOXuvthUe2mAo+4+9iIsmeApe5+U7h9DPC0u3dBRERS\nIpn3bPYHtlclmtAsgie4ow0I6yLbxeoBxTIAmBi1b56ZdXT39ZENzWw0QS+KtOzcYRlt99pZ1zNX\nA/UAduzYQVqazkU0nZfd6ZzE1tTPy8KFC9e5e+fa2iUz2bRm9yeYi4FYzyq0Dusi27WOesCsps+J\n3pfwc3ZJNmHP6WGAFl17e9cL7gGgoF020244upaPaR4KCwsZOXJkqsNocHRedqdzEltTPy9mtiye\ndslMtyUEU4NEygU2x9E2l2CCv3iu+cXal2o+ZzctM9O47oQ+8TQVEZE4JTPZLAQyzKx3RNlggofk\nos0L62prF0usfddEX0KrzmmD8xk1tCDOjxIRkXgkLdm4+xZgPHCrmeWY2QjgNIKnh6M9AVxtZgVm\nlg9cQ8QkhWaWZcFiUgZkmlnLiLVFngAuDqf6aAf8kjgmOOyZm0b/rrnM/HIjetBVRKR+Jfuu1U+B\nbIKZdf8BXO7u88zsSNt1BcCHCBZ0mkMwO+6ksKzKvwnWHxlOcM+lFPgOgLu/BvyBYM6r5cAygvVK\nanXREb1YuKaEaYvj6gSJiEickjqDgLtvIGL53ojyd/hm1tuq+Z2uD1+xjjOyls/5I/DHusZ3yuCu\n/P7V+Tw2bQlH9O5U+w4iIhKXpjsebw+0yEjn3MN6MOWztXzxVbVLrYuISB0p2UQ559AeZKWnMfbd\npakORUSkyVCyidK5TQtOHZLPPz9cQfHWRKyVJSLS/CjZxPDjET0prahk3IfLUx2KiEiToGQTw4D8\nthy2Twcef3cZ2yvjWrNKRERqoGRTjYtG9KJoYyn//nRNqkMREWn0lGyqcUy/PLp3aMVjU5ekOhQR\nkUZPyaYa6WnGhcN78uGyr5n15cZUhyMi0qgp2dTgrIP2pnWLDP4+Tb0bEZFvQ8mmBm1aZnL2Qd14\nefYq1mwqS3U4IiKNlpJNLS4c3pNKd558L64lG0REJAYlm1p079iK4/vn8fT0ZZRVVKY6HBGRRknJ\nJg4XjejF11sreGFmUapDERFplJRs4nBIrw4MyM/lsWlLtNaNiMgeULKJg5lx0QitdSMisqeUbOJ0\n8uCudGrdgsc0DFpEpM6UbOLUIiOd88K1bj7XWjciInWiZFMH5xzWPVjrZtrSVIciItKoKNnUQafW\nLThtSD7/+khr3YiI1IWSTR39eEQvSisqeXaG1roREYmXkk0d9c/P5fB9OvL4u0u11o2ISJyUbPbA\nRUf0YmVxGZPnaa0bEZF4KNnsgaP77kXHnEyuGvcJvW6YxIjfT9HsAiIiNchIdQCN0UuzVrKpbDsV\nlcFsAkUbS7lx/BwARg0tSGVoIiINkno2e+DOyQt2JpoqpRWV3Dl5QYoiEhFp2JRs9sDKjaV1KhcR\nae6UbPZAfrvsOpWLiDR3SjZ74LoT+pCdmb5LWWa6cd0JfVIUkYhIw6YBAnugahDAnZMXsHJjKRnp\nRnZmGsf1z0txZCIiDZOSzR4aNbRgZ9L5aNnXnPnAu9z31mKuP7FviiMTEWl4dBmtHgzr0Z4zDizg\nkXeWsHTdllSHIyLS4CjZ1JMbTuxLVkYav3n501SHIiLS4CjZ1JO9clty5TH78eZna3nrs7WpDkdE\npEFRsqlHFw7vxT6dc7j15U8p316Z6nBERBqMpCYbM+tgZhPMbIuZLTOzH1XTzszsDjNbH77uMDOL\nqB9iZh+Z2dbw55CIuhZm9qCZrTGzDWb2kpklZQ6ZrIw0fnVyf5as28JjU5cm4yNFRBqFZPds7gO2\nAXnAOcADZjYgRrvRwChgMDAIOAW4DMDMsoCJwFNAe+BxYGJYDvAz4PBwv3zga+AvCfo+uxnZZy+O\n7ZfHX6YsYs2msmR9rIhIg5a0ZGNmOcCZwM3uXuLuU4EXgfNiNL8AuNvdV7h7EXA3cGFYN5JgyPY9\n7l7u7vcCBhwd1vcCJrv7GncvA8YBsRJawvzq5P5s3+Hc/sr8ZH6siEiDlcznbPYHtrv7woiyWcBR\nMdoOCOsi2w2IqJvt7pEzYc4Oy18DHgX+bGb5wEaCHtSrsQIys9EEvSg6d+5MYWFhHb9S9U7ons4L\nn6xkQIsN9G6fXvsODVRJSUm9npemQudldzonsem8BJKZbFoDm6LKioE21bQtjmrXOrxvE10XfZxF\nwJdAEVAJzAGuiBWQuz8MPAzQp08fHzlyZJxfpXaHDN/Oh3f/hwnLs3jptCNIT7Pad2qACgsLqc/z\n0lTovOxO5yQ2nZdAMu/ZlAC5UWW5wOY42uYCJWFvprbj3Ae0ADoCOcB4qunZJFKrrAxu+l4/Pl21\niWdnLE/2x4uINCjJTDYLgQwz6x1RNhiYF6PtvLAuVrt5wKDI0WkEgwGq6ocAY919g7uXEwwOOMTM\nOtXDd6iTkwd15dBeHbhr8gI2bt2W7I8XEWkwkpZs3H0LQS/jVjPLMbMRwGnAkzGaPwFcbWYF4b2X\na4CxYV0hweWxK8NhzlWXyKaEP2cA55tZWzPLBH4KrHT3dYn4XjUxM8acOoDi0gr++PrC2ncQEWmi\nkj30+adANrAW+AdwubvPM7Mjzawkot1DwEsE91vmApPCMtx9G8Gw6PMJBgBcBIwKywGuBcoI7t18\nBXwPOD3B36ta/brmct5hPXjq/WV8ujL6lpWISPOQ1Fmf3X0DQaKILn+H4MZ/1bYD14evWMeZCQyr\npm49wQi0BuOq4/bnxVkrGfPSPMaNPoxdrwCKiDR9mq4mCdq1yuK6E/rywZINvDR7VarDERFJOiWb\nJPnBwd0YWJDL7ybNZ+u27akOR0QkqZRskiQ9zRhzygBWbyrjkNvepNcNkxjx+ym8MLMo1aGJiCSc\nVupMohVfl5JuRkl50LMp2ljKjePnAN8sNS0i0hSpZ5NEd05eQOUus+xAaUUld05ekKKIRESSQ8km\niVZuLK1TuYhIU6Fkk0T57bLrVC4i0lQo2STRdSf0ITtz9xmgLx+5TwqiERFJHiWbJBo1tIDbzziA\ngnbZGNC5TQvSDSbPW8OOHV7r/iIijZVGoyXZqKEFu4w8e2b6cm6aMIe/vrWYK4/pXcOeIiKNl3o2\nKfbDQ7px+tAC/vTGQqYtTvpcoSIiSRF3sjGzPDO71sweqJqu38xGmFmvxIXX9JkZvx01kH07t+Zn\nz85kzaayVIckIlLv4ko2ZjYMWEAwweXFfLN42XHAbYkJrfnIaZHBA+ccyJbySv73HzPZXrkj1SGJ\niNSreHs2dwF/dvehQHlE+WRgRL1H1Qz1zmvD784YyAdLNnDXv7X2jYg0LfEmm2HA4zHKVwF59RdO\n83b60L354SHdefA/n/Pm/DWpDkdEpN7Em2xKgfYxyvsSLIQm9eTXp/RnQH4uVz83iy83bE11OCIi\n9SLeZDMR+LWZtQi33cx6AncAzycgrmarZWY6959zIDt2OFc88zHl2ytTHZKIyLcWb7K5FuhAsMxy\nK2AqsJhgWeZfJia05qtHxxzuPGsQs1YUc/srn6U6HBGRby2uhzrdfRNwhJkdDRxIkKQ+dvc3Ehlc\nc3biwK5cfEQvHp26hIN7duC/BnVNdUgiInssrmRjZucD49x9CjAlojwL+G93fyJB8TVrN5zUl5nL\nv+bnz8+mX9c27NO5dapDEhHZI/FeRvs70DZGeZuwThIgMz2Nv/7oQDLTjXP+9j7Db9cKnyLSOMWb\nbAyINVNkd6C4/sKRaPntsjnroL1ZtamclcVlON+s8KmEIyKNRY2X0cxsDkGSceA/ZrY9ojod6AG8\nkrjwBGDS7NW7lVWt8KnlpEWkMajtns2/wp8DgUlASUTdNmApGvqccFrhU0QauxqTjbvfAmBmSwkG\nCGiWyBTIb5dNUYzEohU+RaSxiOuejbs/rkSTOtWt8Pn9YbqEJiKNQ7yzPmeZ2S1mttDMysysMvKV\n6CCbu+gVPrvktqRTTiaPTVvK3CKNzxCRhi/elTp/A/wAuB34E3Ad0BP4b+DmhEQmu4he4bNoYyln\nP/ge5z06nXGXHc7+eW1SGJ2ISM3iHfp8NvATd38IqAQmuvuVwK8J1rSRJCtol83TlxxKZnoa5z4y\nnaXrtqQ6JBGRasWbbPKAT8P3JUC78P1rwPH1HZTEp2enHJ6+5FAqKndwziPTYw4iEBFpCOJNNsuB\n/PD9YuCE8P3hBMsPSIr0zmvDkxcfyqayCs752/us1bLSItIAxZtsJgDHhO//DNxiZkuAscAjCYhL\n6mBgQVvG/vgQ1m4u59xHp7Nhy7ZUhyQisot4hz7f6O63he//BRwB/AU4w91/kcD4JE7DerTnkQsO\nYtn6rZz/2HQ2lVWkOiQRkZ3i7dnswt2nu/sf3f1lM8uJdz8z62BmE8xsi5ktM7MfVdPOzOwOM1sf\nvu4wM4uoH2JmH5nZ1vDnkKj9DzSzt82sxMzWmNnP9uR7NjbD9+3Eg+cOY8Hqzfz47zPYUr699p1E\nRJJgj5INgJm1NLPrgCV12O0+gmlu8oBzgAfMbECMdqOBUcBgYBBwCnBZ+LlZBCuHPkWwVPXjwMSw\nHDPrRDBw4SGgI7Af8O+6fr/G6rt99+Le/x7KzOVfc+kTH1JWocegRCT1akw24cOct5nZDDN718xG\nheXnA18A/0fw3E2twh7QmcDN7l7i7lOBF4HzYjS/ALjb3Ve4exFwN3BhWDeS4Pmge9y93N3vJZiV\n+uiw/mpgsrs/HdZvdvf58cTYVJx0QFfuOmsw732xnjPun6alCUQk5Wp7qHMM8D/A68AI4J9m9jeC\nwQI3As+4e7w3B/YHtrv7woiyWcBRMdoOCOsi2w2IqJvt7pFLHswOy18DDgPmmNm7BL2a6cD/uPvy\n6A8xs9EEvSg6d+5MYWFhnF+l4esAHNE1nXdWbt5ZVrSxlOv/+Qmfzv+U4fmZcR2npKSkSZ2X+qLz\nsjudk9h0XgK1JZuzgQvdfYKZDQZmEly6GuDudb0h0BrYFFVWTLAAW6y2xVHtWof3baLroo+zN8HS\n1ccBc4A/AP8gSJa7cPeHgYcB+vTp4yNHjoz/2zQCv3h/CtEj07ftgEnL07npRyPjOkZhYSFN7bzU\nB52X3emcxKbzEqgt2XQDZgC4+ywz2wbcsQeJBoKHQXOjynKBzXG0zQVK3N3NrLbjlAIT3H0GgJnd\nAqwzs7bu3qwmEtPSBCLSUNQ2QCATKI/YrmDPV+ZcCGSYWe+IssHAvBht54V1sdrNAwZFjk4jGERQ\nVT+bXVcVjbXCaLNQ3RIEndu0SHIkItLcxTMa7XYzu9fM7gWygDFV2xHltXL3LcB44FYzyzGzEcBp\nwJMxmj8BXG1mBWaWD1xD8AApQCHB/GxXmlkLM7siLJ8S/vw7cHo4PDqTYKLQqc2tVwPVL02wuayC\n979Yn4KIRKS5qi3ZvA3sCxwQvt4FukdsH0Cwime8fgpkA2sJ7qNc7u7zzOzI8PJYlYeAlwjuucwl\nWCX0IQB330YwLPp8YCNwETAqLMfdpwA3hfusJRgkEPN5nqYuemmCgnbZ3Pxf/chvl815j07n+Y9W\npDpEEWkmalupc2R9fpi7byBIFNHl7xDc+K/aduD68BXrODOBYTV8zgPAA9823qYgemkCgO8f1I2f\nPv0R1/xzFkvXb+GqY/cnLc2qOYKIyLe3xw91SuPVNjuTsT8+hB8c1I2/TFnMlc/O1MOfIpJQ8S6e\nJk1MZnoavz/zAPbpnMPtr37Gyo2lPHz+QXRqrcEDIlL/1LNpxsyMy47alwfPPZBPV23i9PunsXht\nrJHoIiLfjpKNcOLArowbfThlFTs4/f53mbZ4XapDEpEmRpfRBIDB3drxwv+M4OKxM7jgsQ8488AC\npi5eT9HGUgren8J1J/TZbaCBiEi84ko2Zta9mioHytz9q/oLSVKloF02//zJ4Zz14HuM+/CbYdFF\nG0u5cfwcACUcEdkj8V5GW0qwlED0aymw2sy+NrM/mpl6So1cm5aZMRdeK62o5M7JC1IQkYg0BfEm\nhx8STGj5IMEsygCHEsyYPAZoB/ySYH6yX9dviJJsqzaWxSzXnGoisqfiTTaXA1e5+/iIsilmtgD4\nmbsfZWZrgVtQsmn08ttlUxQjsbRqkU5ZRSUtY0yBIyJSk3gvox1KMHVMtLnAweH79wim95dGLtac\naulpxpbySk7961Tmr4peKUJEpGbxJptlhIuMRbkUqFqUrDOwoT6CktSKnFMNgoEDd581mMcvOoSv\nt1Zw2l+n8ejUJezY0Wwn1BaROor3Mto1wPNm9j3C9W2Agwgm6Twz3D4YeK5+w5NUqZpTLXrhp9d+\ndiQ/f34Ov3n5UwoXrOWuswaTl9sydYGKSKMQV8/G3ScBvYEXCRYqyw3f93H3V8I297v71YkKVBqG\njq1b8Lfzh3Hb6QOZsXQDJ97zNpPnrU51WCLSwMU9VNndvwRuTGAs0kiYGecc2oNDe3Xk/8bN5LIn\nP+KHh3Tj5pP70ypLo99FZHdx/2Uws1bAEGAvonpEUaPUpJnYb6/WjL98BH98fSEPvf0507/YwKih\n+YybsYKVG0vJb5etmQdEBIh/BoFjCRY76xij2gGNhW2msjLSuOGkvhy1f2d+8tSH/PH1RTvrNPOA\niFSJdzTanwlWvtzb3dOiXko0wuH7dox5CU0zD4gIxH8ZrSdwqruvTGAs0sitLtbMAyISW7w9m2lA\nn0QGIo1ffvhcTjQzePaD5XouR6QZizfZPAjcZWaXmNmhZnZg5CuRAUrjEWvmgRYZafTo2Iobxs/h\n9AfeZfaKjSmKTkRSKd7LaP8Kfz4co04DBAT4ZhDAnZMX7DIa7bQh+bzwSRG3TfqM0+6bxg8P6c51\nx/ehfU5WiiMWkWSJN9n0SmgU0mRUzTwQ7fShe3NMvzzueX0Rj7+3lFfmrOL6E/ryg4O7kZ5myQ9U\nRJIqrmTj7ssSHYg0fbktM/nVKf05++C9+dXEedw0YQ7PzljOracNZOm6Lbv1iDRcWqTpqDbZmNkZ\nwEvuXhG+r5Ye6pS66Nsll3GjD+PFWSv57aT5jLpvGulpRmU4gEDP54g0PTX1bP4FdAHW8s09m1h0\nz0bqzMw4bUgBR/fdi8Nvf5OS8spd6quez1GyEWkaqk027p4W671IfWrTMpMtUYmmip7PEWk6lEQk\n5ap7PictzXhuxpdsr9yR5IhEpL7VZSLOvYHvEHsizj/Wc1zSjFx3Qh9uHD+H0opvejhZ6Wnk5bbg\n+udnc3/hYn52bG9OHVygkWsijVS8E3GeAzwGbAe+IrhPU8UBJRvZYzU9n/PG/LX88fWFXDVuFve9\n9TlXHbs/Jw3sQpqSjkijEm/P5lbgbuBmd499gV3kW6ju+Zzj+udxTN+9eG3eav70+kL+55mP6dul\nDVcftz/H9c9j4icrNWRapBGIN9nkAY8o0UgqpKUZ3zugKycM6MLLs1dyzxuLGP3kR3Rrn82aTeVs\nC+/paMi0SMMV7wCBV4BDExmISG3S04Lh0q9f9R3u/P4gVhaX7Uw0VbSkgUjDFG/P5nXgDjMbAMwB\nKiIr9VCnJFNGehpnHdSN6/81O2a9hkyLNDzxJpuHwp83xajTQ52SEvntsimKkVjM4K7JCzj/8B7s\nldsyBZGJSLS4LqPFWJ1zj1bqNLMOZjbBzLaY2TIz+1E17czM7jCz9eHrDjOziPohZvaRmW0Nfw6J\ncYwsM5tvZivijU8al1hLGmRlpDEgP5f7Chcz4o4pXP3cJ8xbWZyiCEWkSq09GzPLBKYC57v7t70Y\nfh+wjWDAwRBgkpnNcvd5Ue1GA6OAwQQ9p9eBJcCDZpYFTATuAe4HLgMmmllvd98WcYzrCIZpt/mW\nMUsDVd2Q6VFDC1i2fgt/n7aUf374JeM/LuKwfTpwyRH7cHTfvUhLM16YWaRRbCJJVGuyCSfi7MWu\nz9bUmZnlAGcCA929BJhqZi8C5wE3RDW/ALjb3VeE+94NXEqwiNvIMO573N2Be83sWuBo4LWwfS/g\nXOBq4G/fJm5p2KobMt2jYw5jTh3AVcftz7gZyxk7bSmXPPEhvTrlcGD3dkyas4qyCo1iE0kWC/5e\n19LI7E4Ad79ujz/IbCgwzd1bRZRdCxzl7qdEtS0Gjnf36eH2QcBb7t7GzK4K606KaP9yWH93xPaj\nwNfAU+6+dzUxjSboRdG5c+dhzz333J5+vSarpKSE1q1bpzqMb61yh/PhmkomL63gi+LY0990bGnc\nPbJVzLpoTeW81Cedk9ia+nn57ne/+5G7H1Rbu3gHCOQA55jZccBHwJbISne/Mo5jtAY2RZUVE/sy\nV+uwLrJd6/C+TXTdLscxs9OBdHefYGYjawrI3R8mXH20T58+PnJkjc2bpcLCQprKeTkGuN6dfW58\nJWY3fUOZx/1dm9J5qS86J7HpvATiTTb9gI/D9/tE1cV7ea0EyI0qywU2x9E2Fyhxdzezao8TXqr7\nA/C9OGOSZsbMqh3FhsGYF+fxg4O70a9r9K+YiHwb8a7U+d16+KyFQEZ4I39RWDYYiB4cQFg2GPgg\nRrt5wDVmZv7NNcBBBIMPegM9gXfCwWtZQFszWw0c5u5L6+F7SCNX3cSf/fPb8Mz05Yx9dymD9m7L\n2Qd149Qh+eS2zExhtCJNQ9woTBL1AAAVaklEQVSzPn9b7r7FzMYDt5rZJQSj0U4Dhsdo/gRwtZm9\nQtBzugb4S1hXCFQCV5rZgwQDBwCmADuAbhHHGQ78FTiQYGSaSI2j2L7eso0XPili3Iwv+eULc/nt\npE/53sCunH1wNw7t1WHnXGxFG0speH+KRrGJxKkuSwx8F/gh0J2gx7CTux8d52F+SjB79FpgPXC5\nu88zsyOBV9296i7aQwSX6+aE24+EZbj7NjMbFZb9HpgPjIoY9rw6IuYNwA5331kmAtWPYmufk8WP\nR/TiwuE9mVNUzLgZX/LiJysZP7OITjmZbCzdznYtXy1SZ3E91GlmFwKvEtyEH0nQS2hP0GP4NN4P\nc/cN7j7K3XPcvbu7PxOWvxORaPDA9e7eIXxdH3HJDHef6e7D3D3b3Q9095nVfF5hdSPRRGpiZgza\nux23nX4AH/ziWO4+azCbyip3JpoqmotNJD7xTsR5LXCFu/+QYF60G919KPAUwc18kSYrOyudM4ft\nTUU1K4YWbSxlwswVbC6riFkvIvFfRtsHeCN8X04w/BiC+yGF7P5QpkiTU90otnSDq8bNIisjjaP7\n7MXJg7tyTN88srM0ZaBIlXiTzXq+eR6mCBgIzAY6ArEXkBdpYmKNYsvOTOd3owbSvVMrXpq1iklz\nVvHavNW0ykrnmH55nDyoK0ft35nX5q7W9DjSrMWbbN4Bjie4Yf8cwRQxxxE8J/d6gmITaVAiR7EV\nbSylICppDOvRgZtP7s8HSzbw8uyVvDp3NS/NWkmLdGP7Dqh0DSyQ5iveZHMFUDVX++3AdmAEQeL5\nbQLiEmmQqkaxVfdUeHqacfi+HTl8347ccuoA3v18PT956iPKK3dd5La0opI7XvtMyUaajXgf6twQ\n8X4HcEfCIhJpIjLS0/jO/p0p3RZ7NfVVxWWc/eB7HNNvL47tn8e+nZvu/FkidXnOJo9ghuZ9gZvd\nfZ2ZjQBWuvuSRAUo0thVN7CgTcsMtmzbzu2vfsbtr35Gr045HNM3SDwH9WhPRnqalkKQJiOuZGNm\nw4A3CdaUGQDcCawDjgP2B2IugiYi1Q8s+M1pAxk1tICVG0t587O1vPHpGp54bxmPTF1C2+xM9uuc\nw+yiYioqda9HGr94ezZ3AX9291+bWeTEmZOBH9d/WCJNR03T40DQ8znvsB6cd1gPSsq3M3XRV7z+\n6VomzFxB1DOklFZU8ofJutcjjU+8yWYYcHGM8lUEq26KSA2qmx4nWusWGZw4sCsnDuzK+I9jr2i+\ncmMZP3nyI47o3Ykje3eiR8ec+g5XpN7Fm2xKCaanidaXYJ4zEaln1d3raZWVzpyiYl6bF0z5171D\nqyDx7NeJ4ft2om2rTN3rkQYn3mQzEfi1mZ0VbruZ9SQYlfZ8AuISafaqfYj09AM4bUg+X6zbwtRF\n63hn0Tpe/GQlz0xfTppBQftsVm0s04Sh0qDEm2yuBV4hmICzFTCV4PLZu8AvExOaSPNW272efTu3\nZt/OrblgeE8qKnfwyZcbeWfROh4oXBxzwtDfTvqU4wfk0SoraSuLiOwU73M2m4AjzOxogpme04CP\n3f2NmvcUkW8j3ns9melpHNyzAwf37MBf3lwUs826km0MGvNvBu3dlkN6deTQXh0Y1rP9LovD6fKb\nJEqd/onj7lMIFikDwMx6AHe6+9n1HZiI7Jnq7vV0zMni7IO78cGSDTw69Qse/M/npBn0z8/lkJ4d\ncZx/fLCcsopgdmtdfpP69G370+2AM+sjEBGpH9Xd67n55P47k0bptkpmLv+a6Us28MGSDTw9fRnl\n23dfQkFDraW+6OKtSBNT270eCNboGb5fJ4bv1wmA8u2V9P3la3iM463cWMZ5j05naPf2DO3ejqHd\n2tGu1TeL9VZdetNS2VITJRuRJijeez1VWmSk1zjUel3JNv46ZdHOh0z36ZTD0O7tMYOXZq3c2SvS\npTepjpKNiAA1D7UeNbSALeXbmb2imJlffs3Hyzbyn4VrWVeybbfjlFZU8rtX5nPq4HzS0iyZX0Ea\nsBqTjZm9WMv+ufUYi4ikUG2X33JaZOxcPgHA3dnnxldiXnpbu7mcwbf8mwEFuRxQ0JaBBW05oKAt\nPTvm7ExAGvnWvNTWs1kfR71mfBZpIupy+c3Mqr301i47k5MHd2VO0SYef28Z28LLbG1aZDCgIJfs\nzDSmLl6vSUabkRqTjbtrkk0RqVZ1l97GnDpgZ9KoqNzBojUlzCnayJyiYuYUbeL9LzbsdqzSikrG\nvDiP7h1b0bdLm2ofPlWPqHHSPRsR2WO1LZUNwQOn/fNz6Z+fyw8ODsp63TAp5uW3jaUVnHH/u5hB\njw6t6Nc1l75dcunXtQ39uuby4dIN3DRh7s7kph5R46FkIyLfSm1LZcdS3eW3vNwW/Oa0gXy2ejPz\nV23is9WbeW3eajzMTAa7JanSikrunLxAyaaBU7IRkaSr7vLbjSf14/gBXTh+QJed5VvKt7NgzWY+\nW7WZmybMiXm8oo2lnPvIdHrntab3Xm3YP681vfPa0DZbU/E0FEo2IpJ08Tx4WiWnRQYHdm/Pgd3b\nc99bi2P2iLIz09lUVsGzH3y5SwLbq00L9s9rQ5rBe19oQEIqKdmISErU9cFTqL5HdPsZwbNAO3Y4\nRRtLWbR2M4vWlLBwTQmL1m5mzorimJffbpowh682l7NP5xz26dyabu2zyUhP26WdekT1Q8lGRBqN\n2npEaWlGtw6t6NahFUf3/WYR4V43TIp5vK3bKrntlfk7tzPTjR4dc9inU5B8iku3Mf7jIs2QUA+U\nbESkUdmTHlF1AxIK2mUz6coj+PyrLXzxVcnOn1+s28JbC9buvOwWqbSikpsnzsUMenbMoWfHHNq2\nytytneaM25WSjYg0edVdfrvuhD60a5XFsB5ZDOvRfpd9tlfuoPcvXo05RHtz2XZ+9uwnO7fbtcqk\nR8ccenVsRY+OOazbUsa/PlSPKJKSjYg0eXUZkFAlIz2t2h5RftuWjL3oEJau28Ky9VtZsn4Ly9Zv\nYcbSr5k4a+XOodqRSisq+eULc9lcVrHzUt/e7bNpkZG+S7umeo9IyUZEmoX6HJBw/Yl92T+vDfvn\ntdltn5qWaygp387NE+ft3DaDLrkt6dY+SD5byit487O1ezRqrqEnKSUbEZFq7EmPqKblGvLbtWTC\nT0ewfMNWvtywleXha8WGUqYtXsfqTWW77VNaUcnPn5/NtMXrKGifTUG7bAraZ7N3u1Z0aduSrIw0\nXphZtEtSbIiX7ZKabMysA/AocDywDrjR3Z+J0c6A3wOXhEWPADe4B51TMxsSHqcfMB+42N0/Ceuu\nAy4AeoSfcb+735nI7yUiTVe99ohO6EtebkvycltycM8Ou+1X3TQ+5dt38Pair1i7uXyXS3RmkNem\nJRu2bGNb5a4rrZZWVHLHa59x2pB8gj+psSWrR5Tsns19wDYgDxgCTDKzWe4+L6rdaGAUMJhgdorX\nCWaXftDMsoCJwD3A/cBlwEQz6+3u2whmtDgfmA3sC/zbzL5092cT/u1ERIhvzrhYaho1N+2Goynf\nXsnq4jJWfF1K0delrNgY/Hz+4xUxj7equIwBv55M17YtyW+XTde2LenaNpv8dsHP+auK+dMbiyir\nqPtAhqokldVlv2G1nhCSmGzMLAc4Exjo7iXA1HC9nPOAG6KaXwDc7e4rwn3vBi4FHgRGhnHfE/Z0\n7jWza4Gjgdfc/Q8Rx1lgZhOBEYCSjYgkzZ7MGVfTqDkILtH16JhDj445u+z3/hfrYyapttkZnHlg\nN1YVl7KyuIwFq7/iq5LymAMYqpRWVHLzC3PZsm07XcJeWJe2LenQKmuXtYii46yNeU2fWo/MbCgw\nzd1bRZRdCxzl7qdEtS0Gjnf36eH2QcBb7t7GzK4K606KaP9yWH931HEM+Bh4yN0fjBHTaIJeFJ07\ndx723HPP1dO3bTpKSkpo3bp1qsNocHRedqdzEltdz8u7Kyt4fmEF68ucji2NM/fPZHj+7s/xRO8z\ndu42tkVcSctKgwsHZu227/YdztdlzoYy5/YPdr9HVJ10g3YtjPYtjeWbduz8rFWP/x/lqxbVuiRr\nMi+jtQY2RZUVA7sP5wjaFke1ax0mj+i6mo4zBkgD/h4rIHd/GHgYoE+fPh7vvz6ak7r8q6w50XnZ\nnc5JbHU9LyOBm+r4GSOB/ntw7+WJhVOqHcjw/OXDWV1cxppNZawuLmP1pvKd7xdvrG1dzd0lM9mU\nsPsy0rnA5jja5gIl7u5mFtdxzOwKgns3R7p7+bcJXESkoavvgQxd22bTtW12zP1G/D52kqpJWu1N\n6s1CIMPMekeUDQaiBwcQlg2upt08YJDtOrxiUORxzOwigvtAx1Td9xERkV2NGlrA7WccQEG7bIxg\nIELVpKY1ue6EPmRnptfYJlrSejbuvsXMxgO3mtklBKPRTgOGx2j+BHC1mb1CMBrtGuAvYV0hUAlc\naWYPEgwcAJgCYGbnAL8DvuvuXyTo64iINAl70iOKHG23Ks59ktmzAfgpkA2sBf4BXO7u88zsyPDy\nWJWHgJeAOcBcYFJYRji8eRTBJbKNwEXAqLAc4LdAR2CGmZWEr90GB4iIyJ4bNbSAaTcczbbViz+K\np31Sn7Nx9w0EiSK6/B2CG/9V2w5cH75iHWcmEHNst7v3qpdgRUSk3iS7ZyMiIs2Qko2IiCScko2I\niCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCSc\nko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2I\niCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCSc\nko2IiCScko2IiCScko2IiCRcUpONmXUwswlmtsXMlpnZj6ppZ2Z2h5mtD193mJlF1A8xs4/MbGv4\nc0i8+4qISPIlu2dzH7ANyAPOAR4wswEx2o0GRgGDgUHAKcBlAGaWBUwEngLaA48DE8PyGvcVEZHU\nSFqyMbMc4EzgZncvcfepwIvAeTGaXwDc7e4r3L0IuBu4MKwbCWQA97h7ubvfCxhwdBz7iohICmQk\n8bP2B7a7+8KIslnAUTHaDgjrItsNiKib7e4eUT87LH+tln13YWajCXpCAOVmNje+r9KsdALWpTqI\nBkjnZXc6J7E19fPSI55GyUw2rYFNUWXFQJtq2hZHtWsd3nuJros+TrX7RiUo3P1h4GEAM/vQ3Q+K\n/+s0Dzovsem87E7nJDadl0Ay79mUALlRZbnA5jja5gIlYbKo7Tg17SsiIimQzGSzEMgws94RZYOB\neTHazgvrYrWbBwyKGmE2KKq+un1FRCQFkpZs3H0LMB641cxyzGwEcBrwZIzmTwBXm1mBmeUD1wBj\nw7pCoBK40sxamNkVYfmUOPatycN1/1bNgs5LbDovu9M5iU3nBbBkXl0ysw7AY8BxwHrgBnd/xsyO\nBF5199ZhOwPuAC4Jd30E+HnVpTAzGxqW9QfmAxe7+8x49hURkeRLarIREZHmSdPViIhIwinZiIhI\nwjX7ZBPvfG3NjZkVmlmZmZWErwWpjinZzOwKM/vQzMrNbGxU3TFm9lk4P99bZhbXg21NQXXnxcx6\nmplH/M6UmNnNKQw1qcIBS4+Gf0c2m9knZnZSRH2z/Z0BJRuIf7625ugKd28dvvqkOpgUWAn8lmBQ\ny05m1olgZOXNQAfgQ2Bc0qNLnZjnJUK7iN+b3yQxrlTLAL4kmBWlLfBL4LkwCTf335mkziDQ4ETM\n1zbQ3UuAqWZWNV/bDSkNTlLO3ccDmNlBwN4RVWcA89z9n2H9GGCdmfV198+SHmiS1XBemrXw8Y4x\nEUUvm9kSYBjQkWb8OwPq2VQ3X5t6NoHbzWydmU0zs5GpDqYB2WX+vfCPzOfo96bKMjNbYWZ/D/9F\n3yyZWR7B35h56Hem2SebuszX1tz8HNgHKCB4KO0lM9s3tSE1GLXNz9dcrQMOJpiYcRjB+Xg6pRGl\niJllEnz3x8OeS7P/nWnuyaYu87U1K+4+3d03h8s4PA5MA76X6rgaCP3exBAuHfKhu2939zXAFcDx\nZtZs/qACmFkawcwo2wjOAeh3ptknm7rM19bcOcG6QRI1/154729f9HsTreqJ8WbzdyacweRRggFH\nZ7p7RVjV7H9nms0vQSx1nK+t2TCzdmZ2gpm1NLMMMzsH+A7BekHNRvjdWwLpQHrV+QAmAAPN7Myw\n/lcEayw1ixu91Z0XMzvUzPqYWZqZdQTuBQrdPfryUVP2ANAPOMXdSyPKm/XvDADu3qxfBMMQXwC2\nAMuBH6U6plS/gM7ADIIu/kbgfeC4VMeVgvMwhuBf55GvMWHdscBnQCnB5LA9Ux1vqs8L8ENgSfj/\n0iqCSXG7pDreJJ6XHuG5KCO4bFb1Oqe5/864u+ZGExGRxGvWl9FERCQ5lGxERCThlGxERCThlGxE\nRCThlGxERCThlGxERCThlGxEmqhwbZnvpzoOEVCyEUkIMxsb/rGPfr2f6thEUqFZr2cjkmBvEKyN\nFGlbKgIRSTX1bEQSp9zdV0e9NsDOS1xXmNmkcJngZWZ2buTOZnaAmb1hZqVmtiHsLbWNanOBmc0J\nl2heY2aPR8XQwcz+GS57/kX0Z4gki5KNSOrcArwIDCFYM+iJcPXLqlmBJxPMrXUIcDownIilmM3s\nMuAh4O/AIIIlIOZGfcavgIkEMw6PAx4zs+6J+0oisWluNJEEMLOxwLkEkzJGus/df25mDjzi7pdG\n7PMGsNrdzzWzS4G7gL3dfXNYPxJ4C+jt7ovNbAXwlLvHXMI8/Izfu/uN4XYGwWKBo939qXr8uiK1\n0j0bkcR5GxgdVbYx4v17UXXvAf8Vvu9HMAV95OJa7wI7gP5mtolgFdU3a4lhdtUbd99uZl8Be8UX\nvkj9UbIRSZyt7r44Acety+WIiqhtR5fPJQX0SyeSOofF2J4fvp8PHBC1pPJwgv9n57v7WqAIOCbh\nUYrUA/VsRBKnhZl1iSqrdPevwvdnmNkMgoW0vk+QOA4N654mGEDwhJn9CmhPMBhgfERv6TbgT2a2\nBpgEtAKOcfe7E/WFRPaUko1I4hxLsGJlpCJg7/D9GOBMguWTvwJ+7O4zANx9q5mdANwDfEAw0GAi\n8LOqA7n7A2a2DbgGuAPYALySqC8j8m1oNJpICoQjxc5y93+lOhaRZNA9GxERSTglGxERSThdRhMR\nkYRTz0ZERBJOyUZERBJOyUZERBJOyUZERBJOyUZERBLu/wFIBub2peel9wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hORSkPbWRwM4",
        "colab_type": "text"
      },
      "source": [
        "### Exponential Scheduling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VYO8BhEyRwM5",
        "colab_type": "text"
      },
      "source": [
        "```lr = lr0 * 0.1**(epoch / s)```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QlrIXjJZRwM5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def exponential_decay_fn(epoch):\n",
        "    return 0.01 * 0.1**(epoch / 20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wAP5S7GhRwM6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def exponential_decay(lr0, s):\n",
        "    def exponential_decay_fn(epoch):\n",
        "        return lr0 * 0.1**(epoch / s)\n",
        "    return exponential_decay_fn\n",
        "\n",
        "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dGwBbzvyRwM9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 25"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x3HcJdKCRwM_",
        "colab_type": "code",
        "outputId": "19aef27f-959f-41a4-b5d2-12448a26df7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid),\n",
        "                    callbacks=[lr_scheduler])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 9s 158us/sample - loss: 0.8623 - accuracy: 0.7554 - val_loss: 1.4451 - val_accuracy: 0.7426\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 9s 156us/sample - loss: 0.7047 - accuracy: 0.7889 - val_loss: 0.6948 - val_accuracy: 0.8198\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 9s 156us/sample - loss: 0.6160 - accuracy: 0.8109 - val_loss: 0.5809 - val_accuracy: 0.8214\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 9s 157us/sample - loss: 0.6028 - accuracy: 0.8201 - val_loss: 0.9157 - val_accuracy: 0.7334\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 9s 158us/sample - loss: 0.6135 - accuracy: 0.8212 - val_loss: 0.7253 - val_accuracy: 0.8254\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 9s 156us/sample - loss: 0.4852 - accuracy: 0.8492 - val_loss: 0.5630 - val_accuracy: 0.8374\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 9s 157us/sample - loss: 0.4277 - accuracy: 0.8621 - val_loss: 0.4656 - val_accuracy: 0.8640\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 9s 155us/sample - loss: 0.3894 - accuracy: 0.8723 - val_loss: 0.4693 - val_accuracy: 0.8622\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 10s 179us/sample - loss: 0.3506 - accuracy: 0.8825 - val_loss: 0.5129 - val_accuracy: 0.8558\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 9s 158us/sample - loss: 0.3433 - accuracy: 0.8867 - val_loss: 0.8301 - val_accuracy: 0.8668\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 9s 155us/sample - loss: 0.3094 - accuracy: 0.8985 - val_loss: 0.4594 - val_accuracy: 0.8854\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 8s 154us/sample - loss: 0.2935 - accuracy: 0.9021 - val_loss: 0.4999 - val_accuracy: 0.8700\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 8s 152us/sample - loss: 0.2686 - accuracy: 0.9092 - val_loss: 0.5222 - val_accuracy: 0.8758\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 9s 164us/sample - loss: 0.2520 - accuracy: 0.9147 - val_loss: 0.4590 - val_accuracy: 0.8886\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 10s 178us/sample - loss: 0.2279 - accuracy: 0.9217 - val_loss: 0.4396 - val_accuracy: 0.8794\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 9s 155us/sample - loss: 0.2139 - accuracy: 0.9266 - val_loss: 0.4883 - val_accuracy: 0.8818\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 9s 155us/sample - loss: 0.1978 - accuracy: 0.9322 - val_loss: 0.5028 - val_accuracy: 0.8894\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 10s 175us/sample - loss: 0.1859 - accuracy: 0.9374 - val_loss: 0.4945 - val_accuracy: 0.8886\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 9s 159us/sample - loss: 0.1710 - accuracy: 0.9411 - val_loss: 0.5028 - val_accuracy: 0.8910\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 9s 170us/sample - loss: 0.1602 - accuracy: 0.9471 - val_loss: 0.5477 - val_accuracy: 0.8932\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 8s 155us/sample - loss: 0.1486 - accuracy: 0.9499 - val_loss: 0.5466 - val_accuracy: 0.8924\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 9s 155us/sample - loss: 0.1381 - accuracy: 0.9544 - val_loss: 0.5284 - val_accuracy: 0.8904\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 8s 154us/sample - loss: 0.1322 - accuracy: 0.9566 - val_loss: 0.5772 - val_accuracy: 0.8936\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 8s 153us/sample - loss: 0.1215 - accuracy: 0.9602 - val_loss: 0.5826 - val_accuracy: 0.8896\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 8s 153us/sample - loss: 0.1153 - accuracy: 0.9632 - val_loss: 0.5869 - val_accuracy: 0.8936\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WF6hd18HRwNA",
        "colab_type": "code",
        "outputId": "28334307-1a49-448a-e404-200f808c1d55",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "source": [
        "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
        "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.title(\"Exponential Scheduling\", fontsize=14)\n",
        "plt.grid(True)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNX9//HXJySBQIAQQJAgmwLK\njog7iluxrVYUa6vWpa1ibf3ZVsVq1VZtrSKlVatV+LovrSuIioJajIKKyiKbCKiI7DuBQCAsn98f\n9waHYSaZYGYmJO/n43Efmbnn3Dufexjyyb333HPM3REREUmmjHQHICIiNZ+SjYiIJJ2SjYiIJJ2S\njYiIJJ2SjYiIJJ2SjYiIJJ2SjUgKmNmlZlZcyW0Kzez+ZMUUfsbXZnZdEvZ7rplV6rmK6DbalzaT\n6kvJRpLKzB43M4+xTE53bMkSHt+5UaufAzok4bMuM7PpZlZsZkVmNtPM/lrVn5MmSWkzSY/MdAcg\ntcLbwEVR60rTEUi6uHsJUFKV+zSzXwD3Ab8H/gdkAd2AY6ryc9IlGW0m6aMzG0mFbe6+ImpZB2Bm\nJ5rZdjPrX1bZzK4ws41m1iF8X2hmD5nZvWa2PlyGmVlGxDZNzOyJsKzEzN42s64R5ZeGf/2fYmaz\nzWyzmb1jZu0jAzWzM81sqpltNbOFZnaHmWVHlH9tZjeb2YgwxiVmNiSyPHz5QniG83Xk50fUO9jM\nxpjZijCWaWZ2RiXb9UfAKHcf4e5fuPtcd3/B3a+JOqYfmNlHYbusNbNXzaxeRJV68Y4n3L6xmY00\ns1VmtsnM3jWzI6LqXGxmi8xsi5m9BrSIKr/VzGZHrSv3MlmMNrs1/Lf7qZl9Gcbyspk1i6iTaWb/\njPie/NPMHjSzwoqbU5JJyUbSyt3fBYYBT4UJ41DgH8D/c/evIqpeSPB9PQa4AhgM/C6i/HHgKOAs\n4EhgCzDOzHIi6tQFbgR+Ee4nD3iorNDMBgDPAPcDXcN65wJ/iwr798As4HBgKHC3mZWdTfQNf14O\nHBjxPlou8AZwGtATeAkYFR5/olYAR5Yl5VjM7HTgFeAtoA9wEvAue/7fj3s8ZmbAWKAAOAPoDbwH\nTDCzA8M6RxG0/0igF/AqcHsljqMy2gE/Ac4GvhfGc0dE+XXApcBlwNEEx3lBkmKRynB3LVqSthD8\nEtoBFEctQyPqZAGfAKOAacBzUfsoBOYDFrHuZmBJ+Loj4MAJEeWNgSLgsvD9pWGdzhF1LgS2le2X\n4JfoLVGfPTCMt6zO18B/o+osAG6OeO/AuVF1LgWKK2iryVH7KQTuL6f+gcCH4ectAJ4GLgayIuq8\nDzxbzj7KPR7g5PD4c6LqfApcH77+D/BWVPnDwa+X3e9vBWaX1yYJvL8V2Ao0jlh3E/BFxPvlwA0R\n7w2YBxSm+/9CbV90ZiOp8B7BX7yRy7CyQnffTvDX5xnAAQRnLtEme/jbI/QhUGBmjYDDgF3hurJ9\nFhH8td4lYptt7j4v4v0yIBtoEr7vA9wUXm4rDi/h/AdoALSM2G5mVGzLwrgTZmYNzOxuM/ssvNxT\nDBwBtEl0H+6+3N2PAboD9xD8Yh0BfGxm9cNqvQnu55SnvOPpA9QHVke1Szfg4LDOYUS0fSj6fVVZ\nFP7b7hWrmTUm+Hf6uKww/M58jKSdOghIKmxx9y8qqFN2ySMPaA5sqKLPjkxQO+KUZUT8vA14IcZ+\nVke83h5jP5X9w+3vwOkEl30WEFz2e5Ig+VWKu88GZgMPmNnxwETgPIKzykSUdzwZwEqgX4ztNlYi\nzF0EyTBSViW2L1MVbS9poH8kSbvwJv39wG8I7i08bWbRfwgdFd4/KHM0sMzdNwJz+fZ+Ttk+GxH8\nxf9ZJUKZBhzqwc326CU6UZVnO1CngjrHA0+6+0vuPhNYwrdnCt9F2fHmhj+nA6d8h/1NI7jZvytG\nm6wK68wl+PeIFP1+NdAi6t+w13eIay/hGc8KIu6ThZ8X776ZpJDObCQV6ppZy6h1O919tZnVAZ4C\n3nX3EWb2IsHlrz8Dt0TUbwXcY2b/JkgiQ4C/Arj7AjMbA4wws8EEZ0V3EPzl/Z9KxHk78JqZLQKe\nJzgT6gYc6e7XV2I/XwOnmNm7BJfu1seoMx84O4x7O8Hx1otRLy4ze5DgMtIEgmR1IMG9rC3Am2G1\nO4BXzewLgrYwghvrI9x9SwIf8zbBfZ8xZnY98DnBparTgbfdfSJB9+sPzOxG4EWgP8EN/EiFQD7w\nRzN7NqwT/SxSVbgXuN7M5hMk3isI2mV5Ej5LKkFnNpIKpxL8Z49cpodlfwQOAX4J4O5rgUuAG8JL\nQmWeIThb+Aj4P+AR4J8R5T8nuDb/SvizPnC6B89qJMTdxwM/JOix9XG43AB8k/ihAnBtuI/FfHuc\n0a4BVhFc8nqDoHPAxEp+zlsEPfCeJ0heo8P1p7n7fAB3f53gF//3w1jeDWPblcgHhPc8fkCQ0P6P\n4Gb780BngkSHu08m+Pe7kuD+zzkEN/Mj9zM3LB8c1jmNvXv5VYW/E/zx8hhBm0LQLluT8FlSCWU9\nbESqrfAZidnuflW6Y5H9j5lNBya5+/9Ldyy1mS6jiUiNYWZtgQEEZ3BZBM879Qh/Shop2YhITbKL\n4FmjYQS3CT4Dvu/uU9IalegymoiIJJ86CIiISNLpMlooLy/PDznkkHSHUe1s3ryZBg0apDuMakft\nsje1SWw1vV2mTp26xt2bV1RPySbUokULpkzRZd1ohYWF9O/fP91hVDtql72pTWKr6e0SPpdWIV1G\nExGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGR\npFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpEtpsjGzfDMbbWabzWyRmV0Qp56Z2VAzWxsu\nQ83MIspHmtk8M9tlZpfG2P73ZrbCzDaa2aNmVrei2L7euIvj7prAy9OXfqdjFBGRvaX6zOYBoBRo\nAVwIPGhmXWPUGwwMBHoCPYAzgSsiymcAvwamRW9oZgOAG4BTgLZAB+C2RIJbuqGEG0fNUsIREali\nKUs2ZtYAGATc4u7F7j4JeAW4KEb1S4Dh7r7E3ZcCw4FLywrd/QF3/x+wNc62j7j7HHdfD/wlctuK\nlGzfybDx8xKtLiIiCUjltNCdgB3uPj9i3QzgxBh1u4ZlkfVinQHF0hUYE7VtCzNr6u5rIyua2WCC\nsyiyWx6ye/3SDSUUFhYm+HE1W3FxsdoiBrXL3tQmsaldAqlMNrnAxqh1RUDDOHWLourlmpm5uyfw\nOdHbEn7OHsnG3UcCIwHqHthx934L8nJq9JzhlVHT50/fV2qXvalNYlO7BFJ5z6YYaBS1rhGwKYG6\njYDiBBJNvG2J8zl7qWPGkAGdE6kqIiIJSmWymQ9kmlnHiHU9gTkx6s4JyyqqF0usbVdGX0KLpWHd\nTHa607h+VoIfJSIiiUhZsnH3zcAo4HYza2BmxwFnAU/FqP4kcI2ZFZhZK+Ba4PGyQjPLNrN6gAFZ\nZlbPzDIitv2lmXUxszzg5sht42nXKIMpt5zKIQfkctOoWRRv27HvBysiIntIddfnXwM5wCrgv8CV\n7j7HzPqZWXFEvRHAq8AsYDYwNlxX5k2gBDiW4J5LCXACgLuPA+4G3gG+ARYBf04kuLqZdRg6qAfL\nN27l7nGf7/NBiojInlLZQQB3X0fw/Ez0+okEN/bL3jtwfbjE2k//Cj7nH8A/9iXGPm2b8PNj2/Po\n+ws5o0crjmyfvy+7ERGRCBquJobrBnSidZMc/vDSTLZu35nucERE9ntKNjHUz87krnN6sHDNZu79\n34J0hyMist9Tsonj+I7NOO+I1ox87ytmLy2qeAMREYlLyaYcN/2gC/kNsrn+xZls37kr3eGIiOy3\nlGzK0bh+Fn85qxufLd/IyPe+Snc4IiL7LSWbCpzerSU/6N6Se/+3gC9WFVe8gYiI7EXJJgG3/qgr\nOVl1uOGlmezalciIOSIiEknJJgEHNKzHLWd0Ycqi9Tw1eVG6wxER2e8o2SRo0OEF9OvYjKHjPmfJ\n+i3pDkdEZL+iZJMgM+NvZ3cH4I+jZ5PYANQiIgJKNpVyUH59/nD6obw3fzWjpmnqaBGRRCnZVNJF\nR7fliLZNuP21z1i9aVu6wxER2S8o2VRSRoZx16AeFG/dzgl3T6D9DWM57q4JvDxdZzoiIvGkdNTn\nmmL20iLMjJLtwagCSzeUcOOoWQAM7F2QztBERKolndnsg2Hj57Ej6nmbku07GTZ+XpoiEhGp3pRs\n9sGyDSWVWi8iUtsp2eyDVnk5lVovIlLbKdnsgyEDOpOTVWev9T/te1AaohERqf6UbPbBwN4F3HlO\ndwrycjDgwMb1aJyTyUvTllC8bUe6wxMRqXbUG20fDexdsEfPs8lfreWC/5vMn8bM5h/n9UpjZCIi\n1Y/ObKrI0R2actXJHRk1bSmjpy9JdzgiItWKkk0VuvrkQ+jbrgk3j57N12s2pzscEZFqQ8mmCmXW\nyeCen/Yms04GVz87ndIdmkpaRASUbKpcQV4OQwd1Z+aSIoa/qYc8RURAySYpTu92IBce1YYR733F\nu/NXpzscEZG0U7JJklvO6EKnFrlc+/ynGh1aRGo9JZskqZdVh3+dfzibtu7gmuc/ZdcuTbYmIrWX\nkk0SdW7ZkFvO6MLEBWt4eNJX6Q5HRCRtlGyS7MKj2nB615bcPW4eMxZvSHc4IiJpoWSTZGbGXYO6\nc0DDulz97HQ2bd2e7pBERFIupcnGzPLNbLSZbTazRWZ2QZx6ZmZDzWxtuAw1M4so72VmU81sS/iz\nV0RZXTN7yMxWmtk6M3vVzNI6o1le/WzuPb83i9dt4U9j5qQzFBGRtEj1mc0DQCnQArgQeNDMusao\nNxgYCPQEegBnAlcAmFk2MAZ4GmgCPAGMCdcD/BY4JtyuFbAe+FeSjidhfdvl89tTOjF6+lJ63/6m\nppMWkVolZcnGzBoAg4Bb3L3Y3ScBrwAXxah+CTDc3Ze4+1JgOHBpWNafYADRe9x9m7vfBxhwclje\nHhjv7ivdfSvwHBAroaXcQU1yyDBYv2U7zrfTSSvhiEhNl8pRnzsBO9x9fsS6GcCJMep2Dcsi63WN\nKJvp7pF9iWeG68cBjwD3mlkrYAPBGdQbsQIys8EEZ1E0b96cwsLCSh5S5dxRuIXoHtAl23fylzEz\nyCtakNTP3lfFxcVJb5f9kdplb2qT2NQugVQmm1xgY9S6IqBhnLpFUfVyw/s20WXR+1kALAaWAjuB\nWcBVsQJy95HASIDOnTt7//79EzyUfbNu3NjY67c6yf7sfVVYWFhtY0sntcve1CaxqV0CCV9GM7MW\nZnadmT1oZs3CdceZWfsEd1EMNIpa1wjYlEDdRkBxeDZT0X4eAOoCTYEGwCjinNmkmqaTFpHaKqFk\nY2Z9gHkEl6R+ybe/7E8D7kjws+YDmWbWMWJdTyBW96w5YVmsenOAHpG90wg6A5SV9wIed/d17r6N\noHPAkWUJMp3iTSf9/W4t0xCNiEjqJHpm83fgXnfvDUQO9DUeOC6RHbj7ZoKzjNvNrIGZHQecBTwV\no/qTwDVmVhDee7kWeDwsKyS4PHZ12M257BLZhPDnJ8DFZtbYzLKAXwPL3H1NYoeaPNHTSbdqXI/W\neTk8+8liFqyMdYInIlIzJHrPpg/BGU205QTdmBP1a+BRYBWwFrjS3eeYWT/gDXfPDeuNADoQ3G8B\neDhch7uXmtnAcN1dwFxgoLuXhnWvA+4juHeTDcwGzq5EjEkVPZ308qISzvzX+1z+5BTG/OZ4GtfP\nSmN0IiLJkWiyKSF4piXaoQSJIyHuvo7g+Zno9RMJbvyXvXfg+nCJtZ/pBAkwVtlagst9+4UDG+cw\n4qLDOX/kR1z132k8dmlfMutoYAcRqVkS/a02BvizmdUN37uZtQOGAi8lIa5apU/bfP46sBsTF6zh\nb69/nu5wRESqXKLJ5jogH1gN1AcmAV8QPMdyc3JCq13O63sQlx7bjkffX8gLUxanOxwRkSqV0GU0\nd98IHG9mJwOHEySpae7+djKDq21u/uFhLFi1iZtGz6ZD81z6tI115VJEZP+TaNfni82srrtPcPe/\nu/vd7v62mWWb2cXJDrK2yKyTwf3nH86BefX41dNTWVG0Nd0hiYhUiUQvoz0GNI6xvmFYJlWkSYNs\n/u/iI9iybQeDn5rC1u070x2SiMh3lmiyMSDWvMZt2HvoGPmOOrVoyD0/7c2spUXc8NJM9hwGTkRk\n/1PuPRszm0WQZBx418x2RBTXAdoCrycvvNrrtC4tuPa0Tvz9zfkcdmAjrjjx4HSHJCKyzyrqIPBi\n+LMbMJZgXLIypcDXqOtz0vzmpEOYu2ITd437nE4tG3JS5wPSHZKIyD4pN9m4+20AZvY18Fw4P4yk\niJkx7NweLFy9mV89OYXG9bNZvWkbrfJyGDKg8x4jEYiIVGcJ3bNx9yeUaNKjfnYmPz6iNdt2Oqs2\nbdOkayKyX0q063O2md1mZvPNbKuZ7Yxckh1kbffwxIV7rSvZvpNh4+elIRoRkcpLtDfaXwinagZ2\nAUMI5o1ZSzC4piTRsg0llVovIlLdJJpszgN+5e4jCIb3H+PuVwN/JpjTRpIo/qRr9VIciYjIvkk0\n2bQAPgtfFwN54etxwPeqOijZU7xJ13q0zotRW0Sk+kk02XwDtApffwEMCF8fQzD9gCTRXpOu5dWj\nb9smvDF7BU9NXpTu8EREKpTofDajgVOAycC9wH/N7HKgABiWpNgkQvSka9t37uJXT03lT2Nmk18/\nmx/2ODCN0YmIlC/RUZ9vjHj9opktJpgOer67v5as4CS+rDoZ3H/B4Vz86Ef87rnpNM7J4viOzdId\nlohITPs0JaS7f+Tu/3D318ysQVUHJYnJya7Dw5f05eDmuQx+agozFm9Id0giIjHt8/zDZlbPzIYA\nez8EIinTOCeLJ39xJPkNsrn0sY/5YlVxxRuJiKRYuckmfJjzDjP7xMw+MLOB4fqLga+A3wH/TEGc\nUo4DGtXj6V8eRZ0M4+JHPmJ5kfpsiEj1UtGZza3AVcAioD3wgpn9G7gJuBFo5+53JjVCSUi7Zg14\n/OdHsmnrDi565GPWby5Nd0giIrtVlGzOAy5193OB0wmmFWgCdA3HS9ue7AAlcd0KGjPy4iP4Zt0W\nfv74J2wp3VHxRiIiKVBRsjkI+ATA3WcQTCsw1N31W6yaOubgpvzr/N7MXLKBXz09jdIdu9IdkohI\nhV2fs4BtEe+3o5k5q70BXVty5znd+cNLs/jpiA9ZsWkryzds1dQEIpI2iTxnc6eZbQlfZwO3mtke\nCSccJ02qkZ/0bcPEBat5beaK3evKpiYAlHBEJKUqSjbvAZHzEX8AtImq41UakVSZ6d/s/dxN2dQE\nSjYikkoVzdTZP0VxSBIs2xB7vjtNTSAiqbbPD3VK9aepCUSkulCyqcHiTU3QoXku7rr6KSKpk9Jk\nY2b5ZjbazDab2SIzuyBOPTOzoWa2NlyGmplFlPcys6lmtiX82Stq+8PN7D0zKzazlWb222QfW3UU\nPTVBQV49TuzYjIkL1nDTy7PZtUsJR0RSI9EpBqrKAwTP6rQAegFjzWyGu8+JqjcYGAj0JOiA8BbB\nGGwPmVk2MAa4B/g3cAUwxsw6unupmTUjmNTt98CLBD3oWif9yKqp6KkJ3J27x8/jwcIv2b5jF3cN\n6kGdDCtnDyIi313KzmzC0aEHAbe4e7G7TwJeAS6KUf0SYLi7L3H3pcBw4NKwrD9BkrzH3be5+32A\nASeH5dcA4939mbB8k7vPTdqB7WfMjOsHdOa3p3TkhalLuPb5T9mxUw9+ikhyJXRmY2bR3Z3LOLDV\n3VcnsJtOwA53nx+xbgZwYoy6XcOyyHpdI8pm+p43HWaG68cBRwOzzOwD4BDgI+A37v5N9IeY2WCC\nsyiaN29OYWFhAodRM/TOgkEds3jp02UsXbGSK3rUJTPGGU5xcXGtapdEqV32pjaJTe0SSPQy2teU\n8zyNmW0EHgOuL2com1xgY9S6IqBhnLpFUfVyw/s20WXR+2kNHA6cBswC7gb+SzDZ2x7cfSQwEqBz\n587ev3//OKHXTP37w2ETv+KvY+eSl9+Q+y/oTd3MPTsUFBYWUtvaJRFql72pTWJTuwQSvYx2PrAE\nuJngl/hp4etvgF8QjA59EXBLOfsoBhpFrWsEbEqgbiOgODybqWg/JcBod//E3bcCtwHHmlnjcmKr\ntS7r14HbftSVtz5byRVPTWXr9p3pDklEaqBEk82VwO/d/U53nxAudwLXAr9w93uBqwmSUjzzgUwz\n6xixricQ3TmAcF3POPXmAD0ie6cBPSLKZ7LnWZi6XFXgkmPb8bezu/Pu/NVc9sQUSkqVcESkaiWa\nbI4iuCQVbTbQN3z9IeX0+nL3zcAo4HYza2BmxwFnAU/FqP4kcI2ZFZhZK4Kk9nhYVgjsBK42s7pm\ndlW4fkL48zHg7LB7dBbB2dYkd9cAouW44Kg2DDu3Jx98uYZLH/uYzds0sLeIVJ1E79ksIriRPiRq\n/eUEl9IAmgPrKtjPr4FHgVXAWuBKd59jZv2AN9w9N6w3AujAtwnu4XAdYffmgeG6u4C5wEB3Lw3L\nJ5jZH4GxQH1gEhDzeR7Z07l9WpNVx7jm+Rn88L6JbNuxi+VFWymYPEGjRYvId5JosrkWeMnMfkA4\nvw1wBMEgnYPC932B58vbibuvI3h+Jnr9RIIb/2XvHbg+XGLtZzrQp5zPeRB4sLxYJLazehXw6Tfr\neeyDRbvXabRoEfmuErqM5u5jgY4Ez8U0CpdXgM7u/npY59/ufk2yApXUefOzVXutKxstWkRkXyQ8\ngoC7LwZuTGIsUk3EGxVao0WLyL5KONmYWX2CIWYOIOqMyN1HVXFckkat8nJYGiOx5DfITkM0IlIT\nJHQZzcxOJegkMImgR9mLEcsLSYtO0iLWaNFmsHZzKU99+HVaYhKR/VuiXZ/vJejd1drdM6KWvcew\nl/1a5GjRAAV5Odx1dndOOfQAbhkzh7+9PlcjRotIpSR6Ga0d8CN3X5bEWKQaKRstOnKojXOPOIjb\nXp3DyPe+YvG6LfzzJ72oF2O+HBGRaIme2bwPdE5mIFL91ckwbvtRV27+4WGMm7OC8/9vMmuLt6U7\nLBHZDyR6ZvMQ8Pfwaf5ZwPbIQnefVtWBSfVkZlzWrwOtm+Tw22c/5ex/f8BjP+/Lwc1zK95YRGqt\nRM9sXgQOJRgh+UNgSsTySTnbSQ11ercDeXbw0WzetoNz/v0BH321Nt0hiUg1lmiyaV/O0iE5oUl1\n17tNE0b/+jia5mZz0SMfM+bTpekOSUSqqYQuo7n7ooprSW3Upml9Rl15LFc8NZXfPvspb85ZwaeL\nN7Bsw1Za5eVoTDURAcpJNmZ2DvCqu28PX8elhzprt7z62Tz5yyO5YORkxs5asXu9xlQTkTLlndm8\nCLQkGKH5xXLqOaD+r7Vc3cw6rNi4da/1ZWOqKdmI1G5xk427Z8R6LRLPsg17J5tgvcZUE6ntlESk\nyrQKRxyI1ignk2DWCBGprRJONmbW2swuMLPfmdk1kUsyA5T9R6wx1TIMikp28Jv/TKNYs3+K1FoJ\n9UYzswsJZtjcAawmuE9TxoF/VH1osr8puy8zbPw8lm0ooVVeDtd9rxOrNm3j7vHz+HzFJB76WR86\ntWiY5khFJNUSHUHgdmA4cIu770xiPLKfKxtTLVrPg/K46j/TOev+97lrUHfO6qUOAyK1SaKX0VoA\nDyvRyL46ukNTXr/6eLoVNOK3z37Kn8fMpnTHrnSHJSIpkmiyeR04KpmBSM13QKN6/Ofyo7m8X3ue\n+HAR5434UD3VRGqJRC+jvQUMNbOuxB6IUw91SkKy6mRw0w+7cHibJgx5cSY/vG8i953fm34dm6c7\nNBFJokSTzYjw5x9jlOmhTqm073c/kM4tG3Ll09O4+NGPOb1rS2Ys2cByDXMjUiMldBktxuycmqlT\nvrMOzXMZ/Ztj6dMmjzdmr2DZhq043w5z8/J0DewpUlNUmGzMLMvMPjIzTZ4mVa5+dibLi+IPcyMi\nNUOFycbdtxNMJaBHwCUpNMyNSM2XaG+0J4DLkxmI1F7xhrnJyDCmLlqf4mhEJBkSTTYNgMFm9qmZ\nPWJm90UuyQxQar5Yw9xkZ2bQsG4mP37oA4aN/1zP5Ijs5xLtjXYYMC18HT0zpy6vyXcSa5ibIQM6\nc8phB/CX1z7jgXe+5J3PV/PPn/Sic0sNdSOyP0p0ps6Tkh2I1G7xhrm5+9yenNalJTeOmsmZ/5rE\ndQM68cvjO1Anw9IQpYjsK00xINXeaV1aMP53J3DSoc352+ufc/7IySxetyXdYYlIJVRmioGTzGyk\nmY0zswmRSyX2kW9mo81ss5ktMrML4tQzMxtqZmvDZaiZWUR5LzObamZbwp+9Yuwj28zmmtmSROOT\n6qtpbl0e+lkfhv+4J3OXb+T0e97j2Y+/YfS0JRx31wTa3zCW4+6aoGdzRKqphJKNmV0KvAE0BPoT\nTDPQBDgc+KwSn/cAUEowsOeFwIPhEDjRBgMDgZ5AD+BM4IowlmxgDPB0GMMTwJhwfaQhYZxSQ5gZ\ng/q05o3f9aNH6zxuGDWLa1+YwdINJXoYVKSaS/TM5jrgKnc/n2BctBvdvTfBL/ziRHZgZg2AQQTT\nFBS7+yTgFeCiGNUvAYa7+xJ3X0owvcGlYVl/gntN97j7Nne/DzDg5IjPag/8DLgzweOT/UjrJvV5\n5rKjaJyTya6o7il6GFSkekq0N1oH4O3w9TYgN3x9P1AI3JDAPjoBO9x9fsS6GcCJMep2Dcsi63WN\nKJvpe84zPDNcPy58/y+CcdzKfSrQzAYTnEXRvHlzCgsLEziM2qW4uLjatktRSeyZP5duKEl6zNW5\nXdJFbRKb2iWQaLJZS3AJDWAp0I3gF3xTIPYTeXvLBTZGrSuK2G903aKoernhfZvosj32Y2ZnA3Xc\nfbSZ9S8vIHcfCYwE6Ny5s/fvX271WqmwsJDq2i4FkyewNMYoAw3rZnL0cf2ol5W8Yfuqc7uki9ok\nNrVLINHLaBOB74WvnwfuM7NnFl5oAAAW10lEQVTHgP8STD+QiGKgUdS6RsCmBOo2AorDs5m4+wkv\n1d0NXJ1gTLIfi/UwaB0zNm3bwYB73qNw3qo0RSYi0RJNNlcRJBYI7oMMIzireR64LMF9zAcyzaxj\nxLqewJwYdeeEZbHqzQF6RPZOI+hEMAfoCLQDJprZCmAUcKCZrTCzdgnGKfuJgb0LuPOc7hTk5WBA\nQV4Ow8/ryTOXHUUdMy597BN+88w0VsQY6FNEUivRhzrXRbzeBQyt7Ae5+2YzGwXcbmaXAb2As4Bj\nY1R/ErjGzF4nGKHgWoL7MBDcI9oJXG1mD/HtmG0TgF3AQRH7OZbgvtLhqGdajRTvYdA3ftePke9+\nxf3vfEHhvFVc873OXHJMWzLr6NEykXSozHM2LczsOjN70MyaheuOC3t+JerXBPd4VhGcKV3p7nPM\nrJ+ZRfZqGwG8SjAr6GxgbLgOdy8l6BZ9MbAB+AUw0N1L3X2Hu68oW4B1wK7w/c5KxCn7ubqZdfh/\np3Tkrd+fSN/2+fzltc848/73mfaNBvYUSYeEzmzMrA/wP2AhQa+vYcAa4DSCXmYxH86MFp4hDYyx\nfiLf9nAjvDdzfbjE2s90oE8Cn1cItE4kNqmZ2jStz2OX9mXc7BXc9upnDHrwA37atw3dCxrxwDtf\n7jEWm2YGFUmeRHuj/R24193/bGaRN/THAz+v+rBEqo6Z8f3uB9KvU3PueWs+j0xauPsGJHz7MCig\nhCOSJIleRutD8KR+tOUEowGIVHu5dTO5+YwuNG9Yd68yPQwqklyJJpsSgqFhoh1KcP9FZL+xetO2\nmOs1M6hI8iSabMYAfzazsj8JPexKPBR4KQlxiSRNvJlBHfjj6Fms2qSu0iJVrTJjo+UTdB+uD0wC\nviB4cv/m5IQmkhyxHgatl5VBv47NeP6TxfQfVsg/35rP5m2xh8MRkcpL9DmbjcDxZnYywTMrGcA0\nd3+7/C1Fqp94M4MO7F3A12s2M+zNedz7vwU889E3/O7Ujvyk70Fk6fkcke8k0d5oALj7BIKHJwEw\ns7bAMHc/r6oDE0mmeA+DtmvWgAcuOJzLjl/Pna9/zs0vz+bR9xfyh9MP5XtdWjDm02UMGz+PpRtK\nKJg8QV2mRRJUqWQTQx7BtAEiNUrvNk147oqj+d/cVdw17nOueGoq7ZvWZ1nRVrbt2AWoy7RIZeja\ngEgcZsapXVow7rf9uPOc7ixat2V3oimjLtMiiVGyEalAZp0Mzj+yDe6xy9VlWqRiSjYiCYrXZbpu\nZgYzFm9IcTQi+5dy79mY2SsVbB89r4xIjTVkQGduHDWLku3fjumamWGYwVkPvM8JnZpz9cmHcES7\n/DRGKVI9VdRBYG0C5QurKBaRai2yy/TSDSUUhF2mT+3Sgqc+XMTDE7/i3Ic+5OgO+Vx9ckeOObgp\ne067JFJ7lZts3F2DbIpEKOsyHT3V75X9D+aSY9vy348XM+LdL7ng4Y/o07YJV518CP07Nd/dZVqj\nTEtt9V27PotIqH52Jr88vj0XHtWGF6Ys5sHCL/n5Y59wUJMcVm7cSunOoIeBukxLbaQOAiJVrF5W\nHS46ph2FQ05i6KDuLCv6NtGUUZdpqW2UbESSJDszg5/0bcOuXbH7TKvLtNQmSjYiSVbeKNPXPj+D\n2UuLUhuQSBoo2YgkWaxRputmZnD8IU15Y/ZyzvjXJM576EPemLWcHTt3xdmLyP5NHQREkqy8UaaL\nSrbzwpTFPP7B11z5zDQK8nK45Ni2/OSINjSun8XL05eqF5vUCEo2IikQb5TpxjlZXNavAz8/rj1v\nfbaSx95fyN9e/5x/vrWAw9s0ZsqiDRr4U2oEJRuRaqBOhnF6t5ac3q0lc5YV8fj7X/PC1CV71Svr\nxaZkI/sb3bMRqWa6tmrMsB/3JN7YA+rFJvsjJRuRaqq8Xmw/fugDXpiymC2lmrpa9g9KNiLVVKxe\nbPUyMzizx4GsLS5lyIszOfKO/3HjqJlM/2Y9Hm8OBJFqQPdsRKqp8nqxuTtTFq3nuU8W8/L0Zfz3\n48V0apHLeUccxDmHt+a9+avVi02qFSUbkWosXi82M6Nvu3z6tsvnz2d24bWZy3nuk8X8dexc/vb6\nXADKBi5QLzapDnQZTWQ/17BeFucf2YaXf3Mc4393AjnZdYgeIadk+07uHv95egIUQclGpEbp3LIh\nW7btjFm2bMNW7nx9LrOXFun+jqRcSpONmeWb2Wgz22xmi8zsgjj1zMyGmtnacBlqEbNQmVkvM5tq\nZlvCn70iyoaY2Wwz22RmC81sSCqOTaS6iNeLrV5mBo9MWsgZ/5rEycPf5R9vzuOLVZv2qPPy9KUc\nd9cE2t8wluPumsDL05emImSpBVJ9z+YBoBRoAfQCxprZDHefE1VvMDAQ6EnQ0/MtghlBHzKzbGAM\ncA/wb+AKYIyZdXT3UsCAi4GZwMHAm2a22N2fTfrRiVQDsaavzsmqw53ndKd/5+aMm72CV2cu4/53\nvuC+CV9waMuGnNmzFfUyM/j7m/N3b6d7PVKVUpZszKwBMAjo5u7FwCQzewW4CLghqvolwHB3XxJu\nOxy4HHgI6B/GfY8H1wLuM7PrgJOBce5+d8R+5pnZGOA4QMlGaoXyerEB/PTINvz0yDas2rSV12cu\n59WZy+POraMRC6SqWKqu3ZpZb+B9d68fse464ER3PzOqbhHwPXf/KHx/BPCOuzc0s9+HZd+PqP9a\nWD48aj8GTANGuPtDMWIaTHAWRfPmzfs8//zzVXS0NUdxcTG5ubnpDqPaqWntsnrLLoa8F39kgscG\n1CfiSnZMNa1NqkpNb5eTTjppqrsfUVG9VF5GywU2Rq0rAhrGqVsUVS83TB7RZeXt51aC+1KPxQrI\n3UcCIwE6d+7skXPKS6CwsBC1y95qYrvcM3MCS+MMhfPHybs49bAWnNqlBcd0aEp25t63e2tim1QF\ntUsglcmmGGgUta4RsCmBuo2AYnd3M0toP2Z2FcG9m37uvu27BC5SG8S611MvK4OzexewbnMpL05d\nwlOTF5FbN5MTOzfne11a0L/TAbwzbxXDxs9j6YYSCiZP0AOkElMqk818IDO8kb8gXNcTiO4cQLiu\nJ/BxjHpzgGvNzPzba4A9CDofAGBmvyC4D3RC2X0fESlfRfd6tm7fyQdfruGtz1by1merGDtzOQaY\n6QFSqVjKko27bzazUcDtZnYZQW+0s4BjY1R/ErjGzF4nnD0X+FdYVgjsBK42s4cIOg4ATAAwswuB\nvwEnuftXSTockRop3ogFAPWy6nDyoS04+dAW3DHQmbFkAxc98jHF2/YcDLRk+07+8tpnnNalBQ3q\napASCaT6m/Br4FFgFbAWuNLd55hZP+ANdy+7izYC6ADMCt8/HK7D3UvNbGC47i5gLjAw7PYM8Feg\nKfBJxA3Np939V0k9MpFaJCPD6N2mCZu3xR51eu3mUnrd/iZ92jahX8fmnNCxOV1bNSIj49tOBpqF\ntHZJabJx93UEz89Er59IcOO/7L0D14dLrP1MB/rEKWtfJcGKSIVa5eXE7FTQLDebc/sctHtA0GHj\n55HfIJvjD2lGv47N2Fy6g6FvzNMzPbWIznFFZJ/Fe4D05h92YWDvAm74/qGs3rSN979Yw3sLVjNx\nwRpembEs5r70TE/NpmQjIvssslPB0g0lFMS4HNa8Yd3d94LcnXkrN3H6PRNj7m/phhLe/mwlfdvl\n07h+VkqOQVJDyUZEvpOyRJLI8yRmxqEtG1EQ5/IbwGVPTsEMDm3ZiKPa53NU+3z6ts+nWW5dQPd6\n9ldKNiKScvEuv91+Vlfa5Nfno4Xr+HjhOp77ZDGPf/A1AIcckMsBDbP55Ov1bN8Z9LXWvZ79h5KN\niKRcRc/0HNWhKQClO3Yxa2kRHy9cx0cL1/LuvNVED7BVsn0nd4ydyw+6HxhzZAOpHpRsRCQtynum\np0x2ZgZ92jahT9smXNn/YNrfMDZmvdXF2+h263i6FzSm90F59GqTR+82TWjVuN7uMd10+S29lGxE\nZL8Rr6t1k/pZnNunNdO/2cBTkxfx8KSFABzQsC69DsojOzODNz9bSemOXYAuv6WDko2I7Dfi3ev5\n85lddyeN7Tt38fnyTUxfvJ7p32xg+jfr+Xrtlr32VbJ9J38dq5EOUkUtLCL7jYru9QBk1cmge+vG\ndG/dmIuPCda1v2HsXvd6ANYUl9Lt1vG0b9aAbq0a07VVI7oVBD/z6mcDuvxWVZRsRGS/ksi9nmjx\nLr81bZDNxce0Y/ayIqYuWr/HA6cFeTnkN8hi7vJN7Nil3m/flZKNiNR48S6/3XJGlz2SxrrNpcxZ\nVsScZRuZvbSIN2avYOeuPc+JSrbv5KbRsyjetoPOLRvS6YCGMR9ALTsj0tQLASUbEanxErn8BpDf\nIJt+HZvTr2NzgLi93zaX7uTml2fvft+yUT06t2wYJJ8WDVleVMID73zB1u3qkFBGyUZEaoWqvPxW\nkFeP5391LPNXbGLeyk3MWxEsH361dnePt2hlzwOd1PmAcofiqan3iJRsRETiiHf5bciAQynIy6Eg\nL4eTDj1gd9mOnbtYtG4Lpwx/N+b+Vhdvo+ftb9IsN5sOzXI5+IAGe/yctmgdN708p0aOhq1kIyIS\nR6KX38pk1sng4Oa5ccd+y2+Qza9O7MCXqzbz1Zpixs9ZybrNi8uNoWT7Tu58Yy4/6tlqj/mAolX3\nMyIlGxGRcuzL5bd4Z0R/iuqQALB+cylfrSnmy9Wbuf7FmTH3t3LjNg69ZRyt83No17QBbfLr07Zp\nsLTJb8Cn36znljHV+4xIyUZEpIolMvVCmSYNsunTIJ8+bfO59+0FMc+I8nKy+Enfg1i0dguL1m3h\no6/Wsrl05171IpU9tHpEuya0bFSPzDqxx41L1RmRko2ISBJUZuqFMvHOiG79Udc9EoC7s3ZzKYvW\nbmbR2i1c8/yMmPtbU1zK8UPfoU6G0bJRPQqa5NC6SQ6t83IoaJLDorVbeGTSQrbtwzA+ZUkqu+Uh\nMWdNjqZkIyJSTSR6j8jMaJZbl2a5denTNp/hb86P+9DqdQM6s3R9CUs3lLBk/RYmf7mWFRu3sivW\nkAp8+xzR8qKtHNi4Hi0b1+PAxvVo0age9bLqAEGiiU6KFVGyERGpRqryHlH0Q6tltu/cxYqirfS7\n+52Y+9tcupOh4z7fa31+g2xaNqrHV6uL2Rqni3c8SjYiIvu5yvaay6qTwUH59eP2mivIy+HN35/A\nio1bWVG0leVFW1lRVBL+3MpnyzdWOkYlGxGRGqAqz4iGDOhMg7qZHNw8l4Ob5+613XF3TYg7rXc8\nmtZORKSWGti7gDvP6U5BXg5GcEZz5zndK0xaQwZ0Jie8f5MondmIiNRi+3JGFHnZbnmC2+jMRkRE\nKm1g7wLev+FkSld8MTWR+ko2IiKSdEo2IiKSdEo2IiKSdEo2IiKSdEo2IiKSdClNNmaWb2ajzWyz\nmS0yswvi1DMzG2pma8NlqJlZRHkvM5tqZlvCn70S3VZERFIv1Wc2DwClQAvgQuBBM+sao95gYCDQ\nE+gBnAlcAWBm2cAY4GmgCfAEMCZcX+62IiKSHilLNmbWABgE3OLuxe4+CXgFuChG9UuA4e6+xN2X\nAsOBS8Oy/gQPo97j7tvc/T7AgJMT2FZERNIglSMIdAJ2uPv8iHUzgBNj1O0alkXW6xpRNtPdIwfI\nnhmuH1fBtnsws8EEZ0IA28xsdmKHUqs0A9akO4hqSO2yN7VJbDW9XdomUimVySYXiB4qtAhoGKdu\nUVS93PDeS3RZ9H7ibhuVoHD3kcBIADOb4u5HJH44tYPaJTa1y97UJrGpXQKpvGdTDDSKWtcI2JRA\n3UZAcZgsKtpPeduKiEgapDLZzAcyzaxjxLqewJwYdeeEZbHqzQF6RPUw6xFVHm9bERFJg5QlG3ff\nDIwCbjezBmZ2HHAW8FSM6k8C15hZgZm1Aq4FHg/LCoGdwNVmVtfMrgrXT0hg2/KMrPxR1Qpql9jU\nLntTm8SmdgEslVeXzCwfeBQ4DVgL3ODu/zGzfsAb7p4b1jNgKHBZuOnDwB/KLoWZWe9wXRdgLvBL\nd5+eyLYiIpJ6KU02IiJSO2m4GhERSTolGxERSbpan2wSHa+ttjGzQjPbambF4TIv3TGlmpldZWZT\nzGybmT0eVXaKmX0ejs/3jpkl9GBbTRCvXcysnZl5xHem2MxuSWOoKRV2WHok/D2yycw+NbPvR5TX\n2u8MKNlA4uO11UZXuXtuuHROdzBpsAz4K0Gnlt3MrBlBz8pbgHxgCvBcyqNLn5jtEiEv4nvzlxTG\nlW6ZwGKCUVEaAzcDz4dJuLZ/Z1I6gkC1EzFeWzd3LwYmmVnZeG03pDU4STt3HwVgZkcArSOKzgHm\nuPsLYfmtwBozO9TdP095oClWTrvUauHjHbdGrHrNzBYCfYCm1OLvDOjMJt54bTqzCdxpZmvM7H0z\n65/uYKqRPcbfC3/JfIm+N2UWmdkSM3ss/Iu+VjKzFgS/Y+ag70ytTzaVGa+ttvkD0AEoIHgo7VUz\nOzi9IVUbFY3PV1utAfoSDMzYh6A9nklrRGliZlkEx/5EeOZS678ztT3ZVGa8tlrF3T9y903hNA5P\nAO8DP0h3XNWEvjcxhFOHTHH3He6+ErgK+J6Z1ZpfqABmlkEwMkopQRuAvjO1PtlUZry22s4J5g2S\nqPH3wnt/B6PvTbSyJ8Zrze+ZcASTRwg6HA1y9+1hUa3/ztSaL0EslRyvrdYwszwzG2Bm9cws08wu\nBE4gmC+o1giPvR5QB6hT1h7AaKCbmQ0Ky/9EMMdSrbjRG69dzOwoM+tsZhlm1hS4Dyh09+jLRzXZ\ng8BhwJnuXhKxvlZ/ZwBw91q9EHRDfBnYDHwDXJDumNK9AM2BTwhO8TcAk4HT0h1XGtrhVoK/ziOX\nW8OyU4HPgRKCwWHbpTvedLcLcD6wMPy/tJxgUNyW6Y43he3SNmyLrQSXzcqWC2v7d8bdNTaaiIgk\nX62+jCYiIqmhZCMiIkmnZCMiIkmnZCMiIkmnZCMiIkmnZCMiIkmnZCNSQ4Vzy5yb7jhEQMlGJCnM\n7PHwl330MjndsYmkQ62ez0Ykyd4mmBspUmk6AhFJN53ZiCTPNndfEbWsg92XuK4ys7HhNMGLzOxn\nkRubWXcze9vMSsxsXXi21DiqziVmNiuconmlmT0RFUO+mb0QTnv+VfRniKSKko1I+twGvAL0Ipgz\n6Mlw9suyUYHHE4ytdSRwNnAsEVMxm9kVwAjgMaAHwRQQs6M+40/AGIIRh58DHjWzNsk7JJHYNDaa\nSBKY2ePAzwgGZYz0gLv/wcwceNjdL4/Y5m1ghbv/zMwuB/4OtHb3TWF5f+AdoKO7f2FmS4Cn3T3m\nFObhZ9zl7jeG7zMJJgsc7O5PV+HhilRI92xEkuc9YHDUug0Rrz+MKvsQ+GH4+jCCIegjJ9f6ANgF\ndDGzjQSzqP6vghhmlr1w9x1mtho4ILHwRaqOko1I8mxx9y+SsN/KXI7YHvXe0eVzSQN96UTS5+gY\n7+eGr+cC3aOmVD6W4P/sXHdfBSwFTkl6lCJVQGc2IslT18xaRq3b6e6rw9fnmNknBBNpnUuQOI4K\ny54h6EDwpJn9CWhC0BlgVMTZ0h3AP81sJTAWqA+c4u7Dk3VAIvtKyUYkeU4lmLEy0lKgdfj6VmAQ\nwfTJq4Gfu/snAO6+xcwGAPcAHxN0NBgD/LZsR+7+oJmVAtcCQ4F1wOvJOhiR70K90UTSIOwp9mN3\nfzHdsYikgu7ZiIhI0inZiIhI0ukymoiIJJ3ObEREJOmUbEREJOmUbEREJOmUbEREJOmUbEREJOn+\nP1NmGawFB/rvAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pe2Ys1YLRwND",
        "colab_type": "text"
      },
      "source": [
        "The schedule function can take the current learning rate as a second argument:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPL_opb6RwND",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def exponential_decay_fn(epoch, lr):\n",
        "    return lr * 0.1**(1 / 20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vWCHbBg5RwNG",
        "colab_type": "text"
      },
      "source": [
        "If you want to update the learning rate at each iteration rather than at each epoch, you must write your own callback class:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eyNZMja8RwNG",
        "colab_type": "code",
        "outputId": "c784ac38-28c2-4016-e132-311bd513bf28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "K = keras.backend\n",
        "\n",
        "class ExponentialDecay(keras.callbacks.Callback):\n",
        "    def __init__(self, s=40000):\n",
        "        super().__init__()\n",
        "        self.s = s\n",
        "\n",
        "    def on_batch_begin(self, batch, logs=None):\n",
        "        # Note: the `batch` argument is reset at each epoch\n",
        "        lr = K.get_value(self.model.optimizer.lr)\n",
        "        K.set_value(self.model.optimizer.lr, lr * 0.1**(1 / s))\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        logs = logs or {}\n",
        "        logs['lr'] = K.get_value(self.model.optimizer.lr)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "lr0 = 0.01\n",
        "optimizer = keras.optimizers.Nadam(lr=lr0)\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
        "n_epochs = 25\n",
        "\n",
        "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
        "exp_decay = ExponentialDecay(s)\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid),\n",
        "                    callbacks=[exp_decay])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 9s 168us/sample - loss: 0.8122 - accuracy: 0.7642 - val_loss: 0.6646 - val_accuracy: 0.7872\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.6513 - accuracy: 0.8033 - val_loss: 0.6050 - val_accuracy: 0.7960\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 10s 175us/sample - loss: 0.5594 - accuracy: 0.8241 - val_loss: 0.6042 - val_accuracy: 0.8204\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 9s 171us/sample - loss: 0.5343 - accuracy: 0.8340 - val_loss: 0.6596 - val_accuracy: 0.8074\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 9s 162us/sample - loss: 0.4716 - accuracy: 0.8506 - val_loss: 0.5701 - val_accuracy: 0.8484\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 9s 162us/sample - loss: 0.4443 - accuracy: 0.8578 - val_loss: 0.4518 - val_accuracy: 0.8538\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 9s 164us/sample - loss: 0.3922 - accuracy: 0.8703 - val_loss: 0.4749 - val_accuracy: 0.8538\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 9s 162us/sample - loss: 0.3656 - accuracy: 0.8788 - val_loss: 0.4911 - val_accuracy: 0.8640\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.3424 - accuracy: 0.8857 - val_loss: 0.6337 - val_accuracy: 0.8598\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 9s 161us/sample - loss: 0.3225 - accuracy: 0.8921 - val_loss: 0.4504 - val_accuracy: 0.8642\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.2973 - accuracy: 0.9007 - val_loss: 0.4547 - val_accuracy: 0.8654\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 10s 175us/sample - loss: 0.2769 - accuracy: 0.9056 - val_loss: 0.4345 - val_accuracy: 0.8836\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 9s 165us/sample - loss: 0.2502 - accuracy: 0.9134 - val_loss: 0.4568 - val_accuracy: 0.8802\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 9s 158us/sample - loss: 0.2367 - accuracy: 0.9187 - val_loss: 0.4483 - val_accuracy: 0.8898\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 9s 159us/sample - loss: 0.2157 - accuracy: 0.9246 - val_loss: 0.4847 - val_accuracy: 0.8862\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 9s 170us/sample - loss: 0.2021 - accuracy: 0.9294 - val_loss: 0.4458 - val_accuracy: 0.8950\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 10s 181us/sample - loss: 0.1887 - accuracy: 0.9347 - val_loss: 0.4876 - val_accuracy: 0.8878\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 9s 161us/sample - loss: 0.1750 - accuracy: 0.9393 - val_loss: 0.5012 - val_accuracy: 0.8912\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 9s 159us/sample - loss: 0.1630 - accuracy: 0.9441 - val_loss: 0.4597 - val_accuracy: 0.8938\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.1524 - accuracy: 0.9483 - val_loss: 0.4952 - val_accuracy: 0.8932\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 10s 184us/sample - loss: 0.1434 - accuracy: 0.9511 - val_loss: 0.5246 - val_accuracy: 0.8952\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 9s 167us/sample - loss: 0.1334 - accuracy: 0.9541 - val_loss: 0.5431 - val_accuracy: 0.8950\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 13s 237us/sample - loss: 0.1238 - accuracy: 0.9581 - val_loss: 0.5884 - val_accuracy: 0.8962\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 9s 159us/sample - loss: 0.1162 - accuracy: 0.9613 - val_loss: 0.5765 - val_accuracy: 0.8924\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.1065 - accuracy: 0.9645 - val_loss: 0.5844 - val_accuracy: 0.8974\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "llTw-NcFRwNI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "n_steps = n_epochs * len(X_train) // 32\n",
        "steps = np.arange(n_steps)\n",
        "lrs = lr0 * 0.1**(steps / s)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "NrxFRbs7RwNK",
        "colab_type": "code",
        "outputId": "6240cb2f-9667-4a60-fd40-c99af57925c5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "source": [
        "plt.plot(steps, lrs, \"-\", linewidth=2)\n",
        "plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
        "plt.xlabel(\"Batch\")\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\n",
        "plt.grid(True)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd8FHX6wPHPk94TktA70ntTVEDA\nhnp6Yj/xVEQP6/kTRU89PbseKtZTxHJiPcWCKOqhHgQEFQGRXqSH0BNISEgIhOf3x0xwWTbJJmSz\nKc/79ZpXduf7nZlnJrvz7Mx85zuiqhhjjDGBFBLsAIwxxtR+lmyMMcYEnCUbY4wxAWfJxhhjTMBZ\nsjHGGBNwlmyMMcYEnCUbU+OJyAgRyS3nNGki8q9AxeQuY4OIjAnAfC8WkXLds+C9jSqyzY6FiDwg\nIv+uquX5WL6KyMVBWG6Z21lEbhaRL6oqpmCxZFODichE90vkPfwU7NgCpYSdxodAmwAs6zoRWSgi\nuSKSLSKLReTRyl5OkARkm/kiIg2AO4Aave1E5EERWRqAWb8O9BGRgQGYd7URFuwAzDH7DrjSa1xh\nMAIJFlXNB/Irc54iMhJ4ARgN/A8IB7oCJ1XmcoIlENusFNcBP6vqukAvSETCVfVAoJdTmVR1v4i8\nD9wKfB/seALFjmxqvv2qus1ryAIQkUEickBEBhdXFpHrRSRHRNq479NE5BUReV5EdrvDUyIS4jFN\nPRF5yy3LF5HvRKSLR/kI99f/aSKyVETyRGSGiLT2DFREzhORBSJSICLrReQxEYnwKN8gIveJyAQ3\nxs0icqdnufvyI/cIZ4Pn8j3qHSciU0RkmxvLLyJybjm36x+BT1V1gqquUdUVqvqRqt7utU7niMhc\nd7tkisgXIhLlUSWqpPVxp08UkVdFZIeI7BWRmSLS16vOVSKyUUT2ichUoKFX+VG/uMs6feNjmz3o\n/u/+JCJr3Vg+E5FUjzphIvKsx+fkWREZLyJpZWzL4cARp4n8/NxFiMhYd7vtE5F5IjLUo3yw+zk4\nR0R+FpFCYCglayQiX7rz2igif/aK6Z8issr9X24QkSeL/5ciMgJ4AOgiv59BGOGWJbrbYav72V4h\nIpd5zbvU7wbwOfBHEYkpY1vWXKpqQw0dgInA1DLqPA6kA/WAjkAecLVHeRqwF3jRLb8UyAZu96gz\nBVgJnAJ0w/lipAPRbvkI4ADOUdYJQHdgITDNYx5DgRzgGuA4YAiwCnjao84GIBO4BWgL/BVQ4CS3\nvL77/jqgEVDfY/m5HvPpAdzgxtoW+DvO0V5Hr/X+Vynb7RVgNdCmlDpnAQdxTg91dtd7DBDj5/oI\nMBv40t1ubYFH3O3U2K3TDzjkrkN74Hp3nuoRx4PAUq/YvLdJWe8fBHKBye56nARsBCZ41Lkb2A1c\nBHQAnnc/K2mlbKNkN/7+XuPTKPtz9x7wE87nro27HQuBHm75YHd7LgHOdOvULyEOdbfb9e52/Lsb\nV1+POvcD/YFWwDnAJuARtywaeBrne9DIHaLd/+EcYLn7eWgDnA1c4O93w60XAxQBpwV7vxKoIegB\n2HAM/zwn2Rx0dxKew1iPOuHAPOBT4BfgQ695pOHsVMVj3H3AZvd1O/eLeopHeaK7Y7jOfT/CrdPB\no84VwP7i+QKzgPu9lj3Mjbe4zgbgP151fgPu83ivwMVedUbgseMsYVv95DWfNEpPNo2BH93l/Qa8\nC1wFhHvUmQN8UMo8Sl0f4FR3/aO96vwK3OW+fh/41qv8dQKTbAqARI9xfwfWeLzfCtzt8V5wfjCk\nlbINerrbsHU5P3fH4SSDFl7TfQa87L4e7M77Ij++Kwq85jXuO+DdUqa5wWv9fW3nM9w4O5UwjxGU\n8d3wGJ8FXFvWutTUwU6j1XyzcL7QnsNTxYXqnL8eDpwLNMD5ZeftJ3U/7a4fgaYikgB0wvky/egx\nz2ycX5OdPabZr6qrPN5vASJwjqgA+gB/d0+35bqncN4HYnF+JRZb7BXbFjduv4lIrHsKZLl7eiYX\n6Au08HceqrpVVU/COTp6DmfHOgH42eNURy+c6zmlKW19+uD8ot3ptV264uxswdn+P3rNw/t9Zdno\n/m+PilVEEnH+Tz8XF7qfmZ8pXbT7t8BHWWmfu94423y517b5A79vm2Lzy4jBc/7e7w9/hsVp5Tfb\nPf2aCzxL2Z+ZXsBWVV1RSp2yvhvF8vl9e9U61kCg5tunqmvKqHMizvW5JJxTUXsqadmeO4qDJZSF\nePx9CPjIx3x2erz2vrirlP/a4tM4pzTG4BxJ7APexvmCl4uqLgWWAi+JyACcC7iX4hxV+qO09QkB\ntgO+WiHllCPMQzg7Zk/h5Zi+WGVse2+73L/1cI6M/BXiLv94H3F5N2zIq1hovxORE4EPcD6jo3G+\nI3/E+Swdq7K+G8WSOfK7UKvYkU0t516I/BdwM/At8K6IeP/I6CcinjurE4EtqpoDrMD5nBxuheX+\n8uyGc57aX7/gXDNZ42Pw/jKW5gAQWkadAcDbqvqJqi4GNnP0r+GKKF7fOPfvQuC0Y5jfLzgX+w/5\n2CY73DorcP4fnrzf7wQaev0Pex5DXEdxj3i24ez8AXCXd3yJEznW4iTOzj7KSvvcLcRJoI18bJuM\nCq6Gr+1YfETSH8hQ1UdUdZ6q/ga09KpfyNGfvYVAYxHpVMGYAKdRCxCF85molezIpuaLFJFGXuOK\nVHWniIQC7wAzVXWCiHyMc/rrAZyLocWaAM+JyMs4SeRO3HsiVPU3EZkCTBCRUTi/+B7D2YG8X444\nHwamishGYBLOr72uwAmqelc55rMBOE1EZuKcntjto85q4AI37gM46xvlo16JRGQ8zumO6TjJqjHO\nNYV9wDdutceAL0RkDc62EJwL1RNUdZ8fi/kO57rPFBG5i98vPp8FfKeq3+M0v/5BRO4BPsa5TnGB\n13zScH4V3ysiH7h1AnED4/PAXSKyGifxXo+zXUo8YlHVQyLyHc4PgI+9ikv73K0WkfeAiSJyB85O\nOBln3dap6qcViP9CEZmHs70uxvmh0M8tW41zCu8KnNNrQ4HLvabfALQUkd44jQf24pxGnQt8IiKj\n3fm0BWJV9bNyxDbQXa/fKrBeNYId2dR8p+N82T2HhW7ZvTgf/GsBVDUTuBq42z0lVOw9nF9sc4HX\ngDdwzlcXuwbn3Pzn7t8Y4Cx17tXwi6pOwznfPsSdx884rZs2+b+qgHNz4BCc1nALS6hzO7AD55TX\n1ziNA8p7/8K3ODuiSTg7kMnu+DNUdTWAqn6Fs+M/241lphvbIX8W4F6vOAcnob2Gc7F9Ek5Lry1u\nnZ9w/n834lz/uRDnQrXnfFa45aPcOmfgtEKsbE/j/Hh5E2ebgrNdfF2P8fQqcJn748eTP5+7N4En\ncRLxVJyWaRsrGP+DOC3pFuNsr2tUdR6Aqn6Bc63zOX7fhv/wmv4T4CucBLMTuFxVD+H8/+fgNCJZ\ngZOUy3vK9nKcbVBrFbcCMnWUe4/EUlW9JdixmJpHRBYCs1X1r2XU+xGnFdk77vs07HMHgIh0xUlg\n7b0aaNQqdhrNGOMXEWmJc3ppJk4DhL/g3DfyFz8mvx6n5ZY5WhPgqtqcaMCSjTHGf4dw7jV6CucU\n/HLgbFUts+mx21DDuxm4AVT1m7Jr1Xx2Gs0YY0zAWQMBY4wxAWen0VxJSUnatm3bYIfhU15eHrGx\nscEOwyeLrWIstoqx2ComkLEtWLBgl6rWL7NisPvLqS5D+/bttbqaMWNGsEMokcVWMRZbxVhsFRPI\n2ID5an2jGWOMqQ4s2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAk4SzbGGGMCzpKNMcaY\ngLNkY4wxJuAs2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAk4SzbGGGMCrkqTjYgki8hk\nEckTkY0iMryEeiIiY0Uk0x3Gioh4lL8qIqtE5JCIjPAx/WgR2SYiOSLybxGJDOBqGWOMKUNVH9m8\nBBQCDYErgPEi0sVHvVHAMKAH0B04D7jeo3wRcBPwi/eEIjIUuBs4DWgJtAEeKiuwQ1qe1TDGGFMe\nVZZsRCQWuAi4X1VzVXU28DlwpY/qVwPjVHWzqmYA44ARxYWq+pKq/g8oKGHaN1R1maruBh7xnLYk\nm3MPsWdfYTnXyhhjjD/EeapnFSxIpBcwR1VjPMaNAQap6nledbOBM1V1rvu+LzBDVeO96s0GXlfV\niR7jFgGPq+qH7vtUYCeQqqqZXtOPwjmKIqJR2z7D7/sXV3epfmfccnNziYuLC3YYPllsFWOxVYzF\nVjGBjG3IkCELVLVvWfXCArJ03+KAHK9x2UB8CXWzverFiYho2dnR17S4yzki2ajqq8CrAJGN22na\n5oOMPr8f3ZsllbGIqpWWlsbgwYODHYZPFlvFWGwVY7FVTHWIrSqv2eQCCV7jEoC9ftRNAHL9SDQl\nTUsJy/m9UoSgCvd/tpRDdgHHGGMqVVUmm9VAmIi08xjXA1jmo+4yt6yser74mna79yk0b0mRQqOE\nKBZtzuaDeel+LsoYY4w/qizZqGoe8CnwsIjEikh/4HzgHR/V3wZuF5GmItIEuAOYWFwoIhEiEgUI\nEC4iUSIS4jHttSLSWUSSgPs8py1JiMB953YC4MlpK8nKs8YCxhhTWaq66fNNQDSwA/gPcKOqLhOR\ngSKS61FvAvAFsARYCnzpjiv2DZAPnIxzzSUfOAVAVf8LPAnMADYBG4EH/AnuD90aM6BtKnv2HeCp\naSsrvJLGGGOOVKXJRlWzVHWYqsaqagtVfd8d/72qxnnUU1W9S1WT3eEuz+s1qjpYVcVrSPMof0ZV\nG6pqgqpeo6r7/YlPRHjwj10IDxU+mJfOwk27K3HtjTGm7rLuary0bRDHdQPbOI0FpiylyBoLGGPM\nMbNk48NfT21Lk8Qolmbk8P7cjcEOxxhjajxLNj7ERITxj/M6A/Dkf1exPcdXRwXGGGP8ZcmmBEO7\nNOK0jg3Yu/8gD33hb6trY4wxvliyKYGI8PCwrsREhPLVkm18t3x7sEMyxpgay5JNKZomRTPmzA4A\n/GPKUnL3HwxyRMYYUzNZsinD1Se3onuzRLZkFzDum1XBDscYY2okSzZlCA0RnriwG6EhwsQfNvBr\n+p5gh2SMMTWOJRs/dGmSyHUDWqMK93y6hANFh4IdkjHG1CiWbPz0f6e3o3lyNCu25vDG7PXBDscY\nY2oUSzZ+iokI49Fh3QB47rvVbMzMC3JExhhTc1iyKYdB7eszrGcTCg4c4q6PF9tzb4wxxk+WbMrp\nH+d1ITUugrnrs3jXurIxxhi/WLIpp+TYCB4d1hWAf369kvSsfUGOyBhjqj9LNhVwVtfGnNu9MfsK\ni+x0mjHG+MGSTQU99McupMRG8OO6TN7/eVOwwzHGmGrNkk0FpcRF8vD5zum0J75awebddjrNGGNK\nYsnmGPyhe2PO6daIvMIi7v5kCR4PEzXGGOPBks0xevj8rtSLCWf2ml18MC892OEYY0y1ZMnmGKXG\nRfKQezrt0anLrXWaMcb4YMmmEpzncTrt9km/UmSt04wx5giWbCqBiPDYsG40iI9k3obdvDprXbBD\nMsaYasWSTSWpFxvBU5f0AOCZb1exbEt2kCMyxpjqw5JNJRrUvj5XndSSA0XK6A9/peBAUbBDMsaY\nasGSTSW75+xOtEmNZfX2XJ6aZk/2NMYYsGRT6aIjQnn2sp6EhghvzF7PnDW7gh2SMcYEnSWbAOjR\nPIlbT20HwJiPFpGdfyDIERljTHBZsgmQm4ccR4/mSWzNLuDeyda7gDGmbrNkEyBhoSE8f1lPYiNC\n+XLxVibNt94FjDF1lyWbAGqVGsujFzi9Czzw+TLW7Ngb5IiMMSY4qjTZiEiyiEwWkTwR2Sgiw0uo\nJyIyVkQy3WGsiIhHeU8RWSAi+9y/PT3KIkXkFRHZLiJZIvKFiDStivXz5YJezbiwV1MKDhzilvcX\nWnNoY0ydVNVHNi8BhUBD4ApgvIh08VFvFDAM6AF0B84DrgcQkQhgCvAuUA94C5jijgf4P+Akd7om\nwG7gxQCtj18eHtaVVikxrNy2l8e/WhHMUIwxJiiqLNmISCxwEXC/quaq6mzgc+BKH9WvBsap6mZV\nzQDGASPcssFAGPCcqu5X1RcAAU51y1sD01R1u6oWAB8CvhJalYmLDOPFy3sTHiq8/eNGpi3bFsxw\njDGmyklVtZISkV7AHFWN8Rg3Bhikqud51c0GzlTVue77vsAMVY0XkdFu2dke9ae65ePcus8DlwB7\ngNeBHap6m4+YRuEcRVG/fv0+kyZNqtyV9jJtwwH+s7KQ2HB4+ORoUqL9y/W5ubnExcUFNLaKstgq\nxmKrGIutYgIZ25AhQxaoat+y6oUFZOm+xQE5XuOygfgS6mZ71Ytzr9t4l3nP5zcgHcgAioAlwC2+\nAlLVV4FXATp06KCDBw/2c1UqZpAq2ybOY8aqnXy4MZr3/9KPsNCyE05aWhqBjq2iLLaKsdgqxmKr\nmOoQm9+n0USkoYiMEZHxIpLqjusvIq39nEUukOA1LgHw1UTLu24CkKvOYVhZ83kJiARSgFjgU+Br\nP2MMKBHh6Ut60CA+kp83ZPHMt6uDHZIxxlQJv5KNiPQBVuFc1L+W33f2ZwCP+bms1UCYiLTzGNcD\nWOaj7jK3zFe9ZUB3z9ZpOI0Bist7AhNVNUtV9+M0DjihOEEGW0pcJC9c3osQgZfT1vLd8u3BDskY\nYwLO3yObp4HnVbUXsN9j/DSgvz8zUNU8nKOMh0UkVkT6A+cD7/io/jZwu4g0FZEmwB3ARLcsDef0\n2K1uM+fiU2TT3b/zgKtEJFFEwoGbgC2qWm06KTuxTQp3Du0IwO2TfrWnexpjaj1/k00fnCbG3rbi\nNGP2101ANLAD+A9wo6ouE5GBIpLrUW8C8AXO9ZalwJfuOFS1EKdZ9FU4DQBGAsPc8QBjgAKcazc7\ngXOAC8oRY5W4/pQ2nN6pITkFB7nxvQV2/40xplbzt4FAPs49Ld464iQOv6hqFk6i8B7/Pc6F/+L3\nCtzlDr7msxAnAfoqy8Q53VethYQI4y7pwbn/+p6lGTk8PHU5j1/QLdhhGWNMQPh7ZDMFeEBEIt33\nKiKtgLHAJwGIq05IjAln/BV9iAgL4f25m/j0l83BDskYYwLC32QzBkjGOS0VA8wG1uCcxrovMKHV\nDV2bJvLQH517Tu+dvIRV26z/NGNM7eNXslHVHFUdgHMK7G84N02epaqD3Av/5hj86fjmXNjb6T/t\nhncX2PNvjDG1jr9Nn68SkUhVna6qT6vqk6r6nYhEiMhVgQ6ythMRHhvWjY6N4lm/K4/bPlhI0SF7\n/o0xpvbw9zTam0Cij/Hxbpk5RtERobx2VV+SYsKZsWonz9oNn8aYWsTfZCOAr5/aLTi66xhTQc2T\nY3hpeG9CBP41Yw1fLdka7JCMMaZSlNr0WUSW4CQZBWaKyEGP4lCgJfBV4MKre/q3TeXeczrx6Jcr\nuGPSIlqnxgY7JGOMOWZl3Wfzsfu3K86NlZ43XhYCG7Cmz5Xu2gGtWbYlh8kLMxj1znz+1lPKnsgY\nY6qxUpONqj4EICIbgA/d58OYABMRnriwG2t25LIkI5vxi0I467RDfvUQbYwx1ZG/TZ/fskRTtaLC\nQ5lwZR9SYiNYlnmIx79aGeyQjDGmwvxt+hwhIg+JyGoRKRCRIs8h0EHWVU2Sohn/5z6ECvx7znre\nm7sx2CEZY0yF+Hte5hHcRzUDh4A7cZ4bk4nTuaYJkBNaJzOiSwQA/5iyjNm/VZvOq40xxm/+JptL\ngRtUdQJO9/5TVPVW4AGcZ9qYABrYLJwbBh1H0SHlxvcWsGaHdWljjKlZ/E02DYHl7utcIMl9/V/g\nzMoOyhztrqEdOKtLI/YWHGTkxPlk5RWWPZExxlQT/iabTUAT9/UaYKj7+iScxw+YAAsJEZ65rAfd\nmiayKWsf178zn/0H7XKZMaZm8DfZTAZOc18/DzwkIutxnp75egDiMj7ERITx+tV9aZQQxbwNu7n7\nkyU4j/4xxpjqza+Hp6nqPR6vPxaRdJzHQa9W1amBCs4crWFCFK9f3ZdLXvmRyQszaJkSw22ntw92\nWMYYU6oK3SWoqnNV9RlVnSoi1p9KFevaNJEXLu9FiMBz3/3Gh/M2BTskY4wpVYVvSReRKBG5E1hf\nifEYP53RuSEPn98VgHsnL2X6yu1BjsgYY0pWarJxb+Z8TETmicgPIjLMHX8VsA64DXi2CuI0Pvz5\nxJbcMqQtRYeUm99byK/pe4IdkjHG+FTWkc2DwC3ARqA18JGIvAz8HbgHaKWqTwQ0QlOqO85sz8V9\nmpF/oIiRE+exYZc9ONUYU/2UlWwuBUao6sXAWTiPFagHdHH7S7PnFwdZcaedg9rXJyuvkKv+/TM7\n9+4PdljGGHOEspJNc2AegKouwnmswFhVPVjqVKZKhYeG8PIVvQ/fgzNy4jzy9tu/yBhTfZSVbMIB\nz5/JB7Anc1ZLsZFh/HvE8bRIjmFJRjaj3plPwQG76dMYUz34c5/NEyKyz30dATwoIkckHLefNBNk\n9eMjeXvkCVz8yo/MWZPJrf9ZyMtX9Lbn4Bhjgq6svdAs4Digmzv8ALTweN8N5ymepppolRrLu9ed\nQGJ0ON8s385dHy/m0CHrZcAYE1xlPalzcBXFYSpRx0YJvHnN8fz59bl8ujCD+KgwHvxjF0Ts8dLG\nmOCw8yu1VO8W9Xjtqr5EhIbw1o8beebb1cEOyRhTh1myqcX6t03lxeG9CA0RXpy+hldnrQ12SMaY\nOqpKk42IJIvIZBHJE5GNIjK8hHoiImNFJNMdxorHOSAR6SkiC0Rkn/u3p9f0vUVklojkish2Efm/\nQK9bdTW0SyOeurg7AI9/tZJ3frJHSxtjql5VH9m8hHOvTkPgCmC8iHTxUW8UMAzoAXQHzgOuB6cL\nHWAK8C7ODaZvAVPc8YhIKs5D3SYAKUBb4JvArVL1d2HvZjx8vrOZ7/9sKe/PtY47jTFVq8qSjds7\n9EXA/aqaq6qzgc+BK31UvxoYp6qbVTUDGAeMcMsG4zRseE5V96vqC4AAp7rltwPTVPU9t3yvqq4I\n2IrVEFed1Ir7z+0MwL2TlzBpXnqQIzLG1CXiz8O3RKRFCUUKFKjqTj/m0QuYo6oxHuPGAINU9Tyv\nutnAmao6133fF5ihqvEiMtotO9uj/lS3fJyITAeWAMfjHNXMBW5W1aN+zovIKJyjKOrXr99n0qRJ\nZa1GUOTm5hIXF1cp8/p6/QE+XFWIANd2i2BA0/BqE1tls9gqxmKrmLoa25AhQxaoat8yK6pqmQNw\nCCgqZdgNPAOElTKPgcA2r3F/AdJ81C0COnq8b4eT2AS4H/jAq/57wIPu69XAHpxkEwW8gJPkSl3H\n9u3ba3U1Y8aMSp3f+LQ12vJvU7XV3VP1kwXpxzSvyo6tMllsFWOxVUxdjQ2Yr37kEb+e1AlcDjwJ\nvIJzpADQD+eo4EEgCbgP2As8UMI8coEEr3EJ7jRl1U0AclVVRaSs+eQDk1V1HoCIPATsEpFEVbWu\ndoAbBh1H0SHlqWmrGPPRIkJDhPN7Ng12WMaYWszfZHMjMFpVP/UYN11EVgH/p6qDRGQH8BAlJ5vV\nQJiItFPV39xxPYBlPuouc8t+9lFvGXCHiIibVcFpRPCS+3oxzlFQMbt93oeb3efgPPPtakZ/+Cuq\nMKyXJRxjTGD420CgH851EG9LcU5XAfwINCtpBqqaB3wKPCwisSLSHzgfeMdH9beB20WkqYg0Ae4A\nJrplaTin2W4VkUgRucUdP939+yZwgds8OhzntNtsO6o52q2nteO209txSGH0pF/t8dLGmIDxN9ls\nxL2Q7uUvQPEeqj6QVcZ8bgKigR3Af4AbVXWZiAx0T48VmwB8gZPglgJfuuNQ1UKcZtFX4VybGQkM\nc8ejqtOBe91pduA0EvB5P4+B205vz51DO6AKf/tkCRPn2FO+jTGVz9/TaHcAn4jIObjPtwH64nTS\neZH7/nig1OZcqpqFkyi8x38PxHm8V+Aud/A1n4VAn1KWMx4YX1os5nc3D2lLdHgoD09dzoNfLKfg\n4CFuGHRcsMMyxtQifiUbVf1SRNrhHJl0cEd/DryibpNiVX05MCGaqjByQGuiwkP5+2dL+OfXK8kv\nLOK209tZ553GmErh75ENqpoO3BPAWEyQDe/XgqjwEMZ8tIjn//cbBQeKuPvsjpZwjDHHzO9kIyIx\nQE+gAV7XerxaqZka7MLezYgMC+X/PljIhFnryN1/kIfP70poiCUcY0zF+ZVsROR0nAv6KT6KFQit\nzKBMcP2he2Miw0K46f1feG/uJnbvK+TZy3oSGWb/ZmNMxfjbGu15nNZdzVQ1xGuwPVAtdHrnhrwz\n8gTiI8P4ask2rnlzHnsLDgQ7LGNMDeVvsmkFPKKqWwIYi6lm+rVJ4cPrT6J+fCQ/rM3k8td+Yufe\n/cEOyxhTA/mbbObweys0U4d0bpLAJzecTKuUGJZm5HDJKz+QnrUv2GEZY2oYf5PNK8DTInKdiPRz\nH052eAhkgCb4WqTE8NENJ9OlSQIbMvdx4fgfWL4lJ9hhGWNqEH+TzcdAR+BVnG5p5nsM80qZztQS\n9eMj+WDUiZzUJoWde/dz6YQfmbm6zCdLGGMM4H+yaV3K0CYwoZnqJj4qnDevOZ5zuzcmd/9BRk6c\nR1q6NRowxpTN3x4E7MH1BoCo8FBe+FMvWiTH8HLaWiYuKyTq65XcNbQDIXYvjjGmBCUmGxG5EPhC\nVQ+4r0tkN3XWLSEhwl1ndaRFcgz3Tl7CKzPXkp61j3GX9iAq3FrCG2OOVtqRzcdAI5yekz8upZ7d\n1FlH/emEFuza9BsTlhzkyyVb2Zqdz2tX9SUlLjLYoRljqpkSr9m4N2zu8Hhd0mCJpg7rmhrKRzee\nRJPEKH7ZtIcLXv6B1dt9PXzVGFOX+dtAwJgSdWyUwOSb+9O1aQKbsvZxwUtz+Hb59mCHZYypRsrT\nEWcz4BR8d8T5TCXHZWqYhglRfHT9ydz58SKmLt7KqHfmM+bMDtw0+DjrNdoY43dHnFcA/wYOAjtx\nrtMUU8CSjSE6IpQXL+9Fp8YxhMz6AAAgAElEQVQJPP3NKp6atooVW3N46uIeREfY2VZj6jJ/T6M9\nDIwDElS1laq29hjsPhtzmIhw85C2vHZlX2IjQpm6eCuXTPiBLXvygx2aMSaI/E02DYHXVbUokMGY\n2uP0zg2ZfHN/Wrp9qv3xX7OZuy4z2GEZY4LE32TzFdAvkIGY2qd9w3im3Nyf/m1T2JVbyPDX5/La\nrHWoatkTG2NqFX8bCHwLjBWRLsAS4Ig+SuymTlOSpJgI3rrmBJ7+ZjWvzFzLY1+tYMHG3Tx1SXfi\no8KDHZ4xpor4m2wmuH/v9VFmN3WaUoWFhnD32R3p1SKJMZMW8d9l21i1fS+v/LkPHRrFBzs8Y0wV\n8Os0mt3UaSrD0C6N+PyvA+jYKJ71u/IY9tIcPluYEeywjDFVoMxkIyLhIjJXROzhaeaYtU6NZfJN\n/bmwd1PyDxRx24e/ct9nSyg4YG1PjKnNykw2qnoA51ECdlXXVIroiFDGXdKDxy/oRkRoCO/+tIlh\nL81hzQ7r5saY2srf1mhvAX8JZCCmbhERhvdrwac3nUzr1FhWbtvLeS/OYdK8dGutZkwt5G8DgVjg\nChE5A1gA5HkWquqtlR2YqRu6Nk3ki78O4B+fLeXThRnc9clivl+zi8cu6EqCtVYzptbw98imE/AL\nsBvnyZzdPIaugQnN1BVxkWE8c1lPnrm0BzERoXyxaAt/eOF7fk3fE+zQjDGVxN8ndQ4JdCDGXNi7\nGb1a1OOv//mFpRk5XDz+B0af0Z4bBh1HqD0F1JgazR4xYKqV1qmxfHLjyYzs35qDh5Snpq3i0gk/\nsjEzr+yJjTHVlt/JRkSGiMirIvJfEZnuOZRjHskiMllE8kRko4gML6GeiMhYEcl0h7Hi0U+9iPQU\nkQUiss/929PHPCJEZIWIbPY3PlM9RIaF8o/zOvP2yBNomBDJgo27Ofv573l/7iZrPGBMDeVXshGR\nEcDXQDwwGOcxA/WA3sDycizvJaAQp2PPK4Dxbhc43kYBw4AeQHfgPOB6N5YIYArwrhvDW8AUd7yn\nO904TQ11Svv6TLvtFM7r0YR9hUXcO3kJIyfOY0dOQbBDM8aUk79HNmOAW1T1cpx+0e5R1V44O/xc\nf2YgIrHARcD9qpqrqrOBz4ErfVS/GhinqptVNQPn8QYj3LLBONeanlPV/ar6AiDAqR7Lag38GXjC\nz/Uz1VRSTAQvXt6LFy7vRWJ0ODNW7WToc7P4asnWYIdmjCkH8ee0hIjsAzqr6gYR2QWcqqqLRaQj\nkKaqjfyYRy9gjqrGeIwbAwxS1fO86mYDZ6rqXPd9X2CGqsaLyGi37GyP+lPd8nEe79/AaT33rqo2\nKyGmUThHUdSvX7/PpEmTytwWwZCbm0tcXFyww/CpKmPbXXCIN5YUsjTT6W3g+Eah/LlTJImRvhsP\n2HarGIutYupqbEOGDFmgqn3LqufvfTaZOKfQADJwmjsvBlKAaD/nEQfkeI3L9pivd91sr3px7nUb\n77Ij5iMiFwChqjpZRAaXFpCqvgq8CtChQwcdPLjU6kGTlpaGxeYYNlR596eNPPH1SuZtK+K3nAM8\ncF5nhvVsetTjp227VYzFVjEWW+n8PY32PXCm+3oS8IKIvAn8B+fxA/7IBRK8xiUAvvoo8a6bAOSq\ncxhW4nzcU3VPAnaTaS0lIlx5Uium3XYKA9ulsmffAUZ/uIiRE+fZ00CNqcb8TTa34CQWcK6DPIVz\nVDMJuM7PeawGwkSknce4HsAyH3WXuWW+6i0DusuRP2O7u+PbAa2A70VkG/Ap0FhEtolIKz/jNDVA\n8+QY3h55Ak9e3J2EqDBmrNrJmc/O4r25Gzl0yFqsGVPd+HtTZ5bH60PA2PIuSFXzRORT4GERuQ7o\nCZwPnOyj+tvA7SLyFU4HoHcAL7plaUARcKuIvMLvfbZNBw4BzT3mczLwL5xWc9YyrZYRES7t25zB\n7etz32dL+Wb5dv4+eSmf/7qFxy7oFuzwjDEeynOfTUMRGSMi40Uk1R3X32355a+bcK7x7MA5UrpR\nVZeJyEAR8WzVNgH4AuepoEuBL91xqGohTrPoq4A9wEhgmKoWqupBVd1WPABZwCH3vfVhX0s1SIhi\nwpV9eGl4b1LjIpi7Pouzn5/FJ6sL7dEFxlQT/t5n0wdYhXNvzLX8fs3kDOAxfxemqlmqOkxVY1W1\nhaq+747/XlXjPOqpqt6lqsnucJd6NJtT1YWq2kdVo1W1t6ouLGF5aSW1RDO1i4jwh+6N+Xb0IC4/\noTkHipQv1h3gjGdnMmPljmCHZ0yd5++RzdPA8+69Nfs9xk8D+ld6VMZUUL3YCJ64sDuf3HgSzeND\nSM/K55qJ87jhnQXWgMCYIPI32fTBuVPf21ac3gCMqVb6tEzmwZOiuO8PnYiJCOW/y7Zx+jMzeW3W\nOgoPHgp2eMbUOf4mm3ycrmG8dcS5/mJMtRMaIlw3sA3/u2MQZ3dtxL7CIh77agVnPT+LGavsY2tM\nVfI32UwBHhCRSPe9uk2JxwKfBCAuYypN48Roxv+5D29eczxtUmNZtzOPa96cxzVv/szanX71tmSM\nOUbl6RstGaf5cAwwG1iDc+f+fYEJzZjKNaRDA/572ync94dOxEc69+YMfXYWj0xdTnb+gWCHZ0yt\n5leyUdUcVR2A0+T4b8DzwFmqeoqq2oNGTI0RERbCdQPbMOPOwVx+QnOKVHlj9nqGPJ3Ge3M3crDI\nrucYEwjleniaqk5X1adV9UlV/U5EWopI9ey90phSpMZF8sSF3fnilgGc0DqZrLxC/j55KWc//z3f\nLd9uz80xppId65M6k3AeG2BMjdS1aSIfjjqRfw3vRfPkaH7bkct1b8/nsgk/sWDj7mCHZ0ytYY+F\nNnWeiHBu9yZ8d/sg/nFuZ+rFhPPzhiwuGv8DN7yzwBoRGFMJLNkY44oMC2XkgNbMvGsINw85jqjw\nEP67bBtnPjuLeycvsSeEGnMMLNkY4yUhKpw7h3YkbcwQ/nR8c1SV9+duYuCTM3h06nJ25e4veybG\nmCOU2uuziHxexvTez5UxptZolBjFPy/qznUDW/PUtFVMW7ad12ev5725m7j65FZcf0ob6sVGBDtM\nY2qEsh4xkOlH+fpKisWYaqltg3gmXNmXpRnZPPvtav63cgevzFzLOz9uYOSA1lw3oA2JMeHBDtOY\naq3UZKOq11RVIMZUd12bJvLGiOP5NX0Pz367mpmrd/Li9DVM/GED1w5ozTUnt7akY0wJ7JqNMeXU\ns3kSb408gU9uPIkBbVPZW3CQ5777jf5jp/PE1yvYsdcaEhjjzZKNMRXUp2Uy717Xjw9HncjAdqnk\n7j/IhJnrGDB2Bvd/tpT0rH3BDtGYasOvx0IbY0rWr00K/dqksCh9Dy+nrWHasu2889NG3v95E+f3\nbMKNg46jXcP4YIdpTFDZkY0xlaRH8yQmXNmXb0afwoW9mgLw6S8ZnPHsLEa9PZ95G7KsGxxTZ9mR\njTGVrH3DeJ65rCejz2jPq7PW8eH8dL5Zvp1vlm+nR7NErh3YhphDlnRM3WJHNsYESPPkGB4Z1pXZ\nfxvCrae2pV5MOIs2Z3PrfxZy16x8Xp211h5tYOoMSzbGBFiD+ChuP7MDP9x9Go9f0I029WPJKlAe\n/2olJz/xPx76YhmbMq0xgandLNkYU0WiI0IZ3q8F340exOg+kfRvm0JeYRFvztnAoKdnMHLiPGas\n3EGRnWIztZBdszGmioWECD3qh/F/l5zI8i05vDF7PV8s3sL0lTuYvnIHzZOj+XO/llzStznJ1h2O\nqSXsyMaYIOrcJIFxl/bgp3tO4+6zO9KsXjTpWfk88fVKTnzif9w+6Vd+Td9jrdhMjWdHNsZUA8mx\nEdww6Dj+MrANM1fv4J0fN5K2eief/pLBp79k0LVpApcd34I/9mhCYrR1iWNqHks2xlQjoSHCqR0b\ncmrHhmzMzOP9uZv4cH46SzNyWJqxlEenLuecbo25tG9zTmyTjIgEO2Rj/GLJxphqqmVKLPec04nR\nZ7Rn2rJtfDgvnR/WZjJ5YQaTF2bQMiWGS/s256LezWiUGBXscI0plSUbY6q5qPBQzu/ZlPN7NmVT\n5j4+WpDOR/M3szFzH09NW8W4b1YxuEMDLurdjNM6NSAqPDTYIRtzFEs2xtQgLVJiuOPMDtx2entm\n/baTSfPS+W7F9sMt2eIjwzinW2OG9WpKv9bJhITYaTZTPVRpazQRSRaRySKSJyIbRWR4CfVERMaK\nSKY7jBWPk9Mi0lNEFojIPvdvT4+yO0VkqYjsFZH1InJnVaybMVUpNEQY0qEB4//ch5/uOY37z+1M\nt6aJ7N1/kA/np3P5az/Rf+x0/vn1SlZt2xvscI2p8iObl4BCoCHQE/hSRBap6jKveqOAYUAPQIFv\ncZ4I+oqIRABTgOeAl4HrgSki0k5VCwEBrgIWA8cB34hIuqp+EPC1MyYIUuIiuXZAa64d0Jo1O/by\n2cItTF6YQcaefF6ZuZZXZq6lU+MEhvVswh+6N6ZZvZhgh2zqoCo7shGRWOAi4H5VzVXV2cDnwJU+\nql8NjFPVzaqaAYwDRrhlg3GS5HOqul9VX8BJMKcCqOqTqvqLqh5U1VU4ial/AFfNmGqjbYN4xgzt\nwPd3DeGjG05ieL8WJEaHs2JrDk98vZIBY2dw/ktzeHXWWjbvti5yTNWRqrpZTER6AXNUNcZj3Bhg\nkKqe51U3GzhTVee67/sCM1Q1XkRGu2Vne9Sf6paP85qPAL8AE1T1FR8xjcI5iqJ+/fp9Jk2aVElr\nW7lyc3OJi4sLdhg+WWwVU5WxHTikLN5ZxNytB/l1ZxGFRb+XtUkM4fhGYfRtGEr9mJAqj628LLaK\nCWRsQ4YMWaCqfcuqV5Wn0eKAHK9x2YCvp0rFuWWe9eLc5OFdVtp8HsQ5envTV0Cq+irwKkCHDh10\n8ODBpa5AsKSlpWGxlZ/F9rsz3L/5hUWkrdrB1CVbmb5iB+uyi1iXXciHq6BHs0TO6daYRNnIubbd\nys1iK11VJptcIMFrXALg6+qld90EIFdVVUT8mo+I3IJz7Wagqu4/lsCNqS2iI0I5u1tjzu7W+KjE\ns2hzNos2O7/jXl81kzM6N+T0Tg3p1TzJWrWZY1aVyWY1EOZeyP/NHdcD8G4cgDuuB/Czj3rLgDtE\nRPT3c4DdcRofACAiI4G7gVNUdXPlroYxtYOvxPPfZdv4ZukW1uzIZc2OXManrSU1LpLTOzXgjM4N\n6d821e7jMRVSZclGVfNE5FPgYRG5Dqc12vnAyT6qvw3cLiJf4bRGuwN40S1LA4qAW0XkFeAv7vjp\nACJyBfA4MERV1wVodYypVTwTz3fT9xDdohvfLt/Ot8u3k7Ennw/mpfPBvHSiw0MZ2C6VUzs2YHCH\nBtZzgfFbVTd9vgn4N7ADyARuVNVlIjIQ+FpVi69gTQDaAEvc96+741DVQhEZ5o77J7ACGOY2ewZ4\nFEgB5nncmvOuqt4Q0DUzppYICxH6t02lf9tUHjivMyu27uW7FU7iWZKRffgR1wAdG8UzqEN9BrWv\nT9+WyUSEWUfyxrcqTTaqmoVz/4z3+O9xLvwXv1fgLnfwNZ+FQJ8SylpXSrDGGESEzk0S6NwkgVtP\na8fW7Hy+W7GDmat28sPaXazctpeV2/YyYeY64iLDOPm4FAZ3aMCgDvVpmhQd7PBNNWLd1Rhj/NY4\nMZorT2zJlSe2ZP/BIuZv2M3M1TtJW7WD1dtzjzjqadsgjgFtUzn5uBT6tUmxRyPUcZZsjDEVEhkW\nevh0273ndCJjTz4zV+1k5uodzFmTebiRwcQfNhAi0K1pIie7yadvy2SiI6yhQV1iycYYUymaJkUz\nvF8LhvdrQeHBQ/yavoc5a3bx49pMFqbvPty0enzaWiJCQ+jdMomTj0ulf9sUujVNsus9tZwlG2NM\npYsIC+GE1smc0DqZ0WfAvsKD/Lw+ix/WZvLD2l0s25LDT+uy+GldFs98C1HhIfRsnsQJrZI5vnUy\nvVvUIzbSdk+1if03jTEBFxMRxuAOTnNpgN15hfy0LpMf1mby4zrnlFtx8gGnV+suTRI4vlWyO9Qj\nJS4ymKtgjpElG2NMlasXG3H4vh6AzNz9zN+4m3nrs5i3IYulW3JYvDmbxZuzeWP2egCOqx9L08j9\nbIvZRK8W9WjbII5Q69mgxrBkY4wJupS4SIZ2acTQLo0AyNt/kIWb9vDzhizmrc9iYfpu1u7MYy0w\na7Nz+11sRCg9mifRq0USPZvXo2fzJOrH29FPdWXJxhhT7cRGhjGgXSoD2qUCUHjwEEu3ZPPR9Pns\njUhh4aY9ZOzJd68BZR6ernlyND2b16NX8yR6NE+kc+NEa/VWTViyMcZUexFhIfRuUY+cVuEMHtwb\ngB05BSxM38Ov6XtYuGk3izdnk56VT3pWPl8s2gJAiMBx9ePo1jSRru7QpUmCNT4IAtvixpgaqUFC\n1BGn3g4WHeK3Hbks3OQknyUZ2fy2I/fw8OnCDABEoE1qLF2bJtKtaSJdmiTSpWkCCVF202kgWbIx\nxtQKYaEhdGqcQKfGCQzv1wKAggNFrNy2lyUZ2SzLyGZJRjart+91rv/szGPKr1sOT9+sXjQdG8XT\nsVECHRvH07FRPK1SYgkLtft/KoMlG2NMrRUVHkrP5kn0bJ50eNz+g0Ws3pbL0i3Zh5PQim172bw7\nn827nb7fikWEhdC+YRwdGibQqbGTiDo0ireGCBVgycYYU6dEhoXSrVki3Zolcrk77mDRITZk5jkd\ni27dy8ptOazYupeMPfkszchhacaRDxlOiY2gbYO4w0O7BvHsLjiEquLR27zxYMnGGFPnhYWG0LZB\nPG0bxHNu99/H5xQcYPW2vazYtpeVW3NY5fZynZlXSOb6LOauzzpiPv/48RvaNIijbf042jV0/rZt\nEEfz5Jg6f0+QJRtjjClBQlQ4fVsl07dV8uFxqkrGnvzDHY2u3en8XZGxm737D7IofQ+L0vccMZ+I\nsBBap8TSMiWG1qmxtEqNpVVKLK1SY2gYH1UnHrttycYYY8pBRGhWL4Zm9WIOd78DkJaWRre+J/Gb\nm4Q8E9HW7AJWbd/Lqu17j5pfVHgIrdxE1Co1ltYpvyejhgmRtea0nCUbY4ypJClxkaTERXJim5Qj\nxu8tOMCGXfvYkJnHhl15rHf/bszcR2Ze4eGH0HmLDg+lWb1omtWLpnlyDM3rxRzxOjGm5jTXtmRj\njDEBFh8VfrhRgrfs/ANszMxj/a48Nuza57x2k9HufQcO3yfke75hNKsXQ3M3ATWrF03zejE0T46h\ncVJUtbp3yJKNMcYEUWJ0ON2bJdG9WdJRZdn5B9i8ex/pWfls3r2PzbvzSc/aR7o7bm/BQVZszWHF\n1hwfc4a4yDAaJ0YRWVTA17sW0zgpiiaJ0TROiqJxYjRNkqKIiaiaNGDJxhhjqqnE6HASo51eDryp\nKll5haTvzj+ckNLdhLQ5ax9bsvPJ3X/w8FHR0sx0n8tIiAqjSVI0jROjaJwUTZPEKBokRNEwIYqG\nCZE0iI+iXkz4MV87smRjjDE1kIgcvkbkedNqMVUlO/8AW/YU8M3sn0lp0Y6te/LZll3Alux8tmYX\nsDW7gJyCg+SUcM2oWERoCPXjI2mQEEnD+Cjnb0IUDcpxc6slG2OMqYVEhKSYCJJiItjRIIzBJ7Y8\nqo6qkplX6CSgPU4C2pKdz86c/WzfW8COnP1sz3ESUsaefDL25Fc4Hks2xhhTR4kIqXGRpMZF0rXp\n0afqihUcKHISz94CtucUHH69I2c/z/m5LEs2xhhjShUVHkqLlBhapMQcVfbcn/ybh3VnaowxJuAs\n2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAm4Kk02IpIsIpNFJE9ENorI8BLqiYiMFZFM\ndxgrHn0liEhPEVkgIvvcvz39ndYYY0zVq+ojm5eAQqAhcAUwXkS6+Kg3ChgG9AC6A+cB1wOISAQw\nBXgXqAe8BUxxx5c6rTHGmOCosmQjIrHARcD9qpqrqrOBz4ErfVS/GhinqptVNQMYB4xwywbj3Iz6\nnKruV9UXAAFO9WNaY4wxQVCVPQi0Bw6q6mqPcYuAQT7qdnHLPOt18ShbrKrqUb7YHf/fMqY9goiM\nwjkSAtgvIkv9W5UqlwrsCnYQJbDYKsZiqxiLrWICGdvRna75UJXJJg7wfuhCNhBfQt1sr3px7rUX\n7zLv+ZQ4rVeCQlVfBV4FEJH5qtrX/9WpOhZbxVhsFWOxVYzFVrqqvGaTCyR4jUsAfPVr7V03Ach1\nk0VZ8yltWmOMMUFQlclmNRAmIu08xvUAlvmou8wt81VvGdDdq4VZd6/ykqY1xhgTBFWWbFQ1D/gU\neFhEYkWkP3A+8I6P6m8Dt4tIUxFpAtwBTHTL0oAi4FYRiRSRW9zx0/2YtjSvln+tqozFVjEWW8VY\nbBVjsZVCqvLskogkA/8GzgAygbtV9X0RGQh8rapxbj0BxgLXuZO+Dvyt+FSYiPRyx3UGVgDXqupC\nf6Y1xhhT9ao02RhjjKmbrLsaY4wxAWfJxhhjTMDV+WTjb39tlbi8NBEpEJFcd1jlUTbcjSFPRD5z\nr3H5FWdp05YSyy0iMl9E9ovIRK+y00Rkpdv/3AwRaelRFiki/xaRHBHZJiK3V9a0ZcUmIq1ERD22\nX66I3F9Vsbl13nC39V4R+VVEzq4O26202IK93dx674rIVrfeahG5rjLmH8jYqsN286jfTpx9x7se\n4wKyzyhr2gpR1To9AP8BPsS5GXQAzk2gXQK4vDTgOh/ju+DcK3SKG8v7wAf+xFnWtKXEciFOP3Lj\ngYke41Pd+V8CRAFPAT95lD8BfI/TN10nYBtw1rFO62dsrQAFwkpYp4DGBsQCD7pxhADnutu+VbC3\nWxmxBXW7eXxOI93XHd16fYK93cqILejbzaP+N279dwO9zyht2grv+wK1U60JA86XsxBo7zHuHeCf\nAVxmGr6TzePA+x7vj3Njiy8rztKm9TOmRzlyhz4K+MFrO+UDHd33W4AzPcofKf6gHsu0fsZW1pe/\nymLzqLcYp9+/arPdfMRWrbYb0AHYClxa3babV2zVYrsBfwIm4fyYKE42AdlnlDVtRYe6fhqtpP7a\nfPalVomeEJFdIjJHRAa7447o001V1+L+w/2Is7RpK8J7fnnAWqCLiNQDGlN633UVnbY8NorIZhF5\nU0RSAYIRm4g0xNnOy45x/oGOrVhQt5uIvCwi+4CVODv0r45x/oGOrVjQtpuIJAAPA96n2QK1zwjI\nfrGuJ5vy9NdWWf4GtAGa4txo9YWIHEfpfb6VFWdZ/cWVV1mxwNH9z/kTS1nT+mMXcDxO53993Gnf\n81h2lcUmIuHust9S1ZXHOP9Ax1Yttpuq3uSWDcS5yXv/Mc4/0LFVh+32CPCGqm72Gh+ofUZA9ot1\nPdmUp7+2SqGqc1V1rzqPR3gLmAOcU0Ys5e0Pzru8vMqKBY7uf86fWMqatkzqPJ5ivqoeVNXtwC3A\nmSISX5WxiUgIzqmFQjeGY51/QGOrLtvNjaVInUeMNANuPMb5BzS2YG83cR4MeTrwrI9wA7XPCMh+\nsa4nm/L01xYoivM8niP6dBORNkCkG2NZcZY2bUV4zy8W55zuMlXdjXOKobS+6yo6bUUU35UcUlWx\niYgAb+A8BPAiVT1QCfMPdGzeqny7+RBWPJ9jmH+gY/NW1dttMM51o00isg0YA1wkIr/4mH9l7TMC\ns188lgs+tWEAPsBpeREL9CeArdGAJGAoTsuUMJynlebhnCPtgnPoOtCN5V2ObB1SYpxlTVtKPGFu\nLE/g/BIujqu+O/+L3HFjObIVzT+BmTitaDrifGmKW+BUeFo/Y+uHcwE3BEjBaTEzo4pjewX4CYjz\nGl8dtltJsQV1uwENcC5yxwGhON+DPOCPwd5uZcQW7O0WAzTyGJ4GPnbnHbB9RmnTVnj/F4idak0a\ngGTgM/fDtQkYHsBl1Qfm4RyO7sHZKZzhUT7cjSEP59HXyf7GWdq0pcTzIM4vNc/hQbfsdJwLpfk4\nLehaeUwXidPHXQ6wHbjda74Vnras2IDLgfXuem7F6Xi1UVXFhnPuXoECnNMNxcMVwd5upcVWDbZb\nfZwd6x633hLgL5Ux/0DGFuztVsL34t1A7zPKmrYig/WNZowxJuDq+jUbY4wxVcCSjTHGmICzZGOM\nMSbgLNkYY4wJOEs2xhhjAs6SjTHGmICzZGNMLSMiI0Qkt+yaxlQdSzbGBIiITHQfvFU87BKRqSLS\nsRzzeFBElgYyTmOqgiUbYwLrO5yu5BsDZwLRwOSgRmRMEFiyMSaw9qvqNnf4Baf33o4iEg0gIv8U\nkVUiki8iG0TkSRGJcstGAA/gPP+k+OhohFuWKCLjxXmUcYGIrBCRyzwXLM4jiZe6j/adISKtq3LF\njfEUFuwAjKkr3G7pLwOWqGq+OzoPGAlkAJ1xOtLcD9yP0+ljV5zHOw9262e7PTt/hdOB4zU4vfR2\nwOnssVgkcI877wLgLXfeQwOzdsaUzpKNMYF1lsfF+lggHef5RQCo6iMedTeIyOM43cjfr6r57rQH\nVXVbcSUROQM4CacX3hXu6HVeyw0DblbVVe40TwP/FhFR6xDRBIGdRjMmsGYBPd3hBOB/wDci0hxA\nRC4Wkdkiss1NLM8CLcqYZy9gq0ei8WV/caJxbQEicI6GjKlylmyMCax9qrrGHeYB1+E89XCUiJyI\n89yQacB5OEnkPiC8EpZ78P/bu1uViKIoDMPvSgY12BTDFItVMFk1eAGGwavwBrwBi17IgDBZsIiY\nBIvVIgajzTDLsESGU0RxyYT3SefA3gdO+tg/8A3ev0q//uDb0o+5jSb9rwRmVCnWHvA8v5UWEaPB\n+Heq0GvePbAREdvfrG6khWHYSL2WImL983mN6rBfAabAKrAZEcfALXV4Px7MfwJGEbFDlVi9UVtx\nd8AkIk6oCwJbwHJmXsVYJPQAAABsSURBVPb+jvQ7LqmlXvtUw+MLFRC7wFFmXmfmFDgDzoEH4AA4\nHcyfUDfProBXYJyZM+AQuKHqfB+BC+pMRlpINnVKktq5spEktTNsJEntDBtJUjvDRpLUzrCRJLUz\nbCRJ7QwbSVI7w0aS1O4D96wB2VjPCZQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kialxDVNRwNM",
        "colab_type": "text"
      },
      "source": [
        "### Piecewise Constant Scheduling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jwD-Aq4ARwNO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def piecewise_constant_fn(epoch):\n",
        "    if epoch < 5:\n",
        "        return 0.01\n",
        "    elif epoch < 15:\n",
        "        return 0.005\n",
        "    else:\n",
        "        return 0.001"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRliBWq_RwNQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def piecewise_constant(boundaries, values):\n",
        "    boundaries = np.array([0] + boundaries)\n",
        "    values = np.array(values)\n",
        "    def piecewise_constant_fn(epoch):\n",
        "        return values[np.argmax(boundaries > epoch) - 1]\n",
        "    return piecewise_constant_fn\n",
        "\n",
        "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aRB1cM-LRwNR",
        "colab_type": "code",
        "outputId": "3eae5985-352f-4500-8b46-a7d64579c5f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 25\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid),\n",
        "                    callbacks=[lr_scheduler])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 9s 159us/sample - loss: 0.8756 - accuracy: 0.7524 - val_loss: 0.8918 - val_accuracy: 0.7766\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 8s 151us/sample - loss: 0.8555 - accuracy: 0.7578 - val_loss: 1.2542 - val_accuracy: 0.7864\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.9163 - accuracy: 0.7350 - val_loss: 0.7189 - val_accuracy: 0.7826\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 9s 166us/sample - loss: 0.9085 - accuracy: 0.7340 - val_loss: 0.8384 - val_accuracy: 0.7172\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 9s 161us/sample - loss: 0.8598 - accuracy: 0.7484 - val_loss: 0.9351 - val_accuracy: 0.7756\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.6056 - accuracy: 0.8113 - val_loss: 0.6527 - val_accuracy: 0.8086\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 9s 166us/sample - loss: 0.5567 - accuracy: 0.8304 - val_loss: 0.6955 - val_accuracy: 0.8024\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 9s 166us/sample - loss: 0.5331 - accuracy: 0.8339 - val_loss: 0.5901 - val_accuracy: 0.8346\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 8s 152us/sample - loss: 0.5278 - accuracy: 0.8423 - val_loss: 0.6391 - val_accuracy: 0.8266\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.5033 - accuracy: 0.8445 - val_loss: 0.6013 - val_accuracy: 0.8436\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 8s 153us/sample - loss: 0.4908 - accuracy: 0.8485 - val_loss: 0.6353 - val_accuracy: 0.8456\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 9s 162us/sample - loss: 0.4931 - accuracy: 0.8543 - val_loss: 0.5919 - val_accuracy: 0.8462\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 9s 160us/sample - loss: 0.4883 - accuracy: 0.8556 - val_loss: 0.5782 - val_accuracy: 0.8522\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 9s 167us/sample - loss: 0.4621 - accuracy: 0.8589 - val_loss: 0.7027 - val_accuracy: 0.8340\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.4490 - accuracy: 0.8650 - val_loss: 0.5968 - val_accuracy: 0.8520\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 8s 151us/sample - loss: 0.3373 - accuracy: 0.8928 - val_loss: 0.5148 - val_accuracy: 0.8684\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.3034 - accuracy: 0.9009 - val_loss: 0.5065 - val_accuracy: 0.8722\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.2941 - accuracy: 0.9035 - val_loss: 0.5489 - val_accuracy: 0.8700\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 8s 150us/sample - loss: 0.2849 - accuracy: 0.9067 - val_loss: 0.5131 - val_accuracy: 0.8736\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 8s 149us/sample - loss: 0.2780 - accuracy: 0.9093 - val_loss: 0.5250 - val_accuracy: 0.8726\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 8s 151us/sample - loss: 0.2723 - accuracy: 0.9115 - val_loss: 0.5175 - val_accuracy: 0.8762\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 8s 151us/sample - loss: 0.2644 - accuracy: 0.9133 - val_loss: 0.5525 - val_accuracy: 0.8728\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 9s 170us/sample - loss: 0.2615 - accuracy: 0.9150 - val_loss: 0.5539 - val_accuracy: 0.8728\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 9s 156us/sample - loss: 0.2539 - accuracy: 0.9179 - val_loss: 0.5782 - val_accuracy: 0.8696\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 8s 150us/sample - loss: 0.2514 - accuracy: 0.9189 - val_loss: 0.5723 - val_accuracy: 0.8738\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "INxukUvARwNT",
        "colab_type": "code",
        "outputId": "d290ceed-1e79-42e5-faef-07c06ab0225a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "source": [
        "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
        "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
        "plt.grid(True)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X2cnFV9///XO7dsNgmBTUggaBYE\nFoxCEG+qkRJBpNryI4JfW0GBUoWvli9abiy0ooi1GIVWqBRJrQbEWq0GAmKhxbhVFBW5DRGSyk0k\nCTdJMCGb+5vP749zNkwmM7vXJjszyc77+XjMIzPXOeeaM2cn85lzrjPnKCIwMzOrpUGNroCZmQ18\nDjZmZlZzDjZmZlZzDjZmZlZzDjZmZlZzDjZmZlZzDja2A0ntkkLSGxtdl2okdUr6SqPrYcVIOltS\nV43O/ZikK/pY5hlJF1d7bP3PwaYJSZqVg0lI2iTpKUlXS2rNWZ4F9gcebmA1e3MqcFktn0DJhyXd\nJ2m1pJclPSjpk5JG1/K5y+pRsw/CoueWdJCkWyQtlrRB0lJJd0o6uhb1aoA3Af/c6EoMZEMaXQFr\nmHuADwFDgWOBrwGtwEcjYgvwfAPr1quIeKkOT/NN4DTg74GPAy8Ck4Hz8/1ZdahDw0kaCvw38CTw\nfmAJcADwLmDfBlat30TEskbXYcCLCN+a7Eb6kPxB2bF/AZ7L99uBAN5Ykv5a4E5gNemD9tvAhLJz\nnAXMAzYALwA3laTtDczMZVcD/1N2/ueAPyt5fG/ONyQ/PiTX6cD8uBP4Skn+U4FHgXXAS/n840vS\nTwYeANYDTwOfB4b10Ebvz893apX0MfnfQcDlpN7ghvz6TynJ192Wp5E+sNcCvwFOLMkzFLgOWJrP\n8SzwhZLXGaW3fLwt/w0W59c8H/jzsjp2kr6t/z2wPLf91cCgns5d4bVOyemH9PK+2hu4If8t1wOP\nA3+a084GuoATgMeANcCPgYPKztHj3wnYD5iTX/Mi4Jx8vitK8gTwvrLzPgNc3IfHAZwL/Eeu61PA\nB8vO+RbgwVzXh4D35HLTGv1/fHe8eRjNuq0jfejtQNL+wE9I/6nfDLwTGAnMkTQo5zkPuBH4BnAk\n6T/eYzlNpEA1EfgT4Oh8vrn53JCCw7ScfwRpWGMD0H3daBrwZEQsrlC/CcC/AzcBRwB/SOqVdKef\nBHwL+AqpZ3IO8D7Sh3A1ZwALI2J2pcSIWJnvfhy4BPhr4PXArcBsSVPKinyeFFCOAu4H/l3SyJx2\nAfBe4M+AQ4E/BRbktFNJAeVK0tBmd3vtRfqg+5P8mq4FbpR0QoXXsRl4G6lH9ol8/p7OXW4ZsBU4\nTVLF0ZD8N/4hcBzw56QvJxcCG0uyDScNfZ4DvBUYA3y15BxF/k6zSF883glMB84kBfRa+DQpsB0F\nfAf4uqRX57qOBH4APAEcA3wS+FKN6jEwNDra+Vb/G2U9G1IAWQ58Jz9up6RnQ/ow+lHZOfbJed6c\nHy8mfxuv8HzHk77VtpQdfxj4ZL7/f4EF+f47Sd+KZwGX5WO3AF8rKdtJ7tkAb8h1mVTl+X8CXF52\nbHquk6qU+Q0wp0BbLgE+XXasE7ilrC3PK0mfmI+9PT++DvhRD3V5hpJv3T3U5d8rtNF9ZXn+uyxP\n0XP/Jekbfhfpi8HngMkl6SeSAtIRVcqfnV9zR8mxM0hfKFTk7wQcls8xtSR9ErCF2vRsrip5PITU\nK/1gfnweqQfdUpLndNyzqXpzz6Z5/ZGkLknrgftI/9H/X5W8xwB/mPN35VlFz+a010jaj/QB+qMe\nyo8AlpWd43XAa3KeTuCw3NOZRhpi6cz3IX1j7qxy/kdI16Aek/R9SR+VNK7s+f+27Ln/jXSNakKV\nc6rK8VcypEkCBwA/K0u6l/TNvtSjJfeX5n/3y//OIg1VLZR0vaQ/7u4x9vDcgyX9raRHJa3Ir+lU\n4NU9PG/3c+9HH0XE9aS2Op30+k4BHpb0oZzlaNIw7OM9nGZDRCwoebwUGEb64gK9/52OIAW0X5XU\naxGvtGd/29Z2EbGZ1MPrbrvDgcciYl1J/l/WqB4DgicINK+fkMakNwFLI2JTD3kHkYbBKs1aegFo\n6eW5BuV8x1ZIexkgIp6Q9DzwDlKAuZY03PQVSUcAB1Il2ETEFknvAv6AdNH6L4CrJB0XEY/k5/8s\nafy9XLULwwtJH247q3w59W3tGxGRRp3Sl72IeFBSO3AS6ZrGTcAjkk6MiK1Vzn8xcBFpGG8e6dv/\n37NjICn/uwY7OQs1IlYDtwO3S/oUcDeph/PNHgu+YnOFulBSn6J/p96Wqg92/LJQcYi4F/3WduZg\n08zWRsRvC+Z9kHTBfFGVoLRa0hLSB+V/Vyk/HtgaEU/18Dz/A/wx6TpNZ0Qsk7ScNB5e8XpNt0jj\nGPcB90m6knTB/E9JvZ4HgcP78HohfaP+d0mnRoXrNpLGRMRKSUuBqWzfq3s7aRiusPxB/j3ge5Jm\nAb8gXZtYSLruMbisyNuBOyLim7k+3cNMK+mbSucuUt+Q9ARpCBPSBfL9JR3RS++mJz3+nfLzDSIN\n+/48H3s1qXdZahkl158kjaf69aid9QRwlqSWkt7Nm/v5OQYUR2kr4nrSTKPvSHqLpIMlvVPSTEmj\ncp7PA5+Q9FeSDpM0RdJFOe0e0lDTHEnvzr/ZeKukz0oq7e10koLab+OVqaidwAepPoSGpD+Q9ClJ\nb8ofPv8f8Cpe+cC/Ejhd0pWSXifpcEnvk/TFHl7zd0kXhb8l6fJ87kmS/kjSnaRrCZAuCl8s6QP5\ndV9J6sFd3cO5y+t/YS5/hKRDSENVL5Oug0G6nnCspImSxuZjC4ETJL1d0uGki+oHFX3OEpXOXV6/\nKZLm5DZ7raRDJP0F6QL+rTnbj0jDSN+XdFL+G58oaXqlc1bR498pD8HdRZoI8dY8CWMWaXJLqbnA\nX0p6o9LvgGaRZoz1p38jXSv6l9wm7wT+Jqd5k7AKHGysVxHR/e19K+k/+3xSANqQb0TEDaSLyB8h\nzUK7izSjqLvX8R7Sh8C/kGZafRfoYPvx9k5Sb7uzl2PlVuX6/QD4X+Aa4HMRcUt+/rtJPaZ3kMb7\nfwVcCvyuh9ccwAdIw1R/QrqGNA+4itQD+37Oeh0p4Hwxv+73Aqfl4buiVpNmtP2K9O1+CvDuiFib\n0z9NCp5P8spw0t/l/P9JGhJdQ5rJ1VeVzl1uMWnq76dJPa6HSUN4V5Ov8+XhvneTvlTcQprgcS3p\nmkwhBf9OZ5OmRM8F7iB96D9TdqqLcn07Sb3Fr5Gmffeb3BM9mfQef4j0HrgiJ/d3YBsQumeBmJnZ\nLpB0Cqmnt19ELG90fXY3vmZjZrYTJJ1F6kE9S5pZ+WXSdTQHmgocbMzMds540uy5/UnLO91J+nGv\nVeBhNDMzqzlPEDAzs5rzMFo2ZsyYOOSQQxpdjd3OmjVraG1t7T1jk3G77MhtUtlAb5cHHnhgeUSM\n6y2fg002fvx4fv3rXze6Grudzs5Opk2b1uhq7HbcLjtym1Q20NtF0qIi+TyMZmZmNedgY2ZmNedg\nY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2Zm\nNedgY2ZmNedgY2ZmNedgY2ZmNVfXYCNpX0m3SlojaZGk06vkk6QZklbk2wxJKkmfKWmBpK2Szq5Q\n/q8kPS/pZUlflzS8t7o98/JWpn5hLrc9tKTQa7ntoSVM/cJcDrr0zgFZzsysP9W7Z3M9sBEYD5wB\n3CBpcoV85wLTgaOAI4GTgfNK0h8BPgY8WF5Q0knApcAJwCTgYOCzRSq3ZOU6Lps9r9cP5NseWsJl\ns+exZOU6YgCWMzPrb3XbqVNSK3Aa8LqI6ALulXQ78CFScCh1FnBNRCzOZa8BPgJ8FSAirs/H11d4\nqrOAf42I+TnP54BvVXiOitZt2sLf3DqPe3+7vGqeH857jnWbtuyx5b509wKmHz2xajkzs/5Wz22h\nDwM2R8TCkmOPAMdVyDs5p5Xmq9QDqmQyMKes7HhJbRGxojSjpHNJvSiGTThk2/G1G7fw4/nVv/2v\n3RhVju8Z5ZasXEdnZ2fVcqW6uroK520mbpcduU0qc7sk9Qw2I4GXy46tAkZVybuqLN9ISYqIyp+g\nPZclP892wSYiZgIzAYbvf+i2804c08LPLj2+6hNM/cJclqxct8PxPalc0T3RB/r+6TvL7bIjt0ll\nbpekntdsuoDRZcdGA6sL5B0NdBUINNXKUuV5dtAydDCXnNTRY55LTuqgZejgAVvOzKy/1TPYLASG\nSDq05NhRwPwKeefntN7yVVKp7AvlQ2iVTBzTwlWnvr7X6xnTj57IVae+noljWtAeUK6tdRgAY0cO\nK1TOzKy/1W0YLSLWSJoNXCnpw8AU4BTgbRWy3wxcKOmHQAAXAf/UnShpGClQChgqaS9gY0RszWVn\nSfoWsBT4FDCrt/q1jx7U45BUuelHT9ypD+1GlDvqVWN4x9WdXPruIxxozKwh6j31+WNAC/Ai8G3g\noxExX9KxkrpK8t0I3AHMAx4D7szHuv0XsI4UqGbm+38IEBF3AV8Efgz8DlgEfKaGr2m3d+A+LQwe\nJBatWNPoqphZk6rnBAEi4iXS72fKj/+UdGG/+3EAn8y3SueZ1svz/APwD7tS14Fk6OBBHLhPC08v\nd7Axs8bwcjVNYlJbK4tWrG10NcysSTnYNIn2thE8s2INxSb0mZn1LwebJjGprZXV6zfz+7WbGl0V\nM2tCDjZNor1tBADPeJKAmTWAg02TmNTWCuAZaWbWEA42TeJV+7YgwTPLPUnAzOrPwaZJDB8ymAP2\nbvEwmpk1hINNEzlobCvPePqzmTWAg00TmdQ2wtdszKwhHGyaSHtbKyvXbmLl2o2NroqZNRkHmyYy\nKU9/9koCZlZvDjZNpH1smv7sSQJmVm8ONk3k1fu6Z2NmjeFg00T2GjqY/ffeyz0bM6s7B5sm097W\nyjPeasDM6szBpsm0jx3hYTQzqzsHmyYzqa2VFWs28vJ6r/5sZvXjYNNkuld//p17N2ZWRw42TaZ7\n9WdPEjCzenKwaTL+YaeZNYKDTZMZMWwI+40a7hlpZlZXDjZNqL2t1cNoZlZXDjZNqH3sCG81YGZ1\n5WDThCa1tbJs9QbWbNjc6KqYWZNwsGlC7XlGmicJmFm9ONg0oVdmpPm6jZnVh4NNE+oONr5uY2b1\n4mDThEbtNZSxI4e5Z2NmdeNg06QmefqzmdWRg02TSlsNeBjNzOqjrsFG0r6SbpW0RtIiSadXySdJ\nMyStyLcZklSSPkXSA5LW5n+nlKQNl/RVSS9IeknSHZIm1uP17Una20bw/MvrWbdxS6OrYmZNoN49\nm+uBjcB44AzgBkmTK+Q7F5gOHAUcCZwMnAcgaRgwB7gF2Ae4CZiTjwN8HHhrLncA8Hvgn2r0evZY\nk8am6c+/e8m9GzOrvboFG0mtwGnA5RHRFRH3ArcDH6qQ/SzgmohYHBFLgGuAs3PaNGAI8OWI2BAR\n1wECjs/pBwF3R8QLEbEe+A5QKaA1tfZtM9J83cbMam9IHZ/rMGBzRCwsOfYIcFyFvJNzWmm+ySVp\nj0ZElKQ/mo/fBfwrcK2kA4CVpB7Uf1aqkKRzSb0oxo0bR2dnZx9f0p5rzabUfHN/NY/hy56omq+r\nq6up2qUot8uO3CaVuV2SegabkcDLZcdWAaOq5F1Vlm9kvm5TnlZ+nv8FngWWAFuAecD5lSoUETOB\nmQAdHR0xbdq0gi9lYPjUff/F4DETmDbt9VXzdHZ20mztUoTbZUduk8rcLknhYTRJ4yVdLOkGSWPz\nsamSDip4ii5gdNmx0cDqAnlHA125N9Pbea4HhgNtQCswmyo9m2Y3qa3Vv7Uxs7ooFGwkHQMsIA1J\n/QWvfNifCHy+4HMtBIZIOrTk2FHA/Ap55+e0SvnmA0eWzk4jTQboTp8CzIqIlyJiA2lywJu7A6S9\nor1thKc/m1ldFO3ZXA1cGxFHAxtKjt8NTC1ygohYQ+plXCmpVdJU4BTgmxWy3wxcKGlivvZyETAr\np3WShscuyNOcu4fI5uZ/7wfOlLS3pKHAx4ClEbG82EttHu1jW1m6ah3rN3n6s5nVVtFgcwxpinG5\n50jTmIv6GNACvAh8G/hoRMyXdKykrpJ8NwJ3kK63PAbcmY8RERtJ06LPJE0AOAeYno8DXAysJ127\nWQa8B3hvH+rYNNrbWomAxb9378bMaqvoBIF1pN+0lDucFDgKiYiXSIGi/PhPSRf+ux8H8Ml8q3Se\nh0gBsFLaCtJwn/Vi24Kcy9dyyH6V5mmYmfWPoj2bOcBnJA3Pj0NSOzAD+H4N6mV10L2vjX9rY2a1\nVjTYXAzsSxqWGgHcC/yWNIz1qdpUzWptzIihjN5riDdRM7OaKzSMFhEvA2+XdDzwBlKQejAi7qll\n5ay2JNE+1qs/m1ntFQo2ks4EvhMRc3ll1lf3OmV/FhE316h+VmOT2lp55NmVja6GmQ1wRYfRvgHs\nXeH4qJxme6iD2kaw+Pdr2bh5a6OrYmYDWNFgIyAqHH81Oy4dY3uQSW2tbPX0ZzOrsR6H0STNIwWZ\nAP5H0uaS5MHAJOCHtaue1Vr72DT9edGKtRw8bmQvuc3Mdk5v12y+l/99HemHlaU/vNwIPIOnPu/R\nJnn6s5nVQY/BJiI+CyDpGdIEgfX1qJTVT1vrMEYO9/RnM6utolOfKy1VYwOAJCa1jXDPxsxqquiq\nz8MkfVbSQknrJW0pvdW6klZb7W2t7tmYWU0VnY32OfJWzcBW4BLSvjErSItr2h5sUtsInn1pLZu3\nePqzmdVG0WDzfuD/RsSNpOX950TEBcBnSHva2B6sfWwrm7cGS1aua3RVzGyAKhpsxgO/yfe7gDH5\n/l3Au/q7UlZfryzI6aE0M6uNosHmd8AB+f5vgZPy/beSth+wPVh7W/dvbTxJwMxqo2iwuRU4Id+/\nFvispKdJu2d+rQb1sjoaN2o4LUMHe4toM6uZolOfLyu5/z1Jz5K2g14YET+oVeWsPrqnP7tnY2a1\nUnSnzu1ExC+BXwJIao0If0rt4drbWvnfF1c3uhpmNkAVHUbbgaS9JF0CPN2P9bEGmTR2BM++tI4t\nWyutt2pmtmt6DDb5x5yfl3S/pJ9Lmp6Pnwk8BXwC+Mc61NNq7KC2VjZu2cpST382sxrorWdzBXA+\nsAg4CPgPSf8M/C1wGdAeEVfVtIZWF90LcnolATOrhd6CzfuBsyPifcAfkbYV2AeYHBE3RcSmWlfQ\n6qN7qwGvkWZmtdBbsHkVcD9ARDxC2lZgRkRs7rGU7XHGj9qL4UMGeUaamdVEb8FmKLCh5PEmvDPn\ngDRoUPfqzx5GM7P+V2Tq81WSuj+BhgFXSNou4OR10mwPN6mt1T0bM6uJ3oLNT4DXlDz+OfDqsjye\nKztAtLeN4CcLl7F1azBokBpdHTMbQHrbqXNanephu4FJba1s2LyV519ezwFjWhpdHTMbQHb6R502\n8Bw0tnv1Zw+lmVn/crCxbSZtW/3ZkwTMrH/VNdhI2lfSrZLWSFok6fQq+SRphqQV+TZDkkrSp0h6\nQNLa/O+UsvJvkPQTSV2SXpD08Vq/toFg/71bGDZ4kHs2Ztbv6t2zuZ70W53xwBnADZImV8h3LjAd\nOAo4EjgZOA/SEjrAHOAW0g9MbwLm5ONIGkva1O1GoA04BPiv2r2kgWPwIPGqfVtY5K0GzKyf1S3Y\nSGoFTgMuj4iuiLgXuB34UIXsZwHXRMTiiFgCXAOcndOmkSY2fDkiNkTEdYCA43P6hcDdEfGtnL46\nIh6v2QsbYNrbWt2zMbN+V2iLAUnl0527BbA+IpYVOM1hwOaIWFhy7BHguAp5J+e00nyTS9IejYjS\nKdeP5uN3AX8AzJP0c1Kv5pfAX0bE78qfRNK5pF4U48aNo7Ozs8DLGNgGr9vAU8s28+Mf/xhJdHV1\nuV0qcLvsyG1SmdslKbqfzTP08HsaSS8D3wA+2cNSNiOBl8uOrQJGVcm7qizfyHzdpjyt/DwHAm8A\nTgTmAV8Evk3a7G07ETETmAnQ0dER06ZNq1L15vG74c/wX4vmM/mYt7Lf6L3o7OzE7bIjt8uO3CaV\nuV2SosHmA6QP7a+SN00D3kLqFVwBjAE+BawGPlPlHF3A6LJjo3OZ3vKOBroiIiT1dp51wK0RcT+A\npM8CyyXtHRFeaqcX7Xn156eXr2G/0Xs1uDZmNlAUvWbzUeCvIuKqiJibb1cBFwHnRMS1wAWkoFTN\nQmCIpENLjh0FzK+Qd35Oq5RvPnBk6ew00iSC7vRH2b4X5hUO+qDdWw2YWQ0UDTZvIQ1JlXsMeFO+\nfx9pCKuivHX0bOBKSa2SpgKnAN+skP1m4EJJEyUdQApqs3JaJ7AFuEDScEnn5+Nz87/fAN6bp0cP\nBS4H7nWvppgDxuzFkEHyJAEz61dFg80i8oX0Mh8Bui+8jwNe6uU8HwNagBdJ11E+GhHzJR2bh8e6\n3QjcQQpwjwF35mNExEbStOgzgZXAOcD0fJyImAv8TS7zImmSQMXf89iOhgwexKv2HeGejZn1q6LX\nbC4Cvi/pPeT9bYA3khbpPC0/fhPw3Z5OEhEvkQJF+fGfki78dz8O4JP5Vuk8DwHH9PA8NwA39FQX\nqy5tNeCejZn1n0LBJiLuzNdaPgZ05MO3A1/tnlIcEf9cmypavbW3tfLrZ37P9rPLzcx2XtGeDRHx\nLHBZDetiu4lJbSPo2rCZFWs2NroqZjZAFA42kkYAU4D9KLvWExGz+7le1kDdM9KeWe6hNDPrH0VX\nEHgn6YJ+W4XkAAb3Z6Wssdq3bTWwlrENrouZDQxFZ6NdS5rddWBEDCq7OdAMMBPHtDB4kLxFtJn1\nm6LBph34XEQsrWFdbDcxbMggJo5p4RlPfzazflI02PyMV2ahWROY1DbCPRsz6zdFJwh8Fbg6/5p/\nHrCpNDEiHuzvilljtbe1ctvDS4gY1uiqmNkAUDTYfC//O7NCmicIDECT2kawev1m1mxysDGzXVc0\n2BxU01rYbqd7+vMLa7c2uCZmNhAUXUFgUa0rYruXJ5elpeo+94v1fP2JuVxyUgfTj57Ya7nbHlrC\nl+5ewNKV6zhgTMuALbdk5Tom/sLtYlZU1WAj6VTgjojYlO9X5R91Diy3PbSEf7znlQ1Vl6xcx2Wz\n06LfPX3w3PbQEi6bPY91m7a4XBOVMytC1da/krQVmBARL+b71cRA+K1NR0dHLFiwoNHV2C1M/cJc\nlqxct8PxoYPFaw/Yu2q53yxdxaYtO76fXG7PLjdxTAs/u/T4quW6eUfKygZ6u0h6ICLe2Fu+qj2b\niBhU6b4NfEsrBBqATVuCMS1Dq5ar9EHlcnt+uWrvB7O+KLw2mjWPA8a0VOzZTBzTwk3nvLlquWo9\nIpfbs8sdMKalahmzogr3WCQdKOl0SZ+QdGHprZYVtPq75KQOWoZuPzLaMnQwl5zU8+96Xa45y5kV\nUXQhzjOArwObgWWk39Z0C+Af+r9q1ijdF4O3zboqOCuptFxfZjPtieUGcrt0TxIo+vrMiqg6QWC7\nTNKTwHeAyyNiS81r1QCeIFDZQL+4ubMGcrvc0PkkM+56gkc+/S72HlH9Gk+5gdwmu2Kgt0vRCQJF\nh9HGA18bqIHGzF5x+IRRACx4YXWDa2IDSdFg80PgLbWsiJntHjq6g83zLze4JjaQFJ2N9t/ADEmT\nqbwQp3/UaTZA7L/3XozeawhPPO+ejfWfosHmxvzv31RI80KcZgOIJA6fMJoFDjbWjwoNo1XYndM7\ndZoNYB0TRrHg+dUUmUBkVkSvwUbSUEm/lOTJ9mZNomPCKFZv2FzxR55mO6PXYBMRm0hbDPgrjlmT\n2DYjzUNp1k+Kzka7CfhILStiZruPw3Kw8SQB6y9FJwi0AmdIOhF4ANhuc/qIuKC/K2ZmjTN6r6FM\nHNPino31m6LB5gjgwXz/4LI0D6+ZDUCH50kCZv2h6E6d76h1Rcxs99IxYRT/s3AZGzdvZdgQ7zJi\nu8bvIDOrqGPCKDZvjW1bhJvtir5sMfAOSTMl3SVpbumtD+fYV9KtktZIWiTp9Cr5JGmGpBX5NkOS\nStKnSHpA0tr875QK5xgm6XFJi4vWz8xecfiE0YBnpFn/KBRsJJ0N/CcwCphG2mZgH+ANwG/68HzX\nAxtJC3ueAdyQl8Apdy4wHTgKOBI4GTgv12UYMAe4JdfhJmBOPl7qklxPM9sJB49rZehgeUaa9Yui\nPZuLgfMj4gOkddEui4ijSR/4hfrYklqB00jbFHRFxL3A7cCHKmQ/C7gmIhZHxBLgGuDsnDaNdK3p\nyxGxISKuAwRs2yRd0kHAB4GrCr4+MyszdPAgXjNupBfktH5RdDbawcA9+f4GYGS+/xWgE7i0wDkO\nAzZHxMKSY48Ax1XIOzmnleabXJL2aGy/jsaj+fhd+fE/kdZx6/Hnz5LOJfWiGDduHJ2dnQVeRnPp\n6upyu1TQLO2yj9bzyKJir7VZ2qSv3C5J0WCzgjSEBrAEeB3pA74NKLpB+Uig/CvSqpLzluddVZZv\nZL5uU5623XkkvRcYHBG3SprWU4UiYiYwE9LmaQN5g6OdNdA3ftpZzdIuj/Mk9931BEe/eWqvG6k1\nS5v0ldslKTqM9lPgXfn+d4HrJH0D+DZp+4EiuoDRZcdGA5UGhMvzjga6cm+m6nnyUN0XAf/I1Kwf\neCM16y9Fg835pMAC6TrIl0i9mu8CHy54joXAEEmHlhw7CphfIe/8nFYp33zgyNLZaaRJBPOBQ4F2\n4KeSngdmA/tLel5Se8F6mlnmjdSsvxT9UedLJfe3AjP6+kQRsUbSbOBKSR8GpgCnAG+rkP1m4EJJ\nPyStUHAR6ToMpGtEW4ALJH2VV9ZsmwtsBV5Vcp63ka4rvQHPTDPrs/333otR3kjN+kFffmczXtLF\nkm6QNDYfm5pnfhX1MdI1nhdJPaWPRsR8ScdKKp3VdiNwB2lX0MeAO/MxImIjaVr0mcBK4BxgekRs\njIjNEfF89w14CdiaH2/pQz3NjO6N1Lxsje26Qj0bSccAPwKeJs36+hKwHDiRNMus4o8zy+Ue0vQK\nx3/KKzPcyNdmPplvlc7zEHDCffxwAAANq0lEQVRMgefrBA4sUjczq+zwCaO57eElRATbj16bFVe0\nZ3M1cG3+bc2GkuN3A1P7vVZmttvomDCK1es3s3TV+kZXxfZgRYPNMaRf6pd7jrQagJkNUN0z0p54\nzpMEbOcVDTbrSEvDlDucdP3FzAYob6Rm/aFosJkDfEbS8Pw48lTiGcD3a1AvM9tNeCM16w99WRtt\nX9L04RHAvcBvSb/c/1RtqmZmu4sOz0izXVT0dzYvA2+XdDzpNyuDgAcj4p6eS5rZQHD4hFH8xBup\n2S4oujYaABExl/TjSQAkTQK+FBHv7++Kmdnuo3sjtaeWd23b58asL3b1K8oY0rYBZjaAdQeYJ57z\nUJrtHPeHzaxX3kjNdpWDjZn1yhup2a5ysDGzQjwjzXZFjxMEJN3eS3lfKTRrEodPGM2ch5eyat0m\n9m7peSM1s3K9zUZbUSD96X6qi5ntxrqXrVn4wmre1L5vg2tje5oeg01E/Hm9KmJmu7eOkjXSHGys\nr3zNxswK8UZqtiscbMysEG+kZrvCwcbMCuuYMIoFL6wm7W9oVpyDjZkVdviE0d5IzXaKg42ZFdY9\nI80/7rS+crAxs8K6N1J73GukWR852JhZYd5IzXaWg42Z9YmXrbGd4WBjZn3SMWEUTy7rYuPmrY2u\niu1BHGzMrE8OL9lIzawoBxsz65PujdQ8lGZ94WBjZn3SvZGaZ6RZXzjYmFmfeCM12xkONmbWZ56R\nZn3lYGNmfdYxYRRLV61n1bpNja6K7SHqGmwk7SvpVklrJC2SdHqVfJI0Q9KKfJshSSXpUyQ9IGlt\n/ndKSdolkh6TtFrS05IuqcdrM2smR+RJAgtfcO/Giql3z+Z6YCMwHjgDuEHS5Ar5zgWmA0cBRwIn\nA+cBSBoGzAFuAfYBbgLm5OMAAs7MaX8EnC/pz2r1gsya0baN1DyUZgXVLdhIagVOAy6PiK6IuBe4\nHfhQhexnAddExOKIWAJcA5yd06aRdhj9ckRsiIjrSAHmeICI+GJEPBgRmyNiASkwTa3hSzNrOts2\nUnvOkwSsmB63he5nhwGbI2JhybFHgOMq5J2c00rzTS5JezS231Dj0Xz8rtKT5KG3Y4EbK1VI0rmk\nXhTjxo2js7Oz6GtpGl1dXW6XCtwusH/LVn61YDGdnSsAt0k1bpeknsFmJFD+NWgVMKpK3lVl+Ubm\n4FGe1tN5riD13r5RqUIRMROYCdDR0RHTpk3r8QU0o87OTtwuO3K7wD0r5zHn4aUcd9xxSHKbVOF2\nSep5zaYLGF12bDRQadC3PO9ooCv3ZgqdR9L5pGs3fxwRG3ah3mZWQYc3UrM+qGewWQgMkXRoybGj\ngPkV8s7PaZXyzQeOLJ2dRppEsO08ks4BLgVOiIjF/VB3MyvjjdSsL+oWbCJiDTAbuFJSq6SpwCnA\nNytkvxm4UNJESQcAFwGzclonsAW4QNLw3IMBmAsg6Qzg74ETI+KpWr0es2bnGWnWF/We+vwxoAV4\nEfg28NGImC/pWEmlS8jeCNwBzAMeA+7Mx4iIjaRp0WcCK4FzgOn5OMDfAW3A/ZK68u2rtX9pZs2l\neyO1J7xGmhVQzwkCRMRLpEBRfvynpAv/3Y8D+GS+VTrPQ8AxVdIO6pfKmlmvvGyNFeXlasxsp3kj\nNSvKwcbMdpo3UrOiHGzMbKd1bJuR5qE065mDjZnttIPHjmToYHlGmvXKwcbMdtqwId0bqTnYWM8c\nbMxsl3RMGOUFOa1XDjZmtku6N1Jbsyl6z2xNy8HGzHZJ97I1S7o8/dmqc7Axs13SkXftXLzawcaq\nc7Axs11yQN5IzcHGelLX5WrMbOCZ8/BSNmzaytxntzL1C3O55KQOph89sddytz20hC/dvYClK9dx\nwJiWAVtuycp1TPzFwG2XYRMOqbh0WDkHGzPbabc9tITLZs9j45bUq1mych2XzZ4H0OMHVne5dZu2\nuNwAKFeEtt9duXl1dHTEggULGl2N3Y53GazM7ZJM/cJclqxct8Px4UMG8ZaD26qW++VTK9hQYT01\nl9vzyj130yfY8Nz/qmrmzD0bM9tpSysEGoANm7fy8rpNVctV+oBzuT2/XE8cbMxspx0wpqViz2bi\nmBZu+8upVctV6xG53J5drieejWZmO+2SkzpoGTp4u2MtQwdzyUkdLtdk5Xrjno2Z7bTui8jbZl0V\nnM1UWq4vs6D2xHIDvV2e6zHnKzxBIPMEgcp8Ibwyt8uO3CaVDfR2kfRARLyxt3weRjMzs5pzsDEz\ns5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pz\nsDEzs5qra7CRtK+kWyWtkbRI0ulV8knSDEkr8m2GJJWkT5H0gKS1+d8pRcuamVn91btncz2wERgP\nnAHcIGlyhXznAtOBo4AjgZOB8wAkDQPmALcA+wA3AXPy8R7LmplZY9Qt2EhqBU4DLo+Iroi4F7gd\n+FCF7GcB10TE4ohYAlwDnJ3TppH24flyRGyIiOsAAccXKGtmZg1Qz83TDgM2R8TCkmOPAMdVyDs5\np5Xmm1yS9mhsvxHPo/n4Xb2U3Y6kc0k9IYANkh4r9lKaylhgeaMrsRtyu+zIbVLZQG+XSUUy1TPY\njAReLju2ChhVJe+qsnwj87WX8rTy81QtWxagiIiZwEwASb8usgFQs3G7VOZ22ZHbpDK3S1LPazZd\nwOiyY6OB1QXyjga6crDo7Tw9lTUzswaoZ7BZCAyRdGjJsaOA+RXyzs9plfLNB44sm2F2ZFl6tbJm\nZtYAdQs2EbEGmA1cKalV0lTgFOCbFbLfDFwoaaKkA4CLgFk5rRPYAlwgabik8/PxuQXK9mRm319V\nU3C7VOZ22ZHbpDK3C6B6ji5J2hf4OnAisAK4NCL+TdKxwH9GxMicT8AM4MO56NeAv+4eCpN0dD72\nWuBx4C8i4qEiZc3MrP7qGmzMzKw5ebkaMzOrOQcbMzOruaYPNkXXa2s2kjolrZfUlW8LGl2nepN0\nvqRfS9ogaVZZ2gmSnsjr8/1YUqEftg0E1dpFUrukKHnPdEm6vIFVras8Yelf8+fIakkPS3p3SXrT\nvmfAwQaKr9fWjM6PiJH51tHoyjTAUuDvSJNatpE0ljSz8nJgX+DXwHfqXrvGqdguJcaUvG8+V8d6\nNdoQ4FnSqih7A58CvpuDcLO/Z+q6gsBup2S9ttdFRBdwr6Tu9doubWjlrOEiYjaApDcCB5YknQrM\nj4j/yOlXAMslHR4RT9S9onXWQ7s0tfzzjitKDv1A0tPAMUAbTfyeAfdsqq3X5p5NcpWk5ZJ+Jmla\noyuzG9lu/b38IfMkft90WyRpsaRv5G/0TUnSeNJnzHz8nmn6YNOX9dqazV8DBwMTST9Ku0PSaxpb\npd1Gb+vzNavlwJtICzMeQ2qPbzW0Rg0iaSjptd+Uey5N/55p9mDTl/XamkpE/DIiVudtHG4Cfga8\np9H12k34fVNB3jrk1xGxOSJeAM4H3iWpaT5QASQNIq2MspHUBuD3TNMHm76s19bsgrRvkJWtv5ev\n/b0Gv2/Kdf9ivGk+Z/IKJv9KmnB0WkRsyklN/55pmjdBJX1cr61pSBoj6SRJe0kaIukM4A9J+wU1\njfza9wIGA4O72wO4FXidpNNy+qdJeyw1xYXeau0i6S2SOiQNktQGXAd0RkT58NFAdgNwBHByRKwr\nOd7U7xkAIqKpb6RpiLcBa4DfAac3uk6NvgHjgPtJXfyVwC+AExtdrwa0wxWkb+eltyty2juBJ4B1\npMVh2xtd30a3C/AB4On8f+k50qK4Expd3zq2y6TcFutJw2bdtzOa/T0TEV4bzczMaq+ph9HMzKw+\nHGzMzKzmHGzMzKzmHGzMzKzmHGzMzKzmHGzMzKzmHGzMBqi8t8z7Gl0PM3CwMasJSbPyh3357ReN\nrptZIzT1fjZmNXYPaW+kUhsbURGzRnPPxqx2NkTE82W3l2DbENf5ku7M2wQvkvTB0sKSXi/pHknr\nJL2Ue0t7l+U5S9K8vEXzC5JuKqvDvpL+I297/lT5c5jVi4ONWeN8FrgdmELaM+jmvPtl96rAd5PW\n1noz8F7gbZRsxSzpPOBG4BvAkaQtIB4re45PA3NIKw5/B/i6pFfX7iWZVea10cxqQNIs4IOkRRlL\nXR8Rfy0pgK9FxEdKytwDPB8RH5T0EeBq4MCIWJ3TpwE/Bg6NiN9KWgzcEhEVtzDPz/GFiLgsPx5C\n2izw3Ii4pR9frlmvfM3GrHZ+Apxbdmxlyf37ytLuA/443z+CtAR96eZaPwe2Aq+V9DJpF9Uf9VKH\nR7vvRMRmScuA/YpV36z/ONiY1c7aiPhtDc7bl+GITWWPAw+fWwP4TWfWOH9Q4fHj+f7jwOvLtlR+\nG+n/7OMR8SKwBDih5rU06wfu2ZjVznBJE8qObYmIZfn+qZLuJ22k9T5S4HhLTvsWaQLBzZI+DexD\nmgwwu6S39HngHyW9ANwJjABOiIhravWCzHaWg41Z7byTtGNlqSXAgfn+FcBppO2TlwF/HhH3A0TE\nWkknAV8GfkWaaDAH+Hj3iSLiBkkbgYuAGcBLwA9r9WLMdoVno5k1QJ4p9n8i4nuNrotZPfiajZmZ\n1ZyDjZmZ1ZyH0czMrObcszEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5r7/wH9yRB6gLhoJQAA\nAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2TpBQrfzRwNV",
        "colab_type": "text"
      },
      "source": [
        "### Performance Scheduling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9AGPIj3rRwNW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgxLyFOvRwNX",
        "colab_type": "code",
        "outputId": "0f9ea5a0-f795-4acb-b79c-1393188706bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "optimizer = keras.optimizers.SGD(lr=0.02, momentum=0.9)\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
        "n_epochs = 25\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid),\n",
        "                    callbacks=[lr_scheduler])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 6s 104us/sample - loss: 0.5968 - accuracy: 0.8075 - val_loss: 0.5433 - val_accuracy: 0.8212\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.5250 - accuracy: 0.8356 - val_loss: 0.4772 - val_accuracy: 0.8558\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 5s 100us/sample - loss: 0.5208 - accuracy: 0.8423 - val_loss: 0.5324 - val_accuracy: 0.8468\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 6s 101us/sample - loss: 0.5383 - accuracy: 0.8440 - val_loss: 0.9670 - val_accuracy: 0.8304\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.5281 - accuracy: 0.8503 - val_loss: 0.5353 - val_accuracy: 0.8492\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 6s 101us/sample - loss: 0.5198 - accuracy: 0.8496 - val_loss: 0.5268 - val_accuracy: 0.8482\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 6s 101us/sample - loss: 0.5156 - accuracy: 0.8552 - val_loss: 0.8378 - val_accuracy: 0.8270\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 6s 101us/sample - loss: 0.3124 - accuracy: 0.8918 - val_loss: 0.4112 - val_accuracy: 0.8736\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 5s 100us/sample - loss: 0.2564 - accuracy: 0.9077 - val_loss: 0.4131 - val_accuracy: 0.8828\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2394 - accuracy: 0.9119 - val_loss: 0.3803 - val_accuracy: 0.8894\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 6s 102us/sample - loss: 0.2197 - accuracy: 0.9189 - val_loss: 0.4208 - val_accuracy: 0.8902\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 5s 100us/sample - loss: 0.2072 - accuracy: 0.9228 - val_loss: 0.4173 - val_accuracy: 0.8886\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 6s 118us/sample - loss: 0.1946 - accuracy: 0.9264 - val_loss: 0.4700 - val_accuracy: 0.8824\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 6s 115us/sample - loss: 0.1878 - accuracy: 0.9289 - val_loss: 0.4384 - val_accuracy: 0.8816\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 6s 107us/sample - loss: 0.1788 - accuracy: 0.9347 - val_loss: 0.5007 - val_accuracy: 0.8892\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 7s 125us/sample - loss: 0.1260 - accuracy: 0.9515 - val_loss: 0.4424 - val_accuracy: 0.8940\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 6s 104us/sample - loss: 0.1090 - accuracy: 0.9567 - val_loss: 0.4580 - val_accuracy: 0.8954\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.0994 - accuracy: 0.9611 - val_loss: 0.4659 - val_accuracy: 0.8950\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.0936 - accuracy: 0.9635 - val_loss: 0.5127 - val_accuracy: 0.8920\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.0878 - accuracy: 0.9655 - val_loss: 0.5208 - val_accuracy: 0.8968\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 6s 115us/sample - loss: 0.0689 - accuracy: 0.9738 - val_loss: 0.5228 - val_accuracy: 0.8962\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 6s 105us/sample - loss: 0.0629 - accuracy: 0.9761 - val_loss: 0.5397 - val_accuracy: 0.8940\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.0589 - accuracy: 0.9782 - val_loss: 0.5459 - val_accuracy: 0.8942\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 5s 99us/sample - loss: 0.0561 - accuracy: 0.9789 - val_loss: 0.5476 - val_accuracy: 0.8910\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 6s 101us/sample - loss: 0.0532 - accuracy: 0.9805 - val_loss: 0.5685 - val_accuracy: 0.8910\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BfOkaIEnRwNZ",
        "colab_type": "code",
        "outputId": "3bd204bb-7a3e-491c-fdc4-b4e0050f8828",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "source": [
        "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Learning Rate\", color='b')\n",
        "plt.tick_params('y', colors='b')\n",
        "plt.gca().set_xlim(0, n_epochs - 1)\n",
        "plt.grid(True)\n",
        "\n",
        "ax2 = plt.gca().twinx()\n",
        "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
        "ax2.set_ylabel('Validation Loss', color='r')\n",
        "ax2.tick_params('y', colors='r')\n",
        "\n",
        "plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcsAAAEeCAYAAADl8T7bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXl4VdXVuN8VwhAIM0IiSiJWtOAA\nzhMG62w/W6efHfhUqpWqrVqt2lo/60CtYrWtdcZWsYJDa0VxrK0SBRGsM6KCioBCwhAQSJiT9ftj\nn0NObu5w7s0dw3qf5zz33j2dfXaGddfeaxBVxTAMwzCM2BTlegKGYRiGke+YsDQMwzCMBJiwNAzD\nMIwEmLA0DMMwjASYsDQMwzCMBJiwNAzDMIwEmLA0titE5HQRMX+pNCMio0RERaRfrudiGJnAhKWR\nd4jIRO8fr4rIVhFZLCL3iEjvXM8tXXjP+Gyc+oWBNdggIp+IyBUiItmcZ2A+1YH5bBKR+SLyaxHp\n0MYx70znPA0jU5iwNPKV/wDlQCXwY+Ak4O5cTigH3IBbg28CtwK/A8bmcD4PevPZHfgz8Fvg8hzO\nxzCyhglLI1/ZpKq1qvqVqr4EPA4cG2wgIj1FZIKILBeRdSLyqojsH9HmLBFZJCLrPU1uQET9dSLy\nYUTZGBGpjyg7UURme1penYg8IyJdvLpOIjJeRL7y7vNfETkuDWuwzluDhar6F+CDyDWIREQ6i8if\nRGSZiGwUkVkicnig3t8uPcp7nvUi8paI7BtiPusD87kTeBk4OcY8+orIo96abBCRuSLyo0D9RKAK\n+GlAY6306oaKyHPez3S5N05ZoO8BIvKSiKwUkbUiMkNEDom4v4rI6RFlC0XEhLuREiYsjbxHRAYD\nxwNbAmUCPAcMBP4HGAG8BrwiIuVem4OAicAEYDjwDE5bS/b+xwNTgX8D+wFHAq/S/PfzIO4f/w+B\nPYGHgGdEZJ9k7xXj/iIio3Aa5pYEzW8Bvgecg1uTOcCL/poEuAn4FbAvUAdMTmGLdwPQMUZdF+Ad\n3M9mGHA7cJ+IHOXVXwK8QbO2Wg586c3zNeBD4EDgaKAUeFpE/PXuDjwMjPTavAc8LyJ9k5y/YYRH\nVe2yK68unIDbCtTj/iGrd10aaPMtr74kou97wJXe+0eAf0fU/8X92m/7fB3wYUSbMUB94PPrwGMx\n5ror0AQMiih/Crg7wTM+G6d+IbDJe8bN3vNvAA6N06eb1/asQFkH4HPgt97nUd5YxwXaHOaV7RRn\n7GrgTu99Ee7LyyZgfMS4/eKM8Rjwl2hjBspuAF6OKOvtjX1gjHEFqAH+N1CmwOlR1vTyXP9+21WY\nl2mWRr7yGk4bPBC4A3ged07msx/QFVghIvX+hdPsdvXafBOnvQSJ/ByGEbgtx2jsi/tn/VHEPL4d\nmEeq/AG3BlXANOB6VZ0Zp/2uOE3vdb9AVRtxzzw0ou0HgfdLvdf+CeYz1nu2jThNexJwfbSGItJB\nRK4WkQ+8bet64FRgUIJ77AccEbGWXwaeDxHpLyL3eUZGa4B13twTjW0YKVOc6wkYRgzWq+pn3vuL\nRWQacA1OEwSn3SzDbcVFsjaJ+zThhF2QWFuL0SjCaTEH0HqLdEMS40SjzluDz0TkNOBTEZmtqtNS\nGCvSXWZLlLpEX54fxwnHTcBSTxDH4nLgF7jt1jk4Dfl3JBbIRbjt9Whni8u814dwZ8+X0qyBvwx0\nCrRV2vZzNYwWmLA0CoXrgRdEZIKqLsWdhw0AmlR1QYw+HwMHR5RFfl4BDBARUVVfaAyPaPMucBRw\nf5R7vIv7p1yWohALhaqu9tws/igiIwJzDfI5bhv2MO89nmvHIbgt6bayJvAFJhGHA8+o6sPePAQY\nAnwdaLMZt00c5B3gDGCRqsY6nz0cuFhVn/PGHoA78wyyIlgWo41hhMa2YY2CQFWrgY+A//OK/oPb\nbnxaRE4QkV1E5BARuV5EfG3zz8DRInKViOwmIucBp0QMXQ30AX4tIruKyLnA6RFtbgT+n4j81rPU\nHCYil4pIV1WdD0wGJooLeDBYRPYXkctF5NQEj9VDRIZHXJVx2t+Nc9v4fzHWqAG4Bxgvznr3m97n\nAWTf7WY+cJSIHC4iewB3ArtEtFkIHCgilSLSzzPguQvoCTwuIgd563m0OKvn7oGx/9f7WRyAOwvd\nHDH2KzhL2/1FZATujHhjJh7U2D4wYWkUErcB54pIhadZnYj7p3g/MA/4O06YLAVQ1VnAucAFuDO6\nU2nexsVr87FXP9ZrcwxuuzDY5nmckD0Bp0m+irOIbfKa/Ahn1XkL8AnwLHAEsCjB84z0xgtet8Zq\nrKrLcVag1wUsQyP5JW679EGcsdPewPGqWpNgLunmt8CbwAu48+cG3JeKILfihNxHOE1wkLdrcBhu\nbV8E5uIE6CbvAmfpWwq8jROUD+AEb5BfAAtwX4aewBl2LU/TsxnbIRJ9N8cwDMMwDB/TLA3DMAwj\nASYsDcMwDCMBJiwNwzAMIwEmLA3DMAwjAeZnGZKioiItKSnJ9TTyiqamJoqK7PtWJLYu0bF1iU57\nXpeSkhLq6upWquoOuZ5LWzFhGZJOnTrR0NCQ62nkFdXV1YwaNSrX08g7bF2iY+sSnfa+LiLSNddz\nSAft8+uMYRiGYaQRE5aGYRiGkQATloZhGIaRABOWhmEYhpEAE5aGYRiGkYCsCksR+ogwRYQGERaJ\n8MMY7USE8SLUedd4EZebToQhIjwtwgoRVonwLxF2j+h/qQi1IqwV4QEROgfqKkWYJsJ6ET4R4egw\nc9+0qQOVlTA5MhR0DCZPhspKKCoia/0O2KmGV6WKA3auzej9/D7f+lZVVp7NMAwj56hq1i7QR0Ef\nBy0FPRx0DeiwKO1+AjoPdCfQgaAfgZ7v1R0Iei5oH9COoONAPwn0PQ50Gegw0N6g1aA3B+rfAP0D\naAnoaaBfg+6QeO5dFVS7dlWdNEnjMmmSawfNVzb63cUFupUivZMLM3a/bD9bITJt2rRcTyEvsXWJ\nTntfF6BBsyhnMnVlLeuICN2A1cCeqsz3yh4Glqjyq4i2M4GJqkzwPp8LnKfaKnEvIvQB6oB+qtSJ\n8AiwUJVfe/VHAZNVKRNhCC5rez9V1nn10736e+PPv5u6LEPQsydcfHHstn/+M6xZ07o8k/26rlnC\nQnahE1tYTwmDWcDGnmVpv1+6n62iAhYujN2vEGnvfnOpYusSnfa+LiKyXlW75XoebSWbwnIE8Loq\nXQNllwNVqpwU0XYNcKwqs73P+wPTVOlOBCKcDNyj6rKgi/A+8DtVHvc+98PlyuuHyzH4O1W+Geh/\nJ07BvijK2GNxeQ6Bbvv5whIUkdjP6pY0WoPM9XuC0ziNKQBspBN/5cf8jDvTfr90P5uI8sorr8bu\nWIDU19dTWlqa62nkHbYu0Wnv63LkkUe2C2GZzS3YkaC1EWXngVZHadsIukfg827e1p1EtNsJdAno\nDwJln4MeH/jc0etbCXom6KyIMW4EnZh4/l23bR9WVETbbGimokJbbDdmut/+A5fqRjq26NRAie6/\nU03a75ftZytE2vu2WqrYukSnva8L7WQbNpsGPvVAj4iyHuC2QxO07QHUq7JNDRZhB+Al4G5VHk3Q\nF+8+ycwhKl27wo03xm9z442uXbb6Td5jHB1oalFWRCOTdh+X9vtl+9kMwzDygWwKy/lAsQi7Bcr2\nAeZGaTvXq4vaToTeOEE5VZXIf7fR+i5Tpc6rGyzSYjs31hxaUVEBEybA6NHx240e7dpVVIBI5vsN\nqXuDYhpblHVhM7vXzUz7/Vr20aSfbdAg97l793D9DMMw8oJsqrGgj3kWsd1AD4tjDXs+6MeeJeyO\noHMD1rA9QN8EvTPGPY4HrQUdCtoL9JUIa9hZoLeCdgE9Jaw1bOfOnRPvN+SSc891+5onnJC1W6a6\nfTR0qOopp6R3LvlEe99WSxVbl+i093XBtmFT4kKgBFgOPApcoMpcEUaKUB9odx/wDM5y9UPgOa8M\n4BTgAOBHItQHrkEAqrwI3AJMAxYDi4BrA2N/H9gfZ5l7M3C6Kisy8rTZpKam5WseU1nZ/ixgDcNo\n32Q1RZcqq4CTo5RPB0oDnxW40rsi2z4EPJTgPn8A/hCjbiEwKolpFwa1tS1f85iKCpg1K9ezMAzD\nCI+Fu2sv+EJy+XJobIzfNsdUVsKqVbB2ba5nYhiGEQ4Tlu2BxkZYtgx694amJliR37vKlZXuddGi\nnE7DMAwjNCYs2wN1dU5gjhjhPuf5uWVFhXs1YWkYRqFgwrI94AtHX1jm+bmlr1makY9hGIWCCcv2\ngC8cC0Sz7N8funQxzdIwjMLBhGV7wBeWw4e3/Jyn+AEQTLM0DKNQMGHZHvA1yV12cek/8lyzBPO1\nNAyjsDBh2R6orYUePVyw1fLyghCWFRW2DWsYRuFgwrI9UFsLZWXufVlZ3m/DgtMsV6yAhoaETQ3D\nMHKOCcv2QE1Ns7AsEM3St4hdvDin0zAMwwiFCcv2QG2tE5LQrFlqdpJ6p4rva2nnloZhFAImLNsD\nkZrl+vWwLnSKzpxgvpaGYRQSJiwLnYYGJxh9zdJ/zfNzy7Iy6NTJjHwMwygMTFgWOsuWudeggQ/k\n/bllUZFLBG2apWFs54j0QWQKIg2ILELkhzHa9ULkIUSWe9d12ZxmVlN0GRnAF4rBbVjIe80SzNfS\nMAwA7gI2AwOA4cBziLyP6tyIdn8EugKVQH/gZUQWofpgNiZpmmWh4wvFoIEP5L1mCeZraRjbPSLd\ngNOAa1CtR3UGMBU4M0rrk4BbUF2P6kLgr8A52ZqqCctCJ1Kz7NMHOnYsCGFZWelk/caNuZ6JYRiZ\noh8UI/JW4BobqB4CbEV1fqDsfWBYjOEk4v2eaZ5uTGwbttCprYUOHaBfP/dZpGACE/juI4sXw5Ah\nuZ2LYRiZYaUThvvHqC4FItPArwG6R2n7IvArRM7Gbdmeg9uWzQpZ1SxF6CPCFBEaRFgkQtSDXBFE\nhPEi1HnXeJHmbxQiTBBhnghNIoyJ6HuvCPWBa5MI6wL11SJsDNTPy9gDZ4PaWhgwwFnM+BRYYAI7\ntzSM7ZZ6oEdEWQ8gmu/bxcAG4FPgaeBR4KuMzi5Atrdhgwe5o4F7RKKq22OBk4F9gL1xe9U/CdS/\nD1wIvBPZUZXzVSn1L9yC/iOi2c8CbXZv60PllKCPpU+BaJYmLA1ju2c+bpt2t0DZPkCkcQ+orkJ1\nNKplqA7Dya83szPNLApLEbYd5KpSr0q8g9yzgdtU+UqVJcBt0KxBqnKXKi8DcU+7Avd8KD1PkYcE\no/f4FIhmueOOUFxsRj6Gsd2i2gA8CdyASDdEDgO+Czzcqq3Iroj0RaQDIifglKrfZmuq2TyzHAJs\nVSXyILcqStthXl2wXawD33icBqwAXosov0mEm4F5wNWqVEfrLMJY3A+E4mKhujpqs5xyyOLFrCor\nY15gbhWbNrHLypW8+p//oMWZ+xHX19e3eU122OEgZs9eS3X1x+mZVB6QjnVpj9i6RMfWhQuBB4Dl\nQB1wAapzERkJvIBqqdduP+BPQC+cRjo6intJxsimsEzmILfUqwu2KxVBVEkm6OnZwN8i+vwS+Ai3\nHfx94BkRhqvyeWRnVSYAEwC6dFEdNWpUErfOAo2NsHo15SNGUB6c27x5MHEiVXvsATvtlLHbV1dX\n09Y12X132LixhFGjBqRnUnlAOtalPWLrEp3tfl1UV+GO3SLLp+Nkgf/578DfszavCLJ5ZpnMQW5k\n2x5AfTKCUoRBwCjgb8FyVWarsk6VTao8BLwOnBh23Lyirs4JzGjbsFAw55Z2ZmkYRr6TTWE5HygW\nIfFBrivbJ0S7eJwJvK7KggTtlJa+O4VDpI+lTwEFJqishKVLYfPmXM/EMAwjNlkTlqpsO8gVoZsI\nsQ9ynTZ4mQgDRdgR+AUw0a8UoZMIXXBCrqMIXURaPctZwT5ev14iHOe1LxZhNHAEzn+n8IiM3uPj\nfy4AYVlR4bKJffllrmdiGIYRm2y7jlwIlOAOch8FLlBlrggjRagPtLsPeAaYA3wIPOeV+byE87c5\nFHemuAEn9AAQ4RBgJ1q7jHTEWU+tAFYCFwEnRxgdFQ6+sIzULAcMaFmfx5j7iGEYhUBWI/ioEvUg\nV5UWB7ne2eSV3hVtnFEJ7vMG0C1K+QrggKQmnc/E2obt1An69i0IzdKEpWEYhYDFhi1kamuhRw/o\nGiXiU4EEJhg40AUfMl9LwzDyGROWhUy06D0+BRKYoGNH591imqVhGPmMCctCprY2vrAsAM0SLFWX\nYRj5jwnLQiZaqDufsjKnWWoyMRxyg/laGoaR75iwLGQSbcNu3gxff53dOaVAZSV89RVs2ZLrmRiG\nYUTHhGWh0tAA69bF1yyhIM4tKyqgqQmWLMn1TAzDMKJjwrJQWbbMvcbTLCH/hGVNDVRVtThPNfcR\nwzDyHROWhUosH0sfvzzfjHzGjYMZM9yrR0WFezUjH8Mw8hUTloVKrFB3PvmoWdbUwAMPuD3XBx7Y\n9gw77wwiplkahpG/mLAsVBJplj16QJcu+aVZjhsHW7e6942N27TLzp1dImgTloZh5CsmLAuV2lro\n0AH69YteL5JfgQlqauDBB52QBGf6+uCD24S5+VoahpHPmLAsVGprXcD0ojg/wnwKTDBunNt+DRLQ\nLs3X0jCMfMaEZaESz8fSxw9MkA+88UbrpJWbN8PMmYDTLL/8slnxNAzDyCdMWBYq8aL3+OSTZvnu\nu+68sthLdHPOOS660LvvAk6z3LrVJYI2DMPIGCL9ERmDyEHJdDNhWaiE1SxXr4aNG7Mzp0R89VWz\ngU+Exmu+loZhZASR5xH5ufe+G/AW8GdgBiKjww5jwrIQaWyE5csTC0tf88wX7XLBAvfarVsrYWm+\nloZhZIj9gVe896cCDcAOwE+IkTM5GiYsC5G6OicwE23D5ltgAl9YHnxwK2E5aJB7Nc3SMIw00wNY\n7b0/FpiC6ibgP8A3wg4SWliKMECEy0W4R4R+XtlhIuySxKSNdJDIx9In3wITLFjgziwPPBBWrGhh\nzVNS4h7HhKVhGGlmMXAIIl2B44B/e+W9gfVhBwklLEXYD5gHjAbOxUlqgGOAG8PezEgTiaL3+OSj\nZllR4bI9NzW5reQA5mtpGNshIn0QmYJIAyKLEPlhjHadEbkXkWWIrELkGUQGhrjDn4BJOKG5DHjV\nKz8C+DDsNMNqlrcCt6syAtgUKP8XcFjYm4nQR4QpIjSIsEiEqIsigogwXoQ67xovggTqJ4gwT4Qm\nEcZE9B0jQqMI9YFrVKC+UoRpIqwX4RMRjg47/7zBF36JNMv+/V1wgnzSLAcPjqnxmq+lYWyX3AVs\nBgbgFLJ7EBkWpd0lwCHA3sCOuK3VOxKOrno3MBK4ADgUVd/hexHwm7CTDCss9wMeilJeg3vAsLRa\nFBGiLcpY4GRgH9zCnIQ7jPV5H7gQeCfGfd5QpTRwVQfqHgXeBfoCVwNPiLBDEs+Qe8JuwxYXO4GZ\nT5rl4MEx04dVVMDixa1jFxiG0U5x1qmnAdegWo/qDGAqcGaU1rsA/0J1GaobgcchqvxojepsVP+B\n6jrvvh1QnYrq9LBTDSssN+D2dyPZA1gepbwVImxbFFXqVYm3KGcDt6nylSpLgNugWYNU5S5VXgaS\n8okQYQiwL3CtKhtU+Scwx5tX4VBb62K/du2auG2+BCZYuxZWrkyoWW7enD+y3TCMjDME2Irq/EDZ\n+0QXgn8FDkNkR+/8cTTwQsI7iFyIyKmBz/cBGxGZi8huYSdaHLLd08C1Ivw/77OKUAmMB/4Zcowh\nwFZVIhelKkrbYV5dsF24bxCOESKsBFYBDwM3qbLVG2OBKuvCjC3CWJyWS3GxUF1dncQUMsfQ99+n\ntGdP3gwxn706d6bjp5/yTgbmXl9fH3pNSj/7jP2BDzdsYNX8+RwBfDFzJou+0WyMtnZtH2Bvnnzy\nHfbcc23a55stklmX7Qlbl+i093XpB8WIvBUomoDqBO99KRD5x74G6B5lqE+BL4ElQCNO0flZiClc\nCvwYAJGROCF7NnAKThH7TqgHUdWEF2gP0Bmga0EbQZeAbgV9FbRbyDFGgtZGlJ0HWh2lbSPoHoHP\nu7lwLyoR7WaAjokoGwy6C2gR6F6gH4Fe5dWdCTorov2NoBMTzb9z586aNxxxhLvCMGaM6sCBGZnG\ntGnTwjf+5z9VQfWdd9zn3r1VL7ywRZO5c12TRx5J3xxzQVLrsh1h6xKd9r4uQIPG+t8KIxTWR5T9\nQuGZKG0nKUxR6KPQWeEahdkxx27ut0FhZ+/9LQoTvfdDFVYm7O9dobZhVVmryuG4c8RfArcDx6tS\npUpDKKkM9TRb0fr0gBZaXqy2PYB693wJ57pAlS9UaVJlDnADcHoKc8hfwoS68ykvh2XLcn8Q6PtY\nDh7sXqNkRPEDE5iRj2FsN8zHaZ7B7dB9gLlR2g4HJqK6yvOTvAM4EJEYqZe2sQ622aUcg/OvBGc/\n0yXsRMO6jpwlQmdVXlHlVlVuUeU/InQS4ayQ95oPFIsQZlHmenWJ2oVBYZsl7VxgsEgLFb8tY+eG\nMKHufMrKXIi5urrMzikRCxZAnz7Qs6f7HEVYduvmMo6Z+4hhbCeoNgBPAjcg0g2Rw4Dv4o7PIvkv\ncBYiPRHpiDPyXIrqygR3+Tdwn3dWOYTmc86hwMKwUw1r4PMg0DNKeXevLiGeBvokcIMI3USItyh/\nAy4TYaAIOwK/ACb6lZ6Q7oITgh1F6CLinkWEE0Scha4IewDX4M5c8c5L38Odv3YR4RSctW3Yc9fc\n09AA69Ylp1lC7o18fEtYnxiGR+Y+YhjbHRcCJThj0UeBC1Cdi8hIROoD7S7HGXV+CqwATsSdOybi\np7h4sDsBZ6Dqaw4H4CxqQxHWwEcg6hboINxhbFguBB7ALUodcIEqc0UYCbygSqnX7j5gMO4AF+Av\nXpnPSzQbBh0KTACOBKqBo4CJIpTiHFAnAb8L9P0+TvCuxjmpnq7KiiSeIbeE9bH0CQYm2HvvzMwp\nDAsWwIgRzZ99zVLV+YJ6VFbCnDmtuxuG0U5RXYU74ossnw7bZAKekAsd+DzQ72ucj2Vk+TXJDBNX\nWIowByckFXhVhK2B6g5ABfB82JupEnVRVGmxKN7Z5JXECHKr2hxkIErd5bhvILHqF0Ls/nlPssIy\nHzTLxkanLp4W8NApL3d+IqtXu+1Zj4oKePbZVjLUMAwjdUQ64RSloTh5Nhf4O6qb4/YLkEizfMJ7\n3RN4Dmcg47MZt99bOFuY7YGwoe588iHk3ZIlsGVLy23YYEaUgLCsrHQZxZYvhwHJhLswDMOIhsge\nwItAH5rtU34KjEPkeFTnhRkmrrBU5Xp3LxYCj6smFwTAyABho/f4lJa6K5eaZaQlLLTUeIcO3VYc\ntIg1YWkYRhq4HRcD9n+9LVkQ6QVM9uqODzNIWNeRh0xQ5gm1tdChgzMbDUt5eW41y0TCMoCfBNos\nYg3DSBOHA7/aJijBP8e8yqsLRVjXkU4iXC/CfBE2eoHKt13JztxoA7W1TuUqSiIVaa5D3i1Y4AT8\nzju3nBOYr6VhGJlmE63968F5c4Q+swz7H3ccXrxWoAm4AhcUvQ5n4Wpki2R8LH2i+DRmlc8/d1Kw\nOLDr36OHS2IZMa8ePaB3b9MsDcNIG88BExA5CPFyWokcDNwLPBN2kLDC8gzgfFXuw8Xke1qVi4Fr\ncRERjGyRTPQen7Ky3G/DBrdgwZm6xhDi5mtpGEYauRiXjusNnJ/mRuB1nIHqz8MOElZYDgA+8t7X\nA7289y8Cx4a9mZEGUtUs161zAQ1yQTRhCTHPUk1YGoaRNlRXo/ptnNvI971rGKonobo67DBhheVi\nXLJNgM+A47z3h+DSdxnZoLHR+VQkKyxz6T4STM0VSQzNsqLCbcNqwkjAhmEYIVH9BNUp3vUJIt9A\nZGbY7mGF5RRcZBxwprbXi/AFLhLOX5KasJE6dXVOYCa7DZvLwARffOFed921dV2cbdiGhtyHszUM\no13TDTgobONQ4e5UuSrw/gkRvgQOA+ar8mzSUzRSI1kfS59gAIBsE81txKeszGme69e3SGTtW8Qu\nWpSch4xhGEamSML/oBlVZqvyB1WeFaFbuidlxCDZ6D0+Mdw0skI8YZnA19LOLQ3DyBdSEpYAXtaO\nK4Av0jgfIx6papb9+jk/x1xplr17Q69eretMWBqGUSAkCqTeCececiywBbhFlae8HJY34wLS/jHj\nszQcyQZR9ykqcoEMcqVZRtMqIaaw7NXL+Vuar6VhGCkj8i7Rs2X5dI1T14pEZ5bX4QLO/ht3RvkP\nEe7HGftcBTyiypZkbmi0gdpaJ0W6JvUzduQqMMGCBbDPPtHr4pylmvuIYRhtJK32NImE5RnAGFWm\niLAP8C7QGxim2iJdl5ENUvGx9Ckrc9k/somfmuuUGPlZ+/VzUX3iuI8YhmGkRJL5KhOR6MxyZ+C/\n7r68j4ujN94EZY6orU1dWOZCs1y61OWsjLUNG2d72NcszdfSMIx8IJGw7IgLQuuzBViTuekYcUkl\n1J1PWRmsWOG0vWwRzxLWJ0aQ98pK51Xy9detuxiGYWSbMH6WN4mw3nvfCbhOpKXA9OLEGpmmpgZO\nOCG1vuXl0NTkIgClKnCT5fPP3Ws8YVleDosXtyoO+lr27p2BuRmGYSRBIs3yNWBXYC/vmgkMCnze\nC9gz7M1E6CPCFBEaRFgkwg9jtBMRxotQ513jRZBA/QQR5onQJMKYiL5ni/C2CGtF+EqEW0SavxSI\nUO2lGav3rlBZsnNOQ4OL75qqoMtFYIJoqbkiiRPFB8zIxzCM/CCuZqnKqDTf7y7cuecAYDjwnAjv\nqzI3ot1Y4GRgH5zp779x/pz3evXvA48D46PcoysukvxsYAdgKnA5ztXF52eqBRamL1W3EZ9gYIIR\nI9Izp0QsWACDBkHHjrHblJe72LFbtrRoZ8LSMIx8IlS4u3TgRfo5DdhTlXpghghTgTOBX0U0Pxu4\nTZWvvL63AefhCUtV7vLKN0beR5V7Ah+XiDAZODLNj5N92iosc6VZxtuCBTcvVbc9PHDgtuI+faBb\nN7OINQwjDYjsCBwO9CdyR1XSjTb3AAAgAElEQVT1z2GGyJqwBIYAW1WZHyh7H6iK0naYVxdsNyzF\n+x4BrTTXm0S4GZgHXK1KdbSOIozFabkUFwvV1VGbZYV+r77KnsB/v/qKhhTmUbR5M0cAC2bOZHEi\nARaS+vr6uGty6Lx5rDzsMObHadN35Ur2At5+9lnW7b57i7oddjiAt95aT3V15I8vv0m0Ltsrti7R\nsXXJMCLfxyX9AFhJy0AFCoQSlqhqVi7QkaC1EWXngVZHadsIukfg825O/VCJaDcDdEyce54D+hVo\nv0DZQaDdQTuDng26DnTXRPPv3Lmz5pQ77lAF1WXLUh+jVy/Vn/40bVOaNm1a7Mq1a918b745/iCz\nZrl2U6e2qjrxRNURI9o2x1wQd122Y2xdotPe1wVo0CzJmagXfKZwi0LHtoyTcmzYFKgHekSU9QDW\nhWjbA6h3zx0OEU4GbgJOUGWlX+4FgV+nyiZVHsJlzD4x7Lg5o7bWGcu0JQ1HWVn2tmH91FxhtmEh\nrq+lYRjtGJE+iExBpAGRRYhENfxE5AVE6gPXZkTmhLhDGXAvqm2KNpdNYTkfKBZht0DZPrTeIsUr\n2ydEu6iIcDxwP3CSKokWU6HZ0jZvqalxDvxFbfiRZTMwQRgfS4ibEaWyElavdv6WhmG0W4KGn6OB\nexBpfeymegKqpdsu553xjxDjvwgc0NZJhjqzFGFQjCoFNqqyItEYqjSI8CRwgwg/xlnDfhc4NErz\nvwGXifC8d49fAHcE5tMJJ+gF6ChCF2CzKk0ifAuYDJyiypsRz9ELl+zzVWAr8D3cmeYlieafc9oS\nvcenrAxmz07PfBIRVlh26gR9+0bVeIO+lnvtleb5GYaRe0S2GX6iWg/MQCSW4WewXyUwElq6Dsbg\nBeAWRL4JzIGIeOaqU8NMNayBz0LiRG8XYS3wIHClxg+FdyHwALAcqAMuUGWuCCOBF1Qp9drdBwyG\nbVrhX7wyn5doNgw6FJiAs3itBq4BegLPS7O+OF2VE3ARiX4L7AE0Ap8AJ0cYHeUnbYne4+Nrlqog\nGVamFyxw6UPCRBQI4WtpwtIwCpN+UIzIW4GiCahO8N4PAbaiGsbwM8hZwHRUF4aYwv3e62+i1CnQ\nIcQYoYXlD4BbcK4bvmpyEM5S9DqgF/B/uPPHa2MNosoqnP9kZPl02CYo8c4mr/SuaOOMinOPmG4i\nngbcZnU8J6TDP7K8HDZscMENekQeH6eZMG4jwXnFCKYO5j5iGIXMSicM949RXQpEHrSsAbonGPYs\nnOIThjiO3uEJKywvAC5V5clA2Ste9JtLVKkSYTlwPXGEpZEijY3ODzEd27DgBFM2hGVYdbCsDD75\npFVx//7QpYsZ+RhGOyYZw0+HyOE4o50nQt1BNS0BscNaixwEUQ1lPqRZU3sD2CkdkzIiqKtzAjMd\n27CQeYvYpiZnDZuMZllb2yrFiIhZxBpGO2c+bps2jOGnz9nAk94ZZzhEjkPkFURqEalB5GVEjk1m\nomGF5SI85/wIzgP8KNg7AKuSubkREn+LMp2aZSZZsiR+aq5IystduLu6ulZVltfSMNoxqg3gDD8R\n6YbIYTjDz4ejthcpweVZnhj6HiI/wiWCXoLb+bwOqAGeRWRM2GHCbsP+AvinCCfi5bcE9scFWT/N\n+3wA8PewNzaSwNcE06VZZlpYhrWE9QlqvBF+pJWV8Pbb6ZuaYRh5RyvDT1TnIjISeMFzE/E5Gfga\nmJbE+FcBl6N6e6DsPs/o6CpCCt5QmqUqzwG74YKS9/CuqcDuqjzvtblblctCT98IT7o0y969natG\nprdhUxWWMYx8Vq50SVcMw2iHqK5C9WRUu6E6CNVHvPLpEYISVB9FtcILxxaWCuC5KOXPenWhCB0b\nVpUvcVLYyDZtDaLuIxIz2XJaWbDABU8YFMs9N4IEUXzAbcUOHZqe6RmGsV3xJXAU8FlE+dFeXShC\nC0sRuuICCbSK2h5hJWukm9paZ73atWvbx8pGyLswqbmChBCWCxeasDQMIyX+APwZkeG4qD8Ah+EC\nGlwadpCwEXyOBh4F+kapDu3UaaRITU3btUqf8nL4/PP0jBWLZHwsAUpLXT4u87U0DCPdqN6NyAqc\n7Y0fd/ZjYDSq/ww7TFhr2Ntxe747qVIUcZmgzDTpCHXn47tpZJJkhSXEDExQVuaOWc19xDCMlFH9\nB6oHo9rTuw5ORlBCeGFZCYxTZWnSkzTaTk1N2y1hfcrKnMXM5s3pGS+S+noXQGHXXZPrF0OIFxU5\n7dKEpWEYuSSssHwd2D1hKyMzpFuzBCfQMkHY1FyRxMmIYr6WhmEkhcgqRPp571d7n6NfIQlr4HMv\ncKsIOxIlarsq74S9oZEkDQ0ulms6NUtwgmmnDARcStZtxKe8HJ5/PmpVZSU880zbpmUYxnbFFTSH\nzLuCOIlAwhJWWPox+CZEqTMDn0ySLrcRn0yHvGuLsKyvd1dpS9eqigpYtszFgC8pSdM8DcNov6j+\nNfD+L+kYMuw27C5xriT/KxpJkW5hmemQdwsWQM+e4VJzBUmQBBpg8eJWVYZhGPERmY9InyjlvRAJ\nnZ4xlGapip0Y5Yp0hbrzGTDAvWZSWA4enHy+zKCv5W67tagK+lrubifnhmEkxzeILus6k44IPiKc\nCjyjyhbvfUwsKEEGSVeoO59OnaBv38xtw37+Oey5Z/L94mwPm6+lYRhJI/KdwKfjEFkT+NwBF9Vn\nYdjh4mmWT+Byhi0nft4wO7PMJLW10KFDqwDjbSKO5Wmb8FNzfec7idtGmxNEndeOO0JxsbmPGIaR\nFE95rwo8FFHXiMuY1fYIPqrN55nB90aWqalxW6dFafwRZCowwdKlyaXmCtK3rwuPF0VYdugAO+9s\nmqVhGEnRERDgC1xWrBXbalJICJ1VIShCHxGmiNAgwiKRbaGHItuJCONFqPOu8SJIoH6CCPNEaBJh\nTJT+l4pQK8JaER4QoXOgrlKEaSKsF+ETL5Rf/pJOH0ufTAVTT9USFuIGeZ882cnhRx5x55eTJ4cb\ncvJk176oKLv9vvWtqoKYZ7b6GUZOUG1EdSuqO6Na631uTEVQeuNpqAt0J9Afgv4c9LLglcQYj4I+\nDloKejjoGtBhUdr9BHSed8+BoB+Bnh+o/ynoUaBvgY6J6Hsc6DLQYaC9QatBbw7UvwH6B9AS0NNA\nvwbdIdHcO3furDlh331Vv/3t9I555ZWqnTqpNjW1aZhp06a1LHjwQVVQ/fTT1AY84ADVY45pUTRp\nkmrXrm5Y/+ra1ZXHw/rlR78grX5fDFVt/+sCNGhIGZGxC3oqnKFwucKvW1whxwgbSH00LjnnVpwq\nG3TwVFxU90RjdMMlit5TlXpghghTgTOBX0U0Pxu4TZWvvL63AefhgiOgyl1e+cYotzob+Ksqc702\n44DJwK9EGALsCxyrygZcQuufe/O6N9Ez5ISaGhgxIr1jlpW57dLVq6FPa4vqlEk2NVck5eXN2qnH\n1VfD+vUtm61fDxdcAO/ECYVx//3WLxv9rr4aRo+O3c8wco7IAcALQBPQB6jB2eNsxKXo+l2YYcIG\nJbgBuA24RpXUVFgYAmxVJejX8j5QFaXtMK8u2G5YyPsMA56O6DtAhL5e3QLVbZEd4o4twlhgLEBx\nsVBdXR1yCmmisZGqZctYvGkTX6Tx3v1Xr2Yo8ObUqaz3/TJSoL6+vsWafPONN+i5ww7Mmjkzdqc4\nDFGl35dfMjMw5uLFVUBrN5R165R77on9q7hhQwfrl4V+ixcr1dWvxuwXJPL3xXDYumSc24DHgIuA\ntTiZswGXSSu8khRG/QStBx3cFjUYdCRobUTZeaDVUdo2gu4R+Lybt/UjEe1mRNmG/Rz0+MDnjl7f\nStAzQWdFtL8RdGKi+edkG7a21u133XlnesedNs2N+5//tHGYaS0LDjlE9cgjUx/wuuvcvDZt2lZU\nUdFy68+/KiriD2X98qNfkPa+3Zgq7X1dyPU2LKxR2N17/7XCN733ByrMDztOWAOf54GDQkvg6NQD\nPSLKekALLS9W2x5AvXu+pO/jv1+X5BxyT7qj9/jEcdNoE6mk5griz2vZsm1FN97YOud1166uPB7W\nLz/6GUYesJnmo8NlgH9OtBYIHSA7rLD8NzBehN+K8D0RTg1eIceYDxSLEAzPsg+4s8UI5np1idpF\nI1rfZarUeXWDReie4tjZJd3Re3x84ZtO95GGBifkkk3NFSSKEB89GiZMcIEJRNzrhAmJz8ly208L\nZJ7J9/OPo0tLw/UzjDzgXWB/7/2rwA2IjAb+BHwQepQw6idoU5yrMawaC/oYziK2G+hhxLaGPR/0\nY5wl7I6gc2lpDdsJtAvo695WbhfQIq/ueNBa0KGgvUBfoaU17CzQW70+p5DP1rC+dennn6d33KYm\n1ZIS1csua9MwLbaP5sxxc33ssdQHfPNNN8ZTT7VpXrmmvW+rnXCC6tChyfdr7+uSKu19Xcj9NuyB\nCkd57/sr/FthvcJ7CvuEHSeUZqlKUZwrmeg9FwIluKhAjwIXqDJXhJEi1Afa3Qc8g0sH9iHwnFfm\n8xLugPZQXCaUDcAR3lxfBG4BpuEiNCwCrg30/T7uW8Zq4GbgdNWAs2o+kaltWJH0ByZoi4+lT6a2\nh420UlUFH32UuZSohpFWVN9E9WXv/XJUj0G1K6rDUX0/Qe9tJLSGFaEjMAM4S5V5KU8YUGUVcHKU\n8ulAaeCzAld6V7RxRiW4zx+I4c6iykKI3z9vqK2FHj1aHxalg3QHJkiHsBwwwAlyE5Z5TZVnv/7a\na3D66bmdi2Fki4TCUl0g9V2g7ckzjSSpqUm/VulTXu7Ug3Tx+edOsLfFb7NjRxcDN1NB3o20sN9+\n0K0bvPqqCUsjTxH5lLAyS3VImGZhDXwewgUFMLJJJkLd+WRCs0wlNVckmQrybqSNjh3hsMOcsDSM\nNiPSB5EpiDQgsgiRqGFQvbb7IvIaIvWILEPkkhgt/wL81bseBQYAS3BJQZ4AvvLKHg07zbBBCboB\no0U4BngbaAhWqnJx2BsaSVBTA/vum5mxy8vh669h40bo0qXt4y1YAEOHpmdeJizznqoqF71n5cr0\nJsQxtkvuwrl3DACGA88h8j6qLb0URPoBL+IyhTwBdCKW64fq+EC/B4FbUR0XMd7/QQvvjLiE1Sy/\nCbyDM4oZDOwVuFJIXmiEItOapX+PtuKn5mrLeaWPCcuCwD+3nD49t/MwChwRPwzqNajWozoDtoVB\njeQy4F+oTkZ1E6rrUP04xF1OJboG+ThwStiphtIsVTky7IBGmmhogHXr0u9j6RO0PG1DyLttY2za\nlB5hWVbm/DWbmtKblsxIKwccACUlbiv2lND/boztkX5QjMhbgaIJqE7w3g8BtqIaJgzqwcAcRGYC\n3wBmAz9FdXGCKfjeEp9FlI8E1rduHp2w27BGtsmU24hPOjXLdFjC+pSXw9atbn+vf/+2j2dkhE6d\n4JBD7NzSSMxKJwz3j1FdioukE2QNtAgc47MTLhHGMTi3wltwGuNhCaZwO3A3IvsCs7yyg4FzgN8m\nfACP0MJShCOBH+BCBXUK1qnyrbDjGCHJtLBMp09juoUluOc3YZnXjBoF117rktf07p3r2RgFSjIh\nSDcAU1D9LwAi1wMrEemJ6pqYd1C9CZFFwCXAWV7px8CPUX0k7ERD7XN5CZZfwEn7Ubg0Xb1xUj6N\n/gfGNnwhlqlt2P793TZnujRLP3ZaW7HABAVDVZULp27nlkYbmI/bpg0TBvUDaJUeMhyqj6B6EKo9\nvOugZAQlhDfwuRz4mSo/ALYAV6kyApgELSLvGOki05plhw6www7p0yx33tntzbUVE5YFw4EHQufO\nthVrtAHVBuBJXLzWbogcBnwXeDhK6weBUxAZjkhH4BpgRlytMo2EFZaDgf947zfRHG3nTmBMmudk\ngBOWHTpk1i4/XSHv2pptJIj/5cCEZd7TpQscfLAJS6PNtAqDiupcREYi0qyMqb4C/BoX/nQ5zsgn\nuk+myCrP1QREVnufo18hCXtmWUfzgesSnLvIB0Bf7yGNdFNT48K/ZdIiNF2BCRYsgG9/u+3jgAsN\n0727CcsCYdQoGDcO1qyBnj1zPRujIFGNGgYV1RZhUL2ye4B7Qox6Bc3nnpe3bYKOsMJyOnAszgLp\n78CfvQAFR+HSdxnpJpM+lj7l5TBnTtvGWL/ezTVdmiWYr2UBUVUF118PM2ak7/uSYbQZ1b9Gfd8G\nwqotP6PZqfMm4Pc4rfLvwI/TMREjgtrazBn3+AR9GlMlnZawPunOiGJkjIMPdkfVthVrtKKmBqqq\nKIY2xsDMD8IGJVgVeN8EjI/T3EgHNTUwYkRm75EOn8ZMCcv//jd94xkZo6TEGfqYsDRaccMNMH06\nO0LHrN9bZDXhA6mHyv6QjJ/lAFwIol2Ba1RZKcJhwFJVvgg7jhGCxkaXLDDT27DBwAT5Jixrapxf\nQlsDsxsZZ9QouOkmF3CqezRXcmP7QhX+9je47z5QpW9ugt+k5ZwySKiHEGE/4GXgC2AYbht2JS6S\nwhBiWSQZqbFypROYmd6GDbpp7L13amMsWOD+Q/btm955rV/v/vv2iPRXNvKNqir47W/h9dfh+ONz\nPRsjZ6jC88/DddfBW83R7XLydTdN55RBwp5Z3grc7vlWbgqU/4vEoYaMZMm0j6VPMFpOqqQrNVcQ\ncx8pKA45BIqLbSt2u0UVXnjBHWD/z/+4/ycdm3de28veUFhhuR8up2UkNbi0KkY68YVXNgx8oG1C\nKZ0+lj4WmKCg6NbNBVY3YbmdoQr/+pf7tnTiic5Y8P77nVl0Ph2fiHRE5BpEPvLyYG5ucYUkrLDc\ngAtvF8keOOfQUIjQR4QpIjSIsEgk+vatCCLCeBHqvGu8SPMXFBGGi/C2COu91+GBuhdEqA9cm0WY\nE6hfKMKGQP1LYeefNXwhkWnN0vdpTFWzTGdqriDp0HiNrFJV5WyyGhoStzUKHFX497/h8MPdvntN\njTufnD8ffvxjmD0bNoeWQdngBuA8XN7MDsDVuOTQa3DxYkMRVlg+DVwrQmfvs4pQibOK/WfYm9Ey\nyedo4B4RhkVpNxbnpLoPsDdwEvATABE6efOZhBPgDwFPe+WocoIqpf4FzAT+ETH+SYE2xyYx/+yQ\nrW1Y/x4panCdVq1yyaNNs9zuGTXKGVbPnJnrmRhx8dw5kv4i6vf7xz/giCPg2GNh8WK45x749FMY\nO7Y53OW77zqB6l1vJ5EGK0N8D/gJqncBW4EnUb0QuB7Cp59MJjZsH1wA9a7ADFxusDXA/4UZQIRt\nST5VqVclXpLPs4HbVPlKlSXAbTSH1RuFM0z6kyqbVPkzblu8VeYTT6CPBP4W6injsMvmzdnTdGpq\nnGFL166Zv1cbfBpLfGGWbmHZu7cLOmrCsmA49FAXndG2YvOcceNcBIlx42K3UXWa4ddfw5IlThie\ndx689hqccYbbTbrrLvjsMzj//PTEhM4sZTQHZq8HennvnweOCztIWD/LtcDhInwLl2mkCHhHdVu8\n2DAMAbaqEibJ5zCvLthuWKDuA9UWPjQfeOUvRoxzFjBdlYUR5ZNFKALeBa5QbXGvbYgwFqflsh/K\nkvPP59Of/zzG46WPoR98QGnPnrxZXZ35exUVUfrZZyndq+cXzmNo9ooVbEjzXA/u1Yuv33uPT7Kw\nBummvr6e6gKcd1vZbbd9mTq1iaOPfi9q/fa6LolIZV061dUx9IYb+Ojaa9ncJ7abYNHmzZQsWULJ\nl1/Sfd48Bj32GNLUhN5zD2tefZWixkaKNm2iw6ZNFG3cSAfvkhiBSpo6dGD2rbeyqawM3ngjqTnn\nkC+BcmAx8DnOi+Nt4EBgY9hBkvJ/UeUV4BX/swgVwO9VOSNE92SSfJZ6dcF2pd65ZWRdvHHOonVy\nz9HAOzht9BLgXyLsocrXkZ1VmQBMANhf0IEvvcTAe+/N/PbotdfCLrswatSozN4H4Kmn4K23UrrX\nwokTQYSDzjjDaYLpZJddKGtqoiwba5Bmqqurs/OzyzNOOgluvx0OPHBU1E2R7XVdEpHSulx4IXz4\nIYe+/DLccQd89RXMm+fODYOvixY5TTECAXqtWuVcxrp1c7tY/hX5+fHHnVa5dStFHTpwyPTpTrMs\nHKbiBORs4A5gEiLn4nIz/zHsIG11Fu2F21oNQzJJPiPb9gDqVVGRcOOIcDhO/X4iWK7K64GPN4lw\nNm6r9pmET9DY6LYvMv2LUlMD++6b2Xv4lJVBfb07rH/iiaS+CHRZuhR22in9gtKf1/z5idsZecOo\nUfD738OsWfAtSwefOWbPdlanTU3uzPCBB5ztgE9pKey+u7NSHTMGhgyBPn3g5JOb26m6bdaJE+P/\nzdfUwCWXuANpcNuzDz4I11yTHZuKdKB6ReD944gsAQ4F5qP6VNhhMpjSohXzgWIRwiT5nOvVRWs3\nF9g7aB2LMwKKHOds4EnVhPk2lbCuQP4vSqbPLpcudecK2Tgj9Y1pZs6Mf44RhZKamvSfV/pYfNiC\n4/DDXZIcO7fMAFu3wtNPwwknOH9GX3iBE4z33QfTprn/HWvXuqAAjzzidql+8APXN3Jr1f/yH49x\n41Lrlw+IHB21XHUGqrckIyghi8JSlW1JPkXo5oXKi5Xk82/AZSIMFGFH4BfARK+uGmgELhahswg/\n88qD28MlwBmBPn75IBEOE6GTCF1EuALoBy20zfhs3ZrZX5SGBnctXZqdX8guXdyrqvuD+81voLra\nRc9J1HXpUth118zMq7wcVq2CTZsStzXygh49XDhjE5ZpZOlSF2N1l12cZvjuuy4ChI+q24H5znec\nal9eHt3H8Y03WrtzbN6c2Hw51X75wUuILEDkakQGtnWwbGqWECXJpypzRRjpba/63IfbFp0DfIhL\n9nkfgCqbcW4lZwFfA+cAJ3vlPid7ddMi7t8dlwttNS4v5/HACarUhX6CLVtcXK9M8aJno6SaHS32\nlVecGSM0f2M88kiXnHDoUDjrLHcmMmtWy62ezz+n86pVsMMOmZmX+VoWJFVVrX9VjCRpanJ+jKed\nBoMGOe1w6FCYMsUJzMgct2E0vQh3jm3Xu+9mpl9+MAynoF0ELETkOUROQaRDSqOpaswLdGqCqxq0\nMd4Y7eXq3Lmz6oMPul+VCRM0I9TVqZaWNv9KduqkeuGFmbmXqurSpapdurT8M+jSRfXhh1Wvv171\nf/5HdcCA5rriYtXhw1XPO091//1d2dFHZ2Zuzz7rxn/jjcyMn0GmTZuW6ynkjKlT3Y+turp13fa8\nLjFZulRX7723ak2N6sqVqr//veo3vuEWsW9f1SuuUP3ss+b2w4dHE12uPE8BGjSX/7+hWOFUhWcV\ntijUKoxX2D2ZcRIZ+CTSuOpgO8o4cvbZ7kD8yivdtseANEb6U4Xvf98Z2/hk+jA92nlEU5PbevGN\nmFSdr9V//+uut95y1nFrPcPm6dMzk6jaAhMUJCNHul3AV191WuZ2Q02N+/t9/PHk/hZuuIGec+a4\nxVq0yB07HH64C0Z+2mnNxyQ+haHR5ReqLhABPInIjjif/R8BlyPyOqpHhBkmrrBU5UdtnWe7QgTu\nvRf22QcuvdQdoKeLP/7Rbb106OC2VXwyaYEb5jxCxFm87rQTnHKKK7vgAvjrX92WtGpm5mfB1AuS\nXr3cn8d2dW6pCr/8pfvieMEFzoF/3Tr3xXfdutjv6+rg/fcR9c4dx4yBX/wC9twz10/UflFdisjd\nOO+J60giEUgu8owVNnvsAVddBddf7zTN40IHgIjNrFnuj61nT1gT4UKaycP0VL6l1tQ47XrLFvc5\nU9pv//5OUNuZZcFRVQUTJjglKRNeRRklkYao6pIHvPNO8/XWW84YDZzf8lNRjCw7d3ZxmLt3d64d\n3bu7VHwibsxOnZxPownKzOGsY8/B2bRsxNnN/CVs92wb+LQPrrrK+S5dcIHLu9gWVq1yf5w77QQL\nF+b/YXq2TMmLi53ANM2y4Kiqgg0b3K59wREMB9fYCB99BJMmwWWXOWvT3r3hG99wYd9uu80JvP79\nm43kiovd9uncuW5bddUq94Vy40ZYscIJ2g8+cD7NK1c2/y1lyy1te0NkECLXIvIF8BIuks9YYEdU\nf4pq6H+uJixToXNn52bxxRcu622qqMKPfuTMw//+d7eHle9k05S8vNyEZQFyhHcCVHBbsTU17nih\nqckdt3TvDsOGwZlnOuf/jRvhhz90AQHeftttpT77rPuS6x+dbN3qEiD36eMsWXv3bpHbcRuF7L9Y\nKIj8B1iAS8LxGDAE1SNRnYRq0vbaJixTZdQod8bw+9/Dhx+mNsaf/gRTp8Ktt7qEgIVAwJS8etq0\nzGq/JiwLkr59Ya+9ClBY/u//Nn8RVIXddoOHHoI5c5xgnDUL7r7bpaHad1/3pTlVoVfY/ouFQgNw\nKrAzqleh+llbBjNh2RZ+/3t3zviTn7T+g0nE7NnOqvaUU+CiizIzv0LHhGX+ETLFU1WVc0f2j7bz\nnscecz7HPqou28axx7pzxOIY5h2pCr1sfunMd0T6IDIFkQZEFiESNc8xItchssVL4OxfsUOIqX4X\n1amoNsZskwQmLNtCv37whz+4P4z77w/fb9Uq+N733DnlAw/kV1bxfKKszGVfb0zL77qRDsKkeMIJ\ny/Xrne1L3vPeey74RiSZdPY3grTKc4xItDzHAI+jWhq4FmRrkiYs28qZZ7qIN7/8ZbjD+UI8p8wV\n5eVOY1+5MtczMcBplQ884H4mCYxRCubccvFiOPHE6HW2LZp5RLblOUa1HtV4eY5zignLtuL7Xm7Y\n4HwvE1GI55S5wgIT5Be//nVzrN4EWlf//i5CW14Ly9WrXWDy9eudC4hpiLlgCLAV1cg8x7E0y5MQ\nWYXIXEQuyPz0mjFhmQ6GDIGrr3bnHn5s12jYOWVymLDMH2pq4OFAzoMQrg5VVW7HNpggI2/YuNHF\nWf3sM+cXaf6NGaMfFBXJOScAABl2SURBVCPyVuAaG6hOJs/x34FvAjsA5wG/QeQHGZl0FExYpotf\n/tIFLIjle2nnlMljwjJ/GDOm9dnx5s1xtcuqKhes5p13Mju1pGlqcgFFXnvNBdiwhNQZZaXTHPcP\nXBMC1eHzHKt+hOpSVBtRnQncDpyesYlHYMIyXfi+lwsXupQ6QeycMjUs5F1+sHIlvPxy6/LGxujl\nHn5s2Lzbir3iCvd3eMstLtejkUvm4zTPMHmOIwmfizgNmLBMJ0ccAeec4yJ7zJnTXG7nlKlRUuJc\nc0xY5paLLnJpoT74oPksb/lyl0iguNid10ehrMzlJc4rYfmnPzkL9osugssvz/VsDNVteY4R6YZI\n7DzHIt9FpDcigsiBwMXA09maqgnLdHPLLU5zHDvWZesYMcJ9k7VzytQwX8vc8uST7iz+N79xkQZ8\ndtjBbWHOneuOIGJQVeXii+eF988TT7iwdaec4hIX2FFIvtAqzzGqcxEZiUgwz/H3gc9wW7R/A8aj\n+lC2JmnCMt307ev+EGfNglNPdT5cJSUujJb9cSZPebnFy8wVdXXuDH7EiOgC8fjj4ZJLXHLw55+P\nOkRVlcvm9t57GZ5rIqZPdxF6DjkEJk9ujuVq5B7VVaiejGo3VAeh+ohXPh3V0kC7H6Da1/Ov3APV\nP2dzmiYsM8Ho0S4n3Ztvus9btjSb3BvJYZpl7rjoImeYNnFi9PimADff7CxJf/QjtzUbQV6cW378\nMXz3u1BZ6Y5DSkpyOBmjUDFhmQlEYOedmz/7OR+N5PGFpWquZ5KfhAw/lzRTpsCjj7rUa3vvHbtd\nly4ur+uaNXDuua1+TgMHuiQdOROWNTXOl7JTJ3jhBbfzYxgpYMIyE9TUuH82PpZ+J3XKy51PXGSe\nT8Nxww2hws8lhb/9Ony4S0eXiL32cmf1zz7rAnRE4J9bJhs+uU3U1LjdnWOOcda8zz0Hu+ySxQkY\n7Y2sCksR+ogwRYQGERaJEDVgrggiwngR6rxrvEizibAIw0V4W4T13uvwQN11ImwRoT5wDQ7TN21Y\n+p30Yb6WsVmyxMUkbmpyZ+Lp+jJ2ySVOYMbbfo3koovcGeZll7ltzwAdO7pgOUcdVUVlpTsyDMPk\nyW7ntKiIpPtNGnI9+vrrNM2dy7QL/wH77ZfR+6XSzygwVDVrF+ijoI+DloIeDroGdFiUdj8BnQe6\nE+hA0I9Az/fqOoEuAr0UtDPoxd7nTl79daCTYtw/bt94V+fOnTU0w4dHC5zlytsR06ZNy/xNXn7Z\nrd3LL2f+XmkiK+uiqnrggS1/v444QrWpqW1jPvWUG+u665LvW1Oj2q+f+z3fuFFVVSdNUi0paTnN\nkhLVe+9VXbky9nXvvSn0W1SvU66cqf9XfJNupUgVdBMdtbJLTWbuF6Nf167uucOStd+XHAE0aBbl\nTKaubArKbqCbQYcEyh4GvTlK25mgYwOfzwWd5b0/FnQJqATqF4Mer4mFZdy+8a6khOV2Qlb+yD/6\nyP2aTp6c+XuliaysyyOPRP9CduqpquvWpTZmXZ1qWZnqPvuobtqU2hhTp7p5XH65qqpWVESfZtir\njKVazRE6gJoW5f1YrsfwL72Sm/URvq8fs7s2ItsaNHmvG+ikd3Jhm+aQylVREX7JTFgWxhUjSVtG\nGAJsVSUyYG5VlLbDvLpgu2GBug9UCVoSfOCV+4FZTxJhFVAD3KnKPUn03YYIY4GxAMXFQnV1daJn\n3K6or6/P+JoU19dzOPDZjBl8teOOGb1Xusj0unSpreWAs8+miJbhS5qKipAnn2T9228z97rrWF9Z\nmdS4e/zud/RfsYJ3xo2jPtVsG927s9t3vsPAW2/l/bIyFi++jOhBVpSLLoqdi/eOO74BCNdwAyOZ\nzoOM4W32ZzjvcXi3/9Krodnytq77QJb024MX/3979x4dVXUvcPz7SwIJD0FBRJBr8NooFkUEbX2h\niPjEqxaqS4HS+uBNpdaF4q0gLaLC0uVd3mWpD1ARliAuFNFbabGktyI+EK4g8lBREAkWpChJkATy\nu3/sM+RkMo8kTM6Zyfw+a+3FzHnNL4eT/Gbvs8/eHS7npfd78ifGUIDrfV5ABbfwLFO5jxt/XRrn\n06o/r6FxRtu2TSkurluvpiB+j0wKBJWVQfuA7oxaNhy0OMa2h0C7+d4Xed/YBHQS6Pyo7eeBTvFe\n/xi0M2gu6PmgJaA3e+sS7puoWM2ytkC+EVdVqRYUqN51V+N/Voo06nkpL1ft1Us1Jyd2lebkk1WP\nO861BdanNr54sdt/8uQjj7GsTLVbN9XOnfXMLrsbVPPq13mDPs7YwzVEBa0kRzc26646dKjqo4+6\npvlvv62x35zWo3U/zWt82H6a6/OtxyT8vHg14GRxNnQ/P6tZZkYJsoNP3QfMrb1tG6DUqxEmPI4q\nn6iyQ5VDqkQPtlufGEw6ELFnLSNUYcwYNzL54sWxWwA/+8xNK9W7t3ved/Ro15s4kT17YORI94jI\n73535HG2bOkeJ9m1i9c6jaBlC621etq0GPvt3evGVz7vPN7acRpjeYIqr9ZWQTOey72dVc9+7GY/\n+e1voV8/aNeuxiEGtF9JARU1lhVQwTXtE9eUp01zcdUpziT7tWiRfD+TgYLKylTfsyzyLZtD/HuW\nw33vb6XmPcvt1LzvuJU49x1B7wFd1JB9/cVqlrUF9o34/PNVL7kkmM9KgUY7LzNnupQ4aVLybSsr\nVe++223fq5fqli3xtx02TDU3V3X16tTFqqo6Y4Yq6DvDZ2lhoapIlRYWRnV+OXhQdelS1ZtuUs3P\nd/F27646ebIebFZQ46tAZbMWrhNREnPnqvd5WvvzGnk/UL3hhrrtF2E1y8wowX4YOh/XI7YV6AXE\n7w07CnQDridsZ9D11O4NOx7Xo3UcNXvDXgd6jNdk+xNch55f1mXfRMWSZW2B/ZIPHOia9TJEo5yX\nlStVmzVTvfJKl2DqavFi1aOPdmXx4trrlyypewKur0OHVPv1U23VSvXtt/VfPXpUJ7tNm1TvvVf1\nhBPc5x9zjOrYsaqrVrmm99GjVZvXbE7V5s1VxyRuTg3b1Vertm9fvz5WliwzowSdLNuBvgpahuuF\nOthb3ge01LedgM4A3eOVGVG1wbNAPwTdD7oa9CzfuhdBvwUtBd0IekdUDHH3TVQsWdYW2C/52LHu\nj32GSPl52bnTJZWTTqp1j65OtmxR7d3b/bpPmKBaUaG6Y4ersXfsqHrGGQ3v/ZrMV1+5RNihg1aJ\nqPbt6z4X3H3XAQNUFy48/KjJYRn6+NW777owp0+v+z6WLDOjhB5AphRLlrUF9kv+wAPuUi0vD+bz\njlBKz0tlpUswBQVH1ky6f7+rrYHqhRe6ptdIAlq1KnXxxvL001oj4RUVuSbaHTsa93NDcsUVqh06\nqJaW1m17S5aZUWy4O5P+IqP4ZONwgRMnQnGx6/hy1lkNP05BAfzxj254mQ8/hDlz3PK8PDeAa2Na\nvbp6lo9mzaB/fzdtXeT/tYm5/37YtSvmyH8mg1myNOkvW4e8e+klN5H4mDEwbFhqjjl4sJuBIyIn\np3GHYSwpceMiRya0rKx0w+g14S8+553nvg/MmAHl5WFHY1LFkqVJf9mYLNevh1tvdX95H3ssdcct\nKYFXX61+39iD/GfpOMn33+9mLHvqqbAjMaliydKkv+OPd/9mS7L87js3cXjr1rBwoZteKlWCTl4r\nV7qE7FdRAQ0dIShDXHghXHIJTJ8O+/eHHY1JBUuWJv116OCaCzOh6a6khJ7jxzc81qoq+NWv4PPP\nXTNsqu8nBp281qw53LWnePny6m4+a9Y0zuelkcmT3WXwzDNhR2JSwZKlSX+5udCxY2bULKdOpe26\ndfWvqUUmcb7vPtdM+sgjcNFFqY/Pl7xqlCxIXkHr29f9Fz78cPJBlEz6s2RpMkMmDHlXUgKzZ7u5\nL55+2g35tno1bN8OBw4k3nfqVDdD8kMPwU03uTklTcabPBl27IDZs8OOxBypIGcdMabhOnVykx3X\nV0mJSz4LFlTf+2ws48dXJ8XKSjc2q1/btnDccbVLfr5rq1N1Y+FOm+b+NRmvXz+44AL3Hei229x/\ntclMlixNZmjbFv7yF3cTqD5Jb+pUePtt9+8TTzRefG++6Trj+OXnu4ftKitd10h/+fRTWLECdu+u\n2eEmL889LtKYsZrAiLja5RVXuCdmRo4MOyLTUJYsTWbYtMklneHD3SMV+/ZBaWnt4l++Zw+sW+dq\nbLNmwaRJjVO7fO+9ms8uRqjCBx8kTnzbt0NRUfVNrcpK9yhHY8VqAnfZZXDuufDgg3DLLant3GyC\nY8nSpL+SEli71r1+/XVX/ETcYxbRZe9et07VNY8OG+Zqp6m0YgVcdVXsdXXpZfrgg/Ef5bDaZZMQ\nqV1efbUbOOn228OOyDSEdfAx6W/q1Op7eM2awY03unkbd+6EsjKXXL7/3vWk2LzZdapZsMA1d/oT\n0V//CqNGwcGDqYnr73937WudOsGWLQ17RCJLn0PMNldeCeec474bVVaGHU2aEWmHyCuIlCGyFZHB\nSbZvjsgGRLYHFCFgydKku8hwaZGEUlkJS5ZAq1bucZKWLWN3hon18H1Ojhtj9bLLXCI9Em+95WqU\nhYUuaTb0eUh7lCMrRGqXX3wBc+eGHU3aeQKoADoCQ4CZiHRPsP0EYFcQgflZsjTpraEjzsSqsVVV\nwYknwrvvQu/e7n5iQyxdCtdcAz/6ESxfbvcWTZ0MGAC9ernOzqlq3Mh4Iq2AQcAkVEtRfRt4DfhF\nnO1PAoYCDwUWo8eSpUlvDW2mjFdj27rV3WfMzXVjks2aVb94Xn8drr0WunWDv/3NPfphTB1Eapef\nf+4ewc0Wx0IeIqt8ZYRv9SnAQVQ3+5Z9BMSrWf438J9A4IMIWrI06a0xmil79XLTVF10kettMWpU\n8kEDwI2sM3Ag9OjhmmGPPbbhMZisdO21cOaZ8MAD1ROxNHW7XTI821f8w8u3Br6P2uU74KhaBxL5\nGZCL6iuNF218lixNdmrf3j0bec897j5m376JBz1YuBBuuME13y5bBu3aBRaqaToitctPP4X588OO\nJi2UAm2ilrUB9tVY4pprZwB3BBNWbZYsTfbKzXUDd778snses3dvN+RctBdfhJtvdg/LLV3qBkgw\npoGuvx5OPz27apcJbMY10xb5lp0JrI/argjoCvwDkZ3AIqATIjsR6RpAnMEmSxHaifCKCGUibBUh\nZhdhEUSE6SJ865XpIohvfU8RPhSh3Pu3p2/dBBE+FmGfCF+IMCHq2F+KsF+EUq+k+ME7k3EGDYL3\n33dJsF8/ePxx9xjKxRe710OHQp8+8Oc/Q5voL8HG1E9Ojqtdbtzo+ob163cxXbvCvHl123/ePOja\n1R0nXffz7wMtWsTdULUMl/j+gEgrRC4ArgNeiNryY+DfgJ5euR34xnv9Vd1+kiOkqoEV0BdBF4C2\nBr0Q9DvQ7jG2Gwm6CbQL6Amgn4CO8tY1B90KeidoPugd3vvm3vq7QXuB5oGe6q27yXfsL0H71zf2\n/Px8NTUtX7487BBSa+9e1WuvdXdFi4pURdzr/v1Vy8rqfJgmd15SxM5LtRdeqL68IqVlS9W5cxPv\nN3eu2y6d96u9T0vVRH9foZ3CqwplCtsUBnvL+yiUxtmnr8L2hMdNcRFVDSQpi9AK+BdwuiqbvWUv\nAF+rMjFq23eA51R5ynt/GzBclXNFuBx4FujizhmIsA0YocqbMT73cUBU+bX3/kvgdlWW1Sf+goIC\n/cHm2amhuLiYvn37hh1GalVVwb33wowZ7n1OjhsA4aST6nyIJnleUsDOS7WuXV3H7Gh5eXDKKfH3\n27w59mMn6bRf7X1aoVqW8TMDBDnc3SnAwUii9HwEXBxj2+7eOv923X3r1kYSpWett7xGsvSabvsA\nT0Ydf54IOcAaYIJqjc/y7z8CGAGQlycUFxfH/eGyUWlpaZM8J0UbN9IpN5ecQ4eoysmh5M47+fQ3\nv6nz/k31vBwpOy/Vtm27GKidPw4eVDp0iP+8/SefdEj7/eLtk/GCqsKC9gHdGbVsOGhxjG0PgXbz\nvS/yqvQCOgl0ftT280CnxDjO70E/As33LbsAtAVoS9B7QXeCHp0sfmuGra1JNqvt2KFaUFCz3alF\nC9WSkjofokmelxSw81KtsLDmJRYphYWZv1/tfZI0w2ZICbKDT926CMfetg1Qqq42WafjiDAOGAYM\nUOXwQ3SqrFBlvyrlqjwE7MXVPo1p+IhBxtTDtGlupEa/li3d8kzfL9Y+TUGQyXIzkCdCsi7CeMvO\njLPdeqCHv3cs0MN/HBFuBSYCl6qSbLBdpUm2GZgGsYHNTQCGDIGnnnJDC4sohYXuffR84Yn3Iy33\ni97H1S6bgCCrsaDzvR6xrbzm0Hi9YUeBbvB6wnYGXR+jN+x4rzfsuKjesEO8ptXTYhz3RO9zm4MW\ngE4A3QXaPlns1gxbmzWrxWbnJTY7L7E19fMClGkaNKMeaQl6UIIxQAvgn8CLwGhV1ovQR4RS33ZP\nAkuAdbjna97wlqFKBXA9rol1L3ArcL23HOABoD3wge9Zyj95644CZuJ65X4NXAlcpcq3jfUDG2OM\nyXyBTv6syh5coote/g/cGIGR9wrc7ZVYx1kD9I6zLm4ff1XXhFu/qI0xxmQ7G+7OGGOMScKSpTHG\nGJOEJUtjjDEmiUDvWWayAwcOqIgEPuFomssDbM732uy8xGbnJbamfF7Kww4gVSxZ1t1qVT077CDS\niYissnNSm52X2Oy8xGbnJTNYM6wxxhiThCVLY4wxJglLlnX3VNgBpCE7J7HZeYnNzktsdl4yQGDz\nWRpjjDGZymqWxhhjTBKWLI0xxpgkLFkaY4wxSViyTEJE2onIKyJSJiJbRWRw2DGlAxEpFpEfRKTU\nK5vCjiloIjJORFaJyAEReS5q3aUislFEykVkuYgUhhRm4OKdFxHpKiLqu2ZKRWRSiKEGRkTyRWSW\n9zdkn4j8n4hc5VuftddLprBkmdwTQAXQERgCzBSR7uGGlDbGqWprr5wadjAh2IGbEm62f6GIHAss\nAiYB7YBVwILAowtPzPPic7TvupkaYFxhygO+Ai4G2gL3AS95XyCy/XrJCDaCTwIi0goYBJyuqqXA\n2yLyGvALYGKowZnQqeoiABE5G+jiWzUQWK+qC731U4DdItJNVTcGHmjAEpyXrKWqZcAU36LXReQL\n3FSD7cni6yVTWM0ysVOAg6q62bfsI8Bqls5DIrJbRFaISN+wg0kj3XHXCXD4D+Xn2HUTsVVEtovI\ns16tKuuISEfc35f12PWSESxZJtYa+D5q2XfAUSHEkm7uAf4dOAH3UPUSETk53JDSRmvcdeJn1w3s\nBs4BCnE1qqOAeaFGFAIRaYb7uZ/3ao52vWQAS5aJlQJtopa1AfaFEEtaUdX3VHWfqh5Q1eeBFcDV\nYceVJuy6iUFVS1V1laoeVNVvgHHA5SKSNUlBRHKAF3D9IMZ5i+16yQCWLBPbDOSJSJFv2Zm4phNT\nkwISdhBpYj3uOgEO3/s+GbtuokWGD8uKv0MiIsAsXGfBQapa6a2y6yUDZMVF2lDevYNFwB9EpJWI\nXABch/tmmLVE5GgRuUJECkQkT0SGABcBb4YdW5C8n70AyAVyI+cDeAU4XUQGeesnA2uzpbNGvPMi\nIj8VkVNFJEdE2gOPA8WqGt0E2VTNBE4D/kNV/XPjZvX1kjFU1UqCguvK/SpQBmwDBocdU9gF6AB8\ngGsm2gu8C1wWdlwhnIcpuNqRv0zx1vUHNgL7gWKga9jxhn1egJuBL7zfpRJgDnB82PEGdE4KvfPw\nA67ZNVKGZPv1kinFBlI3xhhjkrBmWGOMMSYJS5bGGGNMEpYsjTHGmCQsWRpjjDFJWLI0xhhjkrBk\naYwxxiRhydKYLOTNK/nzsOMwJlNYsjQmYCLynJesosu7YcdmjInN5rM0JhzLcPOi+lWEEYgxJjmr\nWRoTjgOqujOq7IHDTaTjROQNESkXka0iMtS/s4icISLLRGS/iOzxaqtto7b5pYisE5EDIvKNiDwf\nFUM7EVkoImUisiX6M4wx1SxZGpOefg+8BvTEzRc6R0TOhsOzUizFjS36E+BnwPnA7MjOIjISeBJ4\nFuiBmz7t46jPmAwsxs14sQCYLSInNt6PZEzmsrFhjQmYiDwHDMUNqu33hKreIyIKPKOqw337LAN2\nqupQERkOPAJ0UdV93vq+wHKgSFU/E5HtwFxVnRgnBgUeVtV7vfd5uInOR6jq3BT+uMY0CXbP0phw\n/C8wImrZXt/rlVHrVgIDvNen4aZw8k8O/A5QBfxYRL4HTgDeShLD2sgLVT0oIruA4+oWvjHZxZKl\nMeEoV9XPGuG49Wkqqox6r9itGWNisl8MY9LTuTHeb/BebwDOEJGjfOvPx/0+b1DVfwJfA5c2epTG\nZAmrWRoTjnwROT5q2SFV3eW9HigiH+AmAv45LvH91Fs3D9cBaI6ITAaOwXXmWeSrrU4DHhORb4A3\ngJbApar6aGP9QMY0ZZYsjQlHf6AkatnXQBfv9RRgEPA4sAu4RVU/AFDVchG5Avgv4H1cR6HFwPjI\ngVR1pohUAHcB04E9wP801g9jTFNnvWGNSTNeT9UbVPXlsGMxxjh2z9IYY4xJwpKlMcYYk4Q1wxpj\njDFJWM3SGGOMScKSpTHGGJOEJUtjjDEmCUuWxhhjTBKWLI0xxpgk/h/0GjHFWNminAAAAABJRU5E\nrkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xsASAa8KRwNc",
        "colab_type": "text"
      },
      "source": [
        "### tf.keras schedulers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XP_g7zXLRwNc",
        "colab_type": "code",
        "outputId": "5d53f483-e056-4d29-8b27-e1bc08442c50",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        }
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
        "learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\n",
        "optimizer = keras.optimizers.SGD(learning_rate)\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
        "n_epochs = 25\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/25\n",
            "55000/55000 [==============================] - 6s 100us/sample - loss: 0.4887 - accuracy: 0.8282 - val_loss: 0.4244 - val_accuracy: 0.8526\n",
            "Epoch 2/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.3830 - accuracy: 0.8642 - val_loss: 0.3802 - val_accuracy: 0.8680\n",
            "Epoch 3/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.3491 - accuracy: 0.8755 - val_loss: 0.3647 - val_accuracy: 0.8738\n",
            "Epoch 4/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.3267 - accuracy: 0.8840 - val_loss: 0.3563 - val_accuracy: 0.8742\n",
            "Epoch 5/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.3103 - accuracy: 0.8894 - val_loss: 0.3493 - val_accuracy: 0.8774\n",
            "Epoch 6/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.2970 - accuracy: 0.8940 - val_loss: 0.3401 - val_accuracy: 0.8816\n",
            "Epoch 7/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.2855 - accuracy: 0.8986 - val_loss: 0.3386 - val_accuracy: 0.8834\n",
            "Epoch 8/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.2765 - accuracy: 0.9026 - val_loss: 0.3370 - val_accuracy: 0.8836\n",
            "Epoch 9/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.2684 - accuracy: 0.9041 - val_loss: 0.3338 - val_accuracy: 0.8844\n",
            "Epoch 10/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2614 - accuracy: 0.9069 - val_loss: 0.3276 - val_accuracy: 0.8864\n",
            "Epoch 11/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2555 - accuracy: 0.9083 - val_loss: 0.3269 - val_accuracy: 0.8854\n",
            "Epoch 12/25\n",
            "55000/55000 [==============================] - 5s 95us/sample - loss: 0.2501 - accuracy: 0.9110 - val_loss: 0.3241 - val_accuracy: 0.8842\n",
            "Epoch 13/25\n",
            "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2455 - accuracy: 0.9124 - val_loss: 0.3191 - val_accuracy: 0.8902\n",
            "Epoch 14/25\n",
            "55000/55000 [==============================] - 6s 117us/sample - loss: 0.2414 - accuracy: 0.9143 - val_loss: 0.3224 - val_accuracy: 0.8876\n",
            "Epoch 15/25\n",
            "55000/55000 [==============================] - 6s 102us/sample - loss: 0.2378 - accuracy: 0.9156 - val_loss: 0.3230 - val_accuracy: 0.8862\n",
            "Epoch 16/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2347 - accuracy: 0.9179 - val_loss: 0.3204 - val_accuracy: 0.8892\n",
            "Epoch 17/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2317 - accuracy: 0.9179 - val_loss: 0.3191 - val_accuracy: 0.8886\n",
            "Epoch 18/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2293 - accuracy: 0.9194 - val_loss: 0.3180 - val_accuracy: 0.8902\n",
            "Epoch 19/25\n",
            "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2272 - accuracy: 0.9202 - val_loss: 0.3192 - val_accuracy: 0.8888\n",
            "Epoch 20/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2253 - accuracy: 0.9204 - val_loss: 0.3186 - val_accuracy: 0.8920\n",
            "Epoch 21/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2235 - accuracy: 0.9215 - val_loss: 0.3173 - val_accuracy: 0.8910\n",
            "Epoch 22/25\n",
            "55000/55000 [==============================] - 5s 98us/sample - loss: 0.2219 - accuracy: 0.9220 - val_loss: 0.3178 - val_accuracy: 0.8896\n",
            "Epoch 23/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2205 - accuracy: 0.9227 - val_loss: 0.3183 - val_accuracy: 0.8892\n",
            "Epoch 24/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2192 - accuracy: 0.9231 - val_loss: 0.3165 - val_accuracy: 0.8906\n",
            "Epoch 25/25\n",
            "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2181 - accuracy: 0.9236 - val_loss: 0.3168 - val_accuracy: 0.8898\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xVjwWJlWRwNe",
        "colab_type": "text"
      },
      "source": [
        "For piecewise constant scheduling, try this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2ndW3DcIRwNe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n",
        "    boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch], values=[0.01, 0.005, 0.001])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xcRUVrv0RwNg",
        "colab_type": "text"
      },
      "source": [
        "# Avoiding Overfitting Through Regularization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTTv1P2MRwNh",
        "colab_type": "text"
      },
      "source": [
        "## $\\ell_1$ and $\\ell_2$ regularization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AW-EUw3ARwNh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "layer = keras.layers.Dense(100, activation=\"elu\",\n",
        "                           kernel_initializer=\"he_normal\",\n",
        "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
        "# or l1(0.1) for ℓ1 regularization with a factor or 0.1\n",
        "# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q2VXs4y-RwNi",
        "colab_type": "code",
        "outputId": "d1d846d8-1455-4036-c686-9e2f8770cfb9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        }
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dense(300, activation=\"elu\",\n",
        "                       kernel_initializer=\"he_normal\",\n",
        "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
        "    keras.layers.Dense(100, activation=\"elu\",\n",
        "                       kernel_initializer=\"he_normal\",\n",
        "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
        "    keras.layers.Dense(10, activation=\"softmax\",\n",
        "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 2\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/2\n",
            "55000/55000 [==============================] - 18s 327us/sample - loss: 1.6073 - accuracy: 0.8112 - val_loss: 0.7314 - val_accuracy: 0.8242\n",
            "Epoch 2/2\n",
            "55000/55000 [==============================] - 16s 300us/sample - loss: 0.7193 - accuracy: 0.8256 - val_loss: 0.7029 - val_accuracy: 0.8304\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6QP7DUz8RwNk",
        "colab_type": "code",
        "outputId": "02ed5152-08be-407a-95fe-6ddb8b787490",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        }
      },
      "source": [
        "from functools import partial\n",
        "\n",
        "RegularizedDense = partial(keras.layers.Dense,\n",
        "                           activation=\"elu\",\n",
        "                           kernel_initializer=\"he_normal\",\n",
        "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    RegularizedDense(300),\n",
        "    RegularizedDense(100),\n",
        "    RegularizedDense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 2\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/2\n",
            "55000/55000 [==============================] - 17s 306us/sample - loss: 1.6597 - accuracy: 0.8128 - val_loss: 0.7630 - val_accuracy: 0.8080\n",
            "Epoch 2/2\n",
            "55000/55000 [==============================] - 16s 298us/sample - loss: 0.7176 - accuracy: 0.8271 - val_loss: 0.6848 - val_accuracy: 0.8360\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6XfOKCeZRwNn",
        "colab_type": "text"
      },
      "source": [
        "## Dropout"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xMrJKnYgRwNo",
        "colab_type": "code",
        "outputId": "edb5c6ea-21a7-4623-91f1-d218c20dff0c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        }
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.Dropout(rate=0.2),\n",
        "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
        "    keras.layers.Dropout(rate=0.2),\n",
        "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
        "    keras.layers.Dropout(rate=0.2),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 2\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/2\n",
            "55000/55000 [==============================] - 16s 283us/sample - loss: 0.5703 - accuracy: 0.8050 - val_loss: 0.3723 - val_accuracy: 0.8646\n",
            "Epoch 2/2\n",
            "55000/55000 [==============================] - 14s 260us/sample - loss: 0.4235 - accuracy: 0.8449 - val_loss: 0.3402 - val_accuracy: 0.8706\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9MJY8IpTRwNq",
        "colab_type": "text"
      },
      "source": [
        "## Alpha Dropout"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TVPABkkYRwNr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "igtVGGb-RwNt",
        "colab_type": "code",
        "outputId": "feebb46b-f461-4b6a-b487-68d0290f5486",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    keras.layers.AlphaDropout(rate=0.2),\n",
        "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.AlphaDropout(rate=0.2),\n",
        "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
        "    keras.layers.AlphaDropout(rate=0.2),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
        "n_epochs = 20\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/20\n",
            "55000/55000 [==============================] - 11s 198us/sample - loss: 0.6640 - accuracy: 0.7583 - val_loss: 0.5812 - val_accuracy: 0.8418\n",
            "Epoch 2/20\n",
            "55000/55000 [==============================] - 12s 217us/sample - loss: 0.5517 - accuracy: 0.7971 - val_loss: 0.5782 - val_accuracy: 0.8434\n",
            "Epoch 3/20\n",
            "55000/55000 [==============================] - 11s 192us/sample - loss: 0.5262 - accuracy: 0.8063 - val_loss: 0.5230 - val_accuracy: 0.8494\n",
            "Epoch 4/20\n",
            "55000/55000 [==============================] - 11s 193us/sample - loss: 0.5054 - accuracy: 0.8137 - val_loss: 0.4724 - val_accuracy: 0.8624\n",
            "Epoch 5/20\n",
            "55000/55000 [==============================] - 11s 193us/sample - loss: 0.4892 - accuracy: 0.8195 - val_loss: 0.5161 - val_accuracy: 0.8592\n",
            "Epoch 6/20\n",
            "55000/55000 [==============================] - 11s 193us/sample - loss: 0.4807 - accuracy: 0.8221 - val_loss: 0.4882 - val_accuracy: 0.8522\n",
            "Epoch 7/20\n",
            "55000/55000 [==============================] - 11s 194us/sample - loss: 0.4688 - accuracy: 0.8251 - val_loss: 0.4751 - val_accuracy: 0.8560\n",
            "Epoch 8/20\n",
            "55000/55000 [==============================] - 11s 208us/sample - loss: 0.4711 - accuracy: 0.8241 - val_loss: 0.4235 - val_accuracy: 0.8634\n",
            "Epoch 9/20\n",
            "55000/55000 [==============================] - 13s 235us/sample - loss: 0.4509 - accuracy: 0.8310 - val_loss: 0.4752 - val_accuracy: 0.8706\n",
            "Epoch 10/20\n",
            "55000/55000 [==============================] - 11s 209us/sample - loss: 0.4511 - accuracy: 0.8311 - val_loss: 0.4940 - val_accuracy: 0.8594\n",
            "Epoch 11/20\n",
            "55000/55000 [==============================] - 11s 193us/sample - loss: 0.4483 - accuracy: 0.8337 - val_loss: 0.4459 - val_accuracy: 0.8672\n",
            "Epoch 12/20\n",
            "55000/55000 [==============================] - 11s 192us/sample - loss: 0.4426 - accuracy: 0.8366 - val_loss: 0.4689 - val_accuracy: 0.8638\n",
            "Epoch 13/20\n",
            "55000/55000 [==============================] - 11s 193us/sample - loss: 0.4412 - accuracy: 0.8371 - val_loss: 0.4311 - val_accuracy: 0.8698\n",
            "Epoch 14/20\n",
            "55000/55000 [==============================] - 11s 194us/sample - loss: 0.4387 - accuracy: 0.8366 - val_loss: 0.4203 - val_accuracy: 0.8734\n",
            "Epoch 15/20\n",
            "55000/55000 [==============================] - 7s 131us/sample - loss: 0.4325 - accuracy: 0.8382 - val_loss: 0.4932 - val_accuracy: 0.8696\n",
            "Epoch 16/20\n",
            "55000/55000 [==============================] - 7s 130us/sample - loss: 0.4321 - accuracy: 0.8394 - val_loss: 0.4401 - val_accuracy: 0.8768\n",
            "Epoch 17/20\n",
            "55000/55000 [==============================] - 8s 150us/sample - loss: 0.4244 - accuracy: 0.8419 - val_loss: 0.4384 - val_accuracy: 0.8738\n",
            "Epoch 18/20\n",
            "55000/55000 [==============================] - 8s 142us/sample - loss: 0.4241 - accuracy: 0.8431 - val_loss: 0.4075 - val_accuracy: 0.8716\n",
            "Epoch 19/20\n",
            "55000/55000 [==============================] - 7s 129us/sample - loss: 0.4210 - accuracy: 0.8436 - val_loss: 0.4711 - val_accuracy: 0.8700\n",
            "Epoch 20/20\n",
            "55000/55000 [==============================] - 7s 127us/sample - loss: 0.4190 - accuracy: 0.8457 - val_loss: 0.4188 - val_accuracy: 0.8800\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xOD5KnOZRwNx",
        "colab_type": "code",
        "outputId": "e3075cd0-8a44-49b2-e33a-af0d83436c68",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "model.evaluate(X_test_scaled, y_test)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "10000/10000 [==============================] - 1s 54us/sample - loss: 0.4580 - accuracy: 0.8663\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4579802789211273, 0.8663]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 126
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Fwci151RwN1",
        "colab_type": "code",
        "outputId": "982545be-a0ea-4e0f-d83d-ed0093149ca4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "model.evaluate(X_train_scaled, y_train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "55000/55000 [==============================] - 3s 54us/sample - loss: 0.3429 - accuracy: 0.8863\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.342865064716881, 0.88634545]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IedKpIvPRwN3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#with keras.backend.learning_phase_scope(1):  # TODO: check https://github.com/tensorflow/tensorflow/issues/25754\n",
        "#    history = model.fit(X_train_scaled, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3YwPoMDRwN6",
        "colab_type": "text"
      },
      "source": [
        "## MC Dropout"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAFYwvkuRwN6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VUkm-QDMRwN7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with keras.backend.learning_phase_scope(1):  # TODO: check https://github.com/tensorflow/tensorflow/issues/25754\n",
        "    y_probas = np.stack([model.predict(X_test_scaled) for sample in range(100)])\n",
        "y_proba = y_probas.mean(axis=0)\n",
        "y_std = y_probas.std(axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EZYuKxWVRwN9",
        "colab_type": "code",
        "outputId": "7432cb09-f785-4992-8230-069b839f45cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "np.round(model.predict(X_test_scaled[:1]), 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 131
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D0FwNDMKRwN_",
        "colab_type": "code",
        "outputId": "d6e3621b-3702-41c5-ba9f-d9ad4c969b14",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3724
        }
      },
      "source": [
        "np.round(y_probas[:, :1], 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "\n",
              "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jvEt98epRwOC",
        "colab_type": "code",
        "outputId": "4049945d-0fdd-4734-a403-f0fc9907a713",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "np.round(y_proba[:1], 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agHSO2gnRwOE",
        "colab_type": "code",
        "outputId": "95f27c16-3f90-40b2-94ee-c7e7edd4ca13",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "y_std = y_probas.std(axis=0)\n",
        "np.round(y_std[:1], 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 134
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rA7ByU_jRwOT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_pred = np.argmax(y_proba, axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W8IpyL9pRwOU",
        "colab_type": "code",
        "outputId": "c4d57a2b-ebf4-4edc-85f4-d61d85ab6db5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "accuracy = np.sum(y_pred == y_test) / len(y_test)\n",
        "accuracy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8663"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1She6VMRwOW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MCDropout(keras.layers.Dropout):\n",
        "    def call(self, inputs):\n",
        "        return super().call(inputs, training=True)\n",
        "\n",
        "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
        "    def call(self, inputs):\n",
        "        return super().call(inputs, training=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6N_W6jZeRwOY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cN4ZtUJeRwOb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mc_model = keras.models.Sequential([\n",
        "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
        "    for layer in model.layers\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Jv50R2tRwOc",
        "colab_type": "code",
        "outputId": "f63e2e9e-07af-4e79-978f-66832784ea32",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 425
        }
      },
      "source": [
        "mc_model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_22\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "flatten_20 (Flatten)         (None, 784)               0         \n",
            "_________________________________________________________________\n",
            "mc_alpha_dropout (MCAlphaDro (None, 784)               0         \n",
            "_________________________________________________________________\n",
            "dense_465 (Dense)            (None, 300)               235500    \n",
            "_________________________________________________________________\n",
            "mc_alpha_dropout_1 (MCAlphaD (None, 300)               0         \n",
            "_________________________________________________________________\n",
            "dense_466 (Dense)            (None, 100)               30100     \n",
            "_________________________________________________________________\n",
            "mc_alpha_dropout_2 (MCAlphaD (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "dense_467 (Dense)            (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 266,610\n",
            "Trainable params: 266,610\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X1a0Pm-mRwOf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
        "mc_model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i8if1TimRwOg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mc_model.set_weights(model.get_weights())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lm2tQFl8RwOi",
        "colab_type": "text"
      },
      "source": [
        "Now we can use the model with MC Dropout:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BJfbN6EnRwOi",
        "colab_type": "code",
        "outputId": "2045023b-e35c-4f45-d742-fb6aff1988c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.29, 0.  , 0.14, 0.  , 0.56]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 143
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mr7K17QgRwOl",
        "colab_type": "text"
      },
      "source": [
        "## Max norm"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MMP2D3EPRwOm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
        "                           kernel_constraint=keras.constraints.max_norm(1.))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fkKV4zRGRwOn",
        "colab_type": "code",
        "outputId": "2bd90a9e-6a8f-4c0a-c75a-5ab5200b227b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        }
      },
      "source": [
        "MaxNormDense = partial(keras.layers.Dense,\n",
        "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
        "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=[28, 28]),\n",
        "    MaxNormDense(300),\n",
        "    MaxNormDense(100),\n",
        "    keras.layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
        "n_epochs = 2\n",
        "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
        "                    validation_data=(X_valid_scaled, y_valid))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 55000 samples, validate on 5000 samples\n",
            "Epoch 1/2\n",
            "55000/55000 [==============================] - 10s 185us/sample - loss: 0.4744 - accuracy: 0.8323 - val_loss: 0.4000 - val_accuracy: 0.8580\n",
            "Epoch 2/2\n",
            "55000/55000 [==============================] - 10s 180us/sample - loss: 0.3559 - accuracy: 0.8697 - val_loss: 0.3655 - val_accuracy: 0.8678\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "bgZb7xtHRwOp",
        "colab_type": "text"
      },
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJNwWlcTRwOp",
        "colab_type": "text"
      },
      "source": [
        "## 1. to 7."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RADuHGa9RwOq",
        "colab_type": "text"
      },
      "source": [
        "See appendix A."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Fcd19UGRwOq",
        "colab_type": "text"
      },
      "source": [
        "## 8. Deep Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_H6W0S5CRwOr",
        "colab_type": "text"
      },
      "source": [
        "### 8.1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vc1_EBsRwOr",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: Build a DNN with five hidden layers of 100 neurons each, He initialization, and the ELU activation function._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rOSiI7ndRwOr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOGkgx5zRwOt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lebcz_nORwOu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Nh9IP1yRwOw",
        "colab_type": "text"
      },
      "source": [
        "### 8.2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "57LgdDBDRwOx",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: Using Adam optimization and early stopping, try training it on MNIST but only on digits 0 to 4, as we will use transfer learning for digits 5 to 9 in the next exercise. You will need a softmax output layer with five neurons, and as always make sure to save checkpoints at regular intervals and save the final model so you can reuse it later._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wvvzKP3ARwOx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CU9qAB9rRwOz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bL5OweDaRwO1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LUY8xPquRwO2",
        "colab_type": "text"
      },
      "source": [
        "### 8.3."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KVSMWcSWRwO3",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: Tune the hyperparameters using cross-validation and see what precision you can achieve._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mO9VUcOJRwO3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fb_pfdmcRwO7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CdQm30zxRwO9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0fnm1fsmRwO-",
        "colab_type": "text"
      },
      "source": [
        "### 8.4."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g5brSgt2RwPA",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: Now try adding Batch Normalization and compare the learning curves: is it converging faster than before? Does it produce a better model?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c59aAxM1RwPB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a3c3wrXRRwPD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XQoGPEUZRwPF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ilcwK_yBRwPH",
        "colab_type": "text"
      },
      "source": [
        "### 8.5."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1V8TebqNRwPH",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: is the model overfitting the training set? Try adding dropout to every layer and try again. Does it help?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WI2eFAFKRwPI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HmAyc6IHRwPJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JssmzQp2RwPL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "o6p5_TJARwPM",
        "colab_type": "text"
      },
      "source": [
        "## 9. Transfer learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SRd8WuudRwPO",
        "colab_type": "text"
      },
      "source": [
        "### 9.1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jHa3S3vfRwPO",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: create a new DNN that reuses all the pretrained hidden layers of the previous model, freezes them, and replaces the softmax output layer with a new one._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o64oP9WERwPP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fsHoIGr3RwPR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y0J-uw6cRwPU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8hHBFwXRwPW",
        "colab_type": "text"
      },
      "source": [
        "### 9.2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92HoLJuvRwPW",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: train this new DNN on digits 5 to 9, using only 100 images per digit, and time how long it takes. Despite this small number of examples, can you achieve high precision?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UgTsAlPURwPX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hxjin-lqRwPY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wGcmJEAfRwPZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eO6xKjfIRwPc",
        "colab_type": "text"
      },
      "source": [
        "### 9.3."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tpJWqn_2RwPd",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: try caching the frozen layers, and train the model again: how much faster is it now?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A60NOT_mRwPd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gSdLKB2YRwPf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gwSr82PERwPg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6N2bBKkRwPi",
        "colab_type": "text"
      },
      "source": [
        "### 9.4."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1_lOU6pLRwPi",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: try again reusing just four hidden layers instead of five. Can you achieve a higher precision?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4YO9MtX8RwPj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AtfCPduHRwPk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aEsvDP7xRwPm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jy5myoRRRwPn",
        "colab_type": "text"
      },
      "source": [
        "### 9.5."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXiHBTsyRwPn",
        "colab_type": "text"
      },
      "source": [
        "_Exercise: now unfreeze the top two hidden layers and continue training: can you get the model to perform even better?_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kkuQaYo_RwPo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pTFe4J2iRwPp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dYpqV9jnRwPr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hQxXEvHaRwPt",
        "colab_type": "text"
      },
      "source": [
        "## 10. Pretraining on an auxiliary task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P0ryhuW0RwPt",
        "colab_type": "text"
      },
      "source": [
        "In this exercise you will build a DNN that compares two MNIST digit images and predicts whether they represent the same digit or not. Then you will reuse the lower layers of this network to train an MNIST classifier using very little training data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8SyTadJcRwPt",
        "colab_type": "text"
      },
      "source": [
        "### 10.1.\n",
        "Exercise: _Start by building two DNNs (let's call them DNN A and B), both similar to the one you built earlier but without the output layer: each DNN should have five hidden layers of 100 neurons each, He initialization, and ELU activation. Next, add one more hidden layer with 10 units on top of both DNNs. You should use the `keras.layers.concatenate()` function to concatenate the outputs of both DNNs, then feed the result to the hidden layer. Finally, add an output layer with a single neuron using the logistic activation function._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JuTtjY6tRwPu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QYZaKIviRwPw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_oWfvbzgRwPx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8dUvqeV1RwPz",
        "colab_type": "text"
      },
      "source": [
        "### 10.2.\n",
        "_Exercise: split the MNIST training set in two sets: split #1 should containing 55,000 images, and split #2 should contain contain 5,000 images. Create a function that generates a training batch where each instance is a pair of MNIST images picked from split #1. Half of the training instances should be pairs of images that belong to the same class, while the other half should be images from different classes. For each pair, the training label should be 0 if the images are from the same class, or 1 if they are from different classes._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iUDK2oTORwP0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i-FhodmoRwP1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rbnFP_-LRwP2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HcQ_O5WkRwP4",
        "colab_type": "text"
      },
      "source": [
        "### 10.3.\n",
        "_Exercise: train the DNN on this training set. For each image pair, you can simultaneously feed the first image to DNN A and the second image to DNN B. The whole network will gradually learn to tell whether two images belong to the same class or not._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRtO0JGdRwP5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pjsY17N-RwP6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MwA4rU1LRwP9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_oCqv7JURwP-",
        "colab_type": "text"
      },
      "source": [
        "### 10.4.\n",
        "_Exercise: now create a new DNN by reusing and freezing the hidden layers of DNN A and adding a softmax output layer on top with 10 neurons. Train this network on split #2 and see if you can achieve high performance despite having only 500 images per class._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8_HTpnkRwP_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jEjoTLoeRwQA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EIqj3SKZRwQC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}