lecture05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d715d6a9-c3d7-492f-aa7d-6465b264d01c",
   "metadata": {},
   "source": [
    "## Lecture 5 - Convolutional neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda90d94-084b-46a7-98e0-19c4c3e63c34",
   "metadata": {},
   "source": [
    "> A deep dive into convolutional neural networks for jet tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33cd450c-b40e-4116-8619-6647371d0661",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "* Know how to implement and train a convolutional neural network in PyTorch\n",
    "* Learn how to debug unstable training runs with activation statistics\n",
    "* Understand what the 1-cycle training policy is\n",
    "* Understand what batch normalization is and how to incorporate it into a CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9df3227f-1c01-4552-804f-d8324dc539db",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "* Chapter 13 of [_Deep Learning for Coders with fastai & PyTorch_](https://github.com/fastai/fastbook) by Jeremy Howard and Sylvain Gugger.\n",
    "* Chapter 11 of [_Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow_](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/ref=sr_1_1?crid=29Z1GSCOWEPDV&keywords=hands+on+machine+learning+with+scikit-learn+and+tensorflow+2&qid=1653288575&sprefix=hands+on+ma%2Caps%2C160&sr=8-1) by Aurélien Geron"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517b33cb-9796-464c-96a2-14da6eb96aef",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a60f6f46-f4a7-49c8-acbb-df68c1318751",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run this cell if using Colab, Kaggle etc\n",
    "# %pip install fastai==2.6.0 datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1dd43ed-3e9b-4359-a3e1-992acef49068",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4bdcfefa-9b67-480c-b15b-320e7d2d0c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from fastai.callback.hook import *\n",
    "from fastai.vision.all import *\n",
    "from sklearn.metrics import accuracy_score\n",
    "from torch.utils.data import TensorDataset\n",
    "from torchvision.transforms import ToTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "88c210f7-1dc4-4ace-9ba8-88f48e5bad4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "\n",
    "# Suppress logs to keep things tidy\n",
    "datasets.logging.set_verbosity_error()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e42b0bd-7d57-4c5a-9792-8cdbb1fb310e",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a8142a-a5d7-4083-aa85-7d863f827dc4",
   "metadata": {},
   "source": [
    "Last lecture we fine-tuned a pretrained CNN on the top tagging dataset, where each jet was represented as a 2D image of \"energy pixels\". Today, we'll take a look at training our own CNN from scratch, and explore some of the techniques that fastai utilizes to stabilise the training of these neural nets. To get started, let's download the same dataset of jet images from last lecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aaa93e67-1da6-47c5-8ba3-ff28ebd4c6d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f75ae7e9089413bb5de76b999c332af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD6CAYAAABnLjEDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAMIklEQVR4nO3df6jdd33H8efL26TpqiUG26xrstm5MilDM+mqMhlduo6s/6SCiIWNDAp1YwVlMsz8Rx0TOqZ2f2w4FLNmw1lL1SVINw1ZwAmSttYY00bX2FVMSBOdy5oOFpfkvT/ON3Kbe25yen7dc+/n+YDD+Z7P93vu9/1t7qvfcz73nO87VYWkle8VS12ApOkw7FIjDLvUCMMuNcKwS40w7FIjRgp7ki1JvpvkSJLt4ypK0vhl2L+zJ5kD/h24AzgKPA7cXVVPL/ac1bmy1nD1UPuTdHn/y//wkzqTfuuuGOHn3gocqapnAZI8BGwFFg37Gq7mzbl9hF1KupT9tXfRdaO8jL8B+MG8x0e7MUkzaJQz+0CS3AvcC7CGn5n07iQtYpQz+zFg47zHG7qxl6iqT1bVLVV1yyquHGF3kkYxStgfB25KcmOS1cC7gN3jKUvSuA39Mr6qzia5D/gyMAfsqKqnxlaZpLEa6T17VT0KPDqmWiRNkJ+gkxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGGHapEYZdaoRhlxph2KVGjHRZqiTPAaeBc8DZqrplHEVJGr9xXDf+N6vqR2P4OZImyJfxUiNGDXsBX0nyja7zi6QZNerL+LdV1bEk1wF7knynqr46fwPbP0mzYaQze1Ud6+5PAl+k19n14m1s/yTNgKHDnuTqJK+6sAz8NnBoXIVJGq9RXsavB76Y5MLP+ceq+pexVCVp7Ebp9fYs8MYx1iJpgvzTm9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9QIwy41wrBLjbhs2JPsSHIyyaF5Y+uS7EnyTHf/6smWKWlUg5zZHwS2XDS2HdhbVTcBe7vHkmbYZcPedXj58UXDW4Gd3fJO4K7xliVp3Ia9lPT6qjreLT9P7xryfdn+SZoNI0/QVVXRa/C42HrbP0kzYNiwn0hyPUB3f3J8JUmahGHDvhvY1i1vA3aNpxxJkzLIn94+C3wd+OUkR5PcA9wP3JHkGeC3useSZthlJ+iq6u5FVt0+5lokTZCfoJMaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaMexlqbQCveHJLBg7+KZFL0KkZcYzu9QIwy41wrBLjTDsUiOGbf/0oSTHkhzobndOtkxJoxpkNv5B4K+Bv79o/IGq+ujYK9KSceZ9ZRu2/ZOkZWaU9+z3JTnYvcy3i6s044YN+yeA1wGbgOPAxxbbMMm9SZ5I8sT/cWbI3Uka1VBhr6oTVXWuqs4DnwJuvcS29nqTZsBQYb/Q563zduDQYttKmg2XnY3v2j/dBrwmyVHgg8BtSTbR6976HPDuyZUoaRyGbf/06QnUImmC/ASd1AjDLjXCsEuN8OIVurRXzC0cO39u+nVoZJ7ZpUYYdqkRhl1qhGGXGuEEnS7NybgVwzO71AjDLjXCsEuNMOxSIwy71Ahn4zU2c9dcs2Ds3AsvLEEl6sczu9QIwy41wrBLjRik/dPGJPuSPJ3kqSTv6cbXJdmT5Jnu3mvHSzNskAm6s8D7qurJJK8CvpFkD/D7wN6quj/JdmA78P7JlapJu2LjhgVjN+86tmBssTZRTsbNtkHaPx2vqie75dPAYeAGYCuws9tsJ3DXhGqUNAYv6z17ktcCvwrsB9ZX1fFu1fPA+vGWJmmcBg57klcCnwfeW1Uveb1WVUXvGvL9nmf7J2kGDBT2JKvoBf0zVfWFbvjEhc4w3f3Jfs+1/ZM0GwbpCBN6TSEOV9XH563aDWwD7u/ud02kQk3N6Tf93IKxv/zZLy0Y+42t9/Z9/lW7Hht4X37abvoGmY3/deD3gG8nOdCNfYBeyB9Ocg/wfeCdE6lQ0lgM0v7pa0AWWX37eMuRNCl+gk5qhGGXGmHYpUb4fXb9VL/Z9Ddf84cLxtbu+nrf58+tv27B2Lkf/mffbZ15nz7P7FIjDLvUCMMuNcKwS41YuRN0/fqKg+2MgFzZ/zsKdWbhF5XW/kP/ybh+zp/67z6DU/7vbT/5RXlmlxph2KVGGHapEYZdaoRhlxqxcmfjnYFdVL9Z91n+uS+L/+6L8swuNcKwS40w7FIjRmn/9KEkx5Ic6G53Tr5cScMapf0TwANV9dHJlSdpXAa54ORx4Hi3fDrJhfZPkpaRUdo/AdyX5GCSHXZxlWbbKO2fPgG8DthE78z/sUWeZ/snaQYM3f6pqk5U1bmqOg98Cri133Nt/yTNhkFm4/u2f7rQ563zduDQ+MuTNC6jtH+6O8kmet1bnwPePYH6xs+LG6hRo7R/enT85UiaFD9BJzXCsEuNMOxSI1bu99kX42ScGuWZXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaMcgFJ9ckeSzJt7r2Tx/uxm9Msj/JkSSfS7J68uVKGtYg32c/A2yuqhe7S0p/Lck/A39Mr/3TQ0n+FriH3rXktQTm1l/Xd/zciZNTrkSz6rJn9up5sXu4qrsVsBl4pBvfCdw1iQIljcegTSLmustInwT2AN8DTlXV2W6To9j/TZppA4W96/yyCdhAr/PL6wfdge2fpNnwsmbjq+oUsA94K7A2yYX3/BuAY4s8x/ZP0gwYZDb+2iRru+WrgDuAw/RC/45us23ArgnVKGkMBpmNvx7YmWSO3v8cHq6qLyV5GngoyZ8D36TXD05LxFl3Xc4g7Z8O0uvJfvH4syzSuVXS7PETdFIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUCMMuNcKwS40w7FIjDLvUiFHaPz2Y5D+SHOhumyZeraShjdL+CeBPquqRSzxX0owY5IKTBfRr/yRpGRmq/VNV7e9WfSTJwSQPJLEDhDTDhmr/lORXgD+l1wbq14B1wPv7Pdf2T9JsGLb905aqOt51eD0D/B2LXEPe9k/SbBi2/dN3klzfjYVeu+ZDkytT0qhGaf/0r0muBQIcAP5gcmVKGtUo7Z82T6QiSRPhJ+ikRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcaYdilRhh2qRGGXWqEYZcakV53pyntLPkh8P3u4WuAH01t59PjcS0/K+nYfqGqru23Yqphf8mOkyeq6pYl2fkEeVzLz0o+tvl8GS81wrBLjVjKsH9yCfc9SR7X8rOSj+2nluw9u6Tp8mW81Iiphz3JliTfTXIkyfZp73+ckuxIcjLJoXlj65LsSfJMd//qpaxxGEk2JtmX5OkkTyV5Tze+rI8tyZokjyX5VndcH+7Gb0yyv/ud/FyS1Utd6yRMNexdJ9i/AX4HuBm4O8nN06xhzB4Etlw0th3YW1U3AXu7x8vNWeB9VXUz8Bbgj7p/p+V+bGeAzVX1RmATsCXJW4C/AB6oql8C/gu4Z+lKnJxpn9lvBY5U1bNV9RPgIWDrlGsYm6r6KvDji4a3Aju75Z30etcvK1V1vKqe7JZPA4eBG1jmx1Y9L3YPV3W3AjYDj3Tjy+64BjXtsN8A/GDe46Pd2EqyvqqOd8vPA+uXsphRJXktvZbd+1kBx5ZkLskB4CSwB/gecKqqznabrMTfScAJuomq3p86lu2fO5K8Evg88N6qemH+uuV6bFV1rqo2ARvovdJ8/dJWND3TDvsxYOO8xxu6sZXkRJLrAbr7k0tcz1CSrKIX9M9U1Re64RVxbABVdQrYB7wVWJvkim7VSvydBKYf9seBm7rZz9XAu4DdU65h0nYD27rlbcCuJaxlKEkCfBo4XFUfn7dqWR9bkmuTrO2WrwLuoDcfsQ94R7fZsjuuQU39QzVJ7gT+CpgDdlTVR6ZawBgl+SxwG71vTZ0APgj8E/Aw8PP0vuH3zqq6eBJvpiV5G/BvwLeB893wB+i9b1+2x5bkDfQm4Obonegerqo/S/KL9CaL1wHfBH63qs4sXaWT4SfopEY4QSc1wrBLjTDsUiMMu9QIwy41wrBLjTDsUiMMu9SI/wcVd3naJg9pBAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "images_ds = load_dataset(\"dl4phys/top_tagging_images\")\n",
    "\n",
    "# Peek at one example\n",
    "plt.imshow(images_ds[\"train\"][0][\"image\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76bcd2ae-aca2-4c0f-acab-99e6038a7954",
   "metadata": {},
   "source": [
    "As we saw last lecture, we need to convert these PIL images into PyTorch tensors, so let's reuse the same helper function to create a training and validation set to experiment with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2130839e-200e-45c0-ae91-7cea9728eb03",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dataset(dataset, num_examples=None):\n",
    "    if num_examples is not None:\n",
    "        dataset = dataset.shuffle(seed=42).select(range(num_examples))\n",
    "\n",
    "    x = torch.cat([ToTensor()(img) for img in dataset[\"image\"]]).unsqueeze(1)\n",
    "    y = torch.cat([torch.tensor(l).unsqueeze(0) for l in dataset[\"label\"]])\n",
    "\n",
    "    return TensorDataset(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570d160e-1d3d-4083-b064-f83d160c333c",
   "metadata": {},
   "source": [
    "With this function, we can now generate a sample of jets as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bed72e0b-996f-4812-83bf-3c68a9685855",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lower the training size if Colab RAM explodes 💣\n",
    "train_ds = get_dataset(images_ds[\"train\"], num_examples=350_000)\n",
    "valid_ds = get_dataset(images_ds[\"validation\"], num_examples=35_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44714b91-61d3-48dc-a486-adcfad3acf60",
   "metadata": {},
   "source": [
    "Since we'll be experimenting a bit with the batch size in this lecture, we'll also implement a helper function to return the dataloaders associated with these datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fb79e523-8551-4594-b2d1-cfe955808b51",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dls(bs=128):\n",
    "    train_dl = DataLoader(train_ds, bs=bs, shuffle=True)\n",
    "    valid_dl = DataLoader(valid_ds, bs=bs)\n",
    "    return DataLoaders(train_dl, valid_dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0d405b-19de-4479-8bee-ebdd3a09c653",
   "metadata": {},
   "source": [
    "Now that we have this function, we can get the dataloaders and grab a batch of data to test with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2d58401f-94a9-48fd-855d-6df0cbd52eba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([128, 1, 40, 40]), torch.Size([128]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dls = get_dls()\n",
    "\n",
    "xb, yb = first(dls.train)\n",
    "xb.shape, yb.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c718b6da-5040-4532-9b88-d47804ba3b32",
   "metadata": {},
   "source": [
    "## Creating a CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b81aff1e-2590-4c51-a666-0ce7f01cdc02",
   "metadata": {},
   "source": [
    "Recall in lecture 3, that we created a neural network for $N$-subjettiness features. We used the `nn.Sequential()` class to create a network that involved stacking fully-connected layers and ReLU activation functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a5a9802d-4595-4195-94e6-323c05060490",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=20, out_features=200, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=200, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(20, 200),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(200, 2),\n",
    ")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "303bd45c-f58d-41ff-917e-d4c8a1a2bedc",
   "metadata": {},
   "source": [
    "In this lecture, we'll take a similar approach, but this time using _convolutional layers_ instead of linear ones. In PyTorch, convolutional layers are created by using the `nn.Conv2d()` module. Let's try replacing the `nn.Linear()` layers in our previous architecture with `nn.Conv2d()` ones instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3cd0c734-2eee-4f19-b70f-32aa1b0d5c1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "broken_cnn = nn.Sequential(\n",
    "    nn.Conv2d(1, 30, kernel_size=3, padding=1),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(30, 1, kernel_size=3, padding=1),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea83b3c3-172f-4960-99f5-3b12a5f37fbc",
   "metadata": {},
   "source": [
    "Note that unlike linear layers, convolutional layers don't require us to specify the number of features in the input. That's because convolutions are applied across each pixel automatically, so the weights only depend on the number of channels and the kernel size. \n",
    "\n",
    "Let's now see what happens if we pass a minibatch of data through this model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "03c20c8f-4c58-43ad-869a-986cbd69308a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 1, 40, 40])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feed a minibatch to the model\n",
    "outputs = broken_cnn(xb)\n",
    "outputs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4ccb3ca-5e3b-49f6-948a-f835932d5ebb",
   "metadata": {},
   "source": [
    "Hmm, this output isn't quite what we want for classification: we need a tensor of shape `(batch_size, num_classes)` but instead have a $40\\times 40$ map of activations. The standard way to handle this is to apply a sequence of _stride-2 convolutions_, which decrease the size of the outputs so that the final layer size is 1. To see why this is the case, recall that for an image of height $n_H$ and width $n_W$, the dimension of the output activation map is given by:\n",
    "\n",
    "$$\\left( \\left\\lfloor\\frac{n_H + 2p - k}{s} + 1 \\right\\rfloor ,  \\left\\lfloor\\frac{n_W + 2p - k}{s} + 1 \\right\\rfloor \\right) \\,,$$\n",
    "\n",
    "where $p$ is the padding, $k$ the kernel of size $k\\times k$, and $s$ the stride. For fixed $p,f$ and $s>1$, we can see that the dimension of the activation map is shrunk with each convolutional layer.\n",
    "\n",
    "With this in mind, let's create a stack of stride-2 convolutional layers with $3\\times 3$ kernels. We'll intersperse each convolutional layer with a ReLU activation, so let's write a helper function that returns the two together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "03e32bc3-ed53-45be-bb81-efe847c56377",
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv(ni, nf, ks=3, act=True):\n",
    "    layer = nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks // 2)\n",
    "    if act:\n",
    "        layer = nn.Sequential(layer, nn.ReLU())\n",
    "    return layer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae27a32-4002-43ae-adf6-34780c300178",
   "metadata": {},
   "source": [
    "As discussed in the fastai book, it's good practice to increase the number of output features `nf` with each convolutional layer. That's because, stride-2 convolutions decrease the size of the activation map, so we increase the number of output features to avoid compressing the capacity of our model. We also set the kernel size `ks` to the typical value of $3\\times 3$, and set the padding to $ks//2$ to ensure the output activation map is the same shape as the input image.\n",
    "\n",
    "We can now build a CNN by stacking the `conv()` function until we reach a $1\\times 1$ activation map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b892abb2-34ec-406b-9079-6618c9f68dc8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Sequential (Input shape: 128 x 1 x 40 x 40)\n",
       "============================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "============================================================================\n",
       "                     128 x 4 x 20 x 20   \n",
       "Conv2d                                    40         True      \n",
       "ReLU                                                           \n",
       "____________________________________________________________________________\n",
       "                     128 x 8 x 10 x 10   \n",
       "Conv2d                                    296        True      \n",
       "ReLU                                                           \n",
       "____________________________________________________________________________\n",
       "                     128 x 16 x 5 x 5    \n",
       "Conv2d                                    1168       True      \n",
       "ReLU                                                           \n",
       "____________________________________________________________________________\n",
       "                     128 x 32 x 3 x 3    \n",
       "Conv2d                                    4640       True      \n",
       "ReLU                                                           \n",
       "____________________________________________________________________________\n",
       "                     128 x 16 x 2 x 2    \n",
       "Conv2d                                    4624       True      \n",
       "ReLU                                                           \n",
       "____________________________________________________________________________\n",
       "                     128 x 2 x 1 x 1     \n",
       "Conv2d                                    290        True      \n",
       "____________________________________________________________________________\n",
       "                     128 x 2             \n",
       "Flatten                                                        \n",
       "____________________________________________________________________________\n",
       "\n",
       "Total params: 11,058\n",
       "Total trainable params: 11,058\n",
       "Total non-trainable params: 0\n",
       "\n",
       "Optimizer used: <function Adam at 0x7fea02760430>\n",
       "Loss function: <function cross_entropy at 0x7fea509cf8b0>\n",
       "\n",
       "Callbacks:\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_cnn = nn.Sequential(\n",
    "    conv(1, 4),  # 20x20\n",
    "    conv(4, 8),  # 10x10\n",
    "    conv(8, 16),  # 5x5\n",
    "    conv(16, 32),  # 3x3\n",
    "    conv(32, 16),  # 2x2\n",
    "    conv(16, 2, act=False),  # 1x1\n",
    "    Flatten(),\n",
    ")\n",
    "\n",
    "learn = Learner(\n",
    "    dls, simple_cnn, metrics=[accuracy, RocAucBinary()], loss_func=F.cross_entropy\n",
    ")\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94522bdd-d222-4640-8cbb-1b05abd7be21",
   "metadata": {},
   "source": [
    "Since the final convolutional layer will be a tensor of shape $128\\times2\\times1\\times1$, we've stripped off those final $1\\times 1$ axes via the `Flatten()` module. We can now verify that the output of this model is in a form suitable for classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6da3d452-ee27-42b9-97e1-782a6c73b8c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 2])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_cnn(xb).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9941e8e6-2e5d-4de8-8597-9a32dc8b8d3a",
   "metadata": {},
   "source": [
    "Okay, this looks good, so let's find a good learning rate and train for a few epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5578dcce-95d0-4507-95a7-c14e0f9655ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(valley=0.03981071710586548)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9rElEQVR4nO3deXyU1dXA8d+ZySRhDZAFMAESNiHsGEFFLC4o7lgrYLVq69Jq1aqvttq+VUu1rb62Wi1VsXWrC0VURAXEBQXZZJF9J2xhS0ggCQnZz/vHPBMmIcuEZMiEnO/nM5/M3HmW8wwhZ+7y3CuqijHGGBMoV2MHYIwxpmmxxGGMMaZOLHEYY4ypE0scxhhj6sQShzHGmDqxxGGMMaZOwho7gJMhJiZGExMTGzsMY4xpUpYvX35QVWMrlzeLxJGYmMiyZcsaOwxjjGlSRGRnVeXWVGWMMaZOLHEYY4ypE0scxhhj6qRZ9HEYY0x1iouLSUtLo6CgoLFDaTSRkZEkJCTg8XgC2t4ShzGmWUtLS6NNmzYkJiYiIo0dzkmnqmRmZpKWlkZSUlJA+1hTlTGmWSsoKCA6OrpZJg0AESE6OrpONS5LHDVYsesQi7ZlNnYYxpgga65Jw6eu12+JowYvfLmFP8/a0NhhGGNMudatWwOwY8cO+vfv3ygxWOKogcftoqikrLHDMMaEktVT4dn+8Hg778/VUxs7opMuqIlDRMaIyCYR2SoiD1fx/rMistJ5bBaRw37v3SwiW5zHzU5ZSxH5VEQ2isg6EflLMOP3hLkoLrXEYYxxrJ4KH98L2bsB9f78+N56JY+HH36YSZMmlb9+/PHHeeKJJ7jwwgsZOnQoAwYM4KOPPqrxGKWlpTz00EOceeaZDBw4kJdffhmAm266ienTp5dvd8MNN9R6rEAELXGIiBuYBFwKJAPXi0iy/zaqer+qDlbVwcALwAfOvh2Ax4DhwDDgMRFp7+z2jKr2AYYAI0Tk0mBdQ7jbRXGpLa1rjHF8ORGKj1YsKz7qLT9B48ePZ+rUY4ln6tSp3HzzzXz44YesWLGCuXPn8j//8z/UtMz3v//9b6Kioli6dClLly7llVdeYfv27dx66628/vrrAGRnZ7Nw4UIuv/zyE47VJ5jDcYcBW1U1FUBEpgBXA+ur2f56vMkC4BLgc1XNcvb9HBijqu8CcwFUtUhEVgAJwbqAMJdYjcMYc0x2Wt3KAzBkyBDS09PZu3cvGRkZtG/fnk6dOnH//fczb948XC4Xe/bs4cCBA3Tq1KnKY8yZM4fVq1czbdo0bzjZ2WzZsoWLL76Yu+66i4yMDN5//32uvfZawsLq/2c/mIkjHtjt9zoNbw3iOCLSDUgCvqph3/hK+7QDrgT+Xs0x7wDuAOjatWudgwdrqjLGVBKV4DRTVVFeD9dddx3Tpk1j//79jB8/nrfffpuMjAyWL1+Ox+MhMTGxxuGyqsoLL7zAJZdcctx7N910E2+99RZTpkzhtddeq1ecPqHSOT4BmKaqpYFsLCJhwLvA874aTWWqOllVU1Q1JTb2uFmBA2JNVcaYCi58FDwtKpZ5WnjL62H8+PFMmTKFadOmcd1115GdnU1cXBwej4e5c+eyc2eVk9SWu+SSS3jxxRcpLi4GYPPmzeTl5QFwyy238NxzzwGQnJxc3SHqJJg1jj1AF7/XCU5ZVSYAv6y076hK+37t93oysEVVn6tvkDXxuK2pyhjjZ+A4788vJ3qbp6ISvEnDV36C+vXrR25uLvHx8XTu3JkbbriBK6+8kgEDBpCSkkKfPn1q3P+2225jx44dDB06FFUlNja2vFO8Y8eO9O3bl7Fjx9YrRn9SU4dLvQ7srRVsBi7EmwiWAj9W1XWVtusDzAaS1AnG6RxfDgx1NlsBnKGqWSLyBNAXuE5VA/qrnpKSoieyHsfTszfyyvxUtjx5WZ33NcY0DRs2bKBv376NHUbQ5OfnM2DAAFasWEFUVFS121X1OYjIclVNqbxt0JqqVLUEuBv4DNgATFXVdSIyUUSu8tt0AjBF/TKY0yn+R7zJZikw0UkaCcDv8I7SWuEM470tWNfgcZqqgpVcjTEmmL744gv69u3LPffcU2PSqKugTnKoqjOBmZXKHq30+vFq9n0VeLVSWRpw0uYG8Li9pyouVcLDmveUBMaYpueiiy6qtX/kRIRK53hI8ri9H4/1cxhjzDGWOGrgSxwlNrLKmFNac2+Oruv1W+KogSfM+/EUWY3DmFNWZGQkmZmZzTZ5+NbjiIyMDHgfW8ipBuHlfRyWOIw5VSUkJJCWlkZGRkZjh9JofCsABsoSRw2sj8OYU5/H4wl45TvjZU1VNQizxGGMMcexxFEDX1NVUUnzbPs0xpiqWOKoQfmoqjKrcRhjjI8ljhpYH4cxxhzPEkcNfInDmqqMMeYYSxw18NhwXGOMOY4ljhpYU5UxxhzPEkcNLHEYY8zxLHHUwDcjrq0CaIwxx1jiqIHVOIwx5niWOGpgicMYY44X1MQhImNEZJOIbBWRh6t4/1lnFb+VIrJZRA77vXeziGxxHjf7lT8pIrtF5EgwYwcI8905bk1VxhhTLmiTHIqIG5gEjAbSgKUiMkNV1/u2UdX7/ba/BxjiPO8APAakAAosd/Y9BHwM/APYEqzYfcJ9NY4Sq3EYY4xPMGscw4CtqpqqqkXAFODqGra/HnjXeX4J8LmqZjnJ4nNgDICqLlbVfUGMu5w1VRljzPGCmTjigd1+r9OcsuOISDcgCfiqrvsG07G5qqypyhhjfEKlc3wCME1VSxvqgCJyh4gsE5FlJ7pAi6d8dlyrcRhjjE8wE8ceoIvf6wSnrCoTONZMVdd9q6Sqk1U1RVVTYmNj67JrORHB4xZrqjLGGD/BTBxLgV4ikiQi4XiTw4zKG4lIH6A9sMiv+DPgYhFpLyLtgYudspMuzOWyxGGMMX6CljhUtQS4G+8f/A3AVFVdJyITReQqv00nAFPUb6V4Vc0C/og3+SwFJjpliMjTIpIGtBSRNBF5PFjXADg1DuvjMMYYn6CuOa6qM4GZlcoerfT68Wr2fRV4tYryXwO/brgoaxYe5qLIahzGGFMuVDrHQ5bH7aLEEocxxpSzxFELj9tlTVXGGOPHEkctPG6xpipjjPFjiaMWHrfLphwxxhg/ljhq4W2qssRhjDE+ljhq4XGLTTlijDF+LHHUwuN22ZQjxhjjxxJHLcLDrKnKGGP8WeKohQ3HNcaYiixx1CLMZZMcGmOMP0sctfDYlCPGGFOBJY5ahLtdlFhTlTHGlLPEUQtbj8MYYyqyxFELuwHQGGMqssRRC7uPwxhjKrLEUQtbyMkYYyqyxFELa6oyxpiKgpo4RGSMiGwSka0i8nAV7z8rIiudx2YROez33s0issV53OxXfoaIrHGO+byISDCvweN2UVKm+K1sa4wxzVrQlo4VETcwCRgNpAFLRWSGqq73baOq9/ttfw8wxHneAXgMSAEUWO7sewh4EbgdWIJ3WdoxwKxgXUd4mDe3Fpcq4WFBzVHGGNMkBLPGMQzYqqqpqloETAGurmH764F3neeXAJ+rapaTLD4HxohIZ6Ctqi5WbxXgTWBs0K4Abx8HYM1VxhjjCGbiiAd2+71Oc8qOIyLdgCTgq1r2jXeeB3LMO0RkmYgsy8jIOKELAG9TFVjiMMYYn1DpHJ8ATFPV0oY6oKpOVtUUVU2JjY094eOEOYnDph0xxhivYCaOPUAXv9cJTllVJnCsmaqmffc4zwM5ZoMIL2+qss5xY4yB4CaOpUAvEUkSkXC8yWFG5Y1EpA/QHljkV/wZcLGItBeR9sDFwGequg/IEZGznNFUNwEfBfEaypuqSqzGYYwxQBBHValqiYjcjTcJuIFXVXWdiEwElqmqL4lMAKao33hXVc0SkT/iTT4AE1U1y3l+F/A60ALvaKqgjagC6+MwxpjKgpY4AFR1Jt4hs/5lj1Z6/Xg1+74KvFpF+TKgf8NFWTNf4igqsaYqY4yB0OkcD1m+ezesxmGMMV6WOGoR5rKmKmOM8WeJoxYeG45rjDEVWOKoha+pylYBNMYYL0sctbBRVcYYU5EljlpY4jDGmIoscdTiWB+HNVUZYwxY4qhV+ey4tnysMcYAljhqZU1VxhhTkSWOWpQnjjJrqjLGGLDEUatwX+KwpipjjAEscdTKY1OOGGNMBZY4amF9HMYYU5EljlqEubw1DhuOa4wxXpY4aiEieNxiNQ5jjHFY4giAx+2yFQCNMcZhiSMAHrfL1hw3xhhHUBOHiIwRkU0islVEHq5mm3Eisl5E1onIO37lT4nIWucx3q/8AhFZ4ZS/ISJBXcUQvInDplU3xhivoCUOEXEDk4BLgWTgehFJrrRNL+ARYISq9gPuc8ovB4YCg4HhwIMi0lZEXMAbwARV7Q/sBG4O1jX4hLvF7uMwxhhHMGscw4CtqpqqqkXAFODqStvcDkxS1UMAqprulCcD81S1RFXzgNXAGCAaKFLVzc52nwPXBvEaAAhzu6xz3BhjHMFMHPHAbr/XaU6Zv95AbxFZICKLRWSMU74KGCMiLUUkBjgf6AIcBMJEJMXZ7kdO+XFE5A4RWSYiyzIyMup1Id5RVdbHYYwxAEHvHwjg/L2AUUACME9EBqjqHBE5E1gIZACLgFJVVRGZADwrIhHAHKC0qgOr6mRgMkBKSkq9/up7rMZhjDHlglnj2EPF2kCCU+YvDZihqsWquh3YjDeRoKpPqupgVR0NiPMeqrpIVUeq6jBgnq88mMLDLHEYY4xPMBPHUqCXiCSJSDgwAZhRaZvpeGsbOE1SvYFUEXGLSLRTPhAYiLd2gYjEOT8jgN8ALwXxGgAbjmuMMf6C1lSlqiUicjfwGeAGXlXVdSIyEVimqjOc9y4WkfV4m5weUtVMEYkE5osIQA5wo6qWOId+SESuwJv0XlTVr4J1DT4et9hwXGOMcQS1j0NVZwIzK5U96vdcgQech/82BXhHVlV1zIeAhxo82Bp43C6OFJbUvqExxjQDdud4AKxz3BhjjrHEEQCPWyixPg5jjAEscQTEphwxxphjAkocItLKme4DEektIleJiCe4oYWOcGuqMsaYcoHWOOYBkSISj3dY7E+A14MVVKjxuF0Ul1hTlTHGQOCJQ1Q1H/gh8E9VvQ7oF7ywQkuYLeRkjDHlAk4cInI2cAPwqVPmDk5Iocf6OIwxTU1qxhF++tp37M7Kb/BjB5o47sM7/fmHzk183YG5DR5NiAoPc9moKmNMk/LkpxtYuuMQkZ6G/44f0A2AqvoN8A2A00l+UFXvbfBoQpStOW6MaUrmbc7gy43pPHxpH2LbRDT48QMdVfWOs5BSK2AtsF5ETurd243J43ZRUqaUlVmtwxgT2kpKy/jjJ+vpFt2Sn45IDMo5Am2qSlbVHGAsMAtIwjuyqlnwuL0fU3GZ1TqMMaHt7SW72JJ+hN9d1peIsOB0RQeaODzOfRtjcaZBB5rN12+PWwBshlxjTEg7nF/Es19sZkTPaEYndwzaeQJNHC8DO4BWeBdb6oZ31tpmobzGYeuOG2NC2L/mbyfnaDG/vyIZZ3bxoAgocajq86oar6qXqddOvMu5NguVm6qKS8t4+ZttFJZUufigMcY0im82Z3BmYgf6dGob1PME2jkeJSJ/863hLSJ/xVv7aBbCfYnDaar6bnsWf561kYXbMhszLGOMKZdTUMy6vdkM7x4d9HMF2lT1KpALjHMeOcBrwQoq1HjCnD4Op6kqM68IgJyjxY0WkzHG+Fu+4xBlCmcldQj6uQJNHD1U9TFVTXUefwC617aTiIwRkU0islVEHq5mm3Eisl5E1onIO37lT4nIWucx3q/8QhFZISIrReRbEekZ4DWcsPKmKudejqwjhQDkFNjiTsaY0LB4eyYetzCka/ugnyvQxHFURM71vRCREcDRmnYQETcwCbgU72p+14tIcqVteuG9I32EqvbDe4c6InI5MBQYDAwHHhQRX6Pdi8ANqjoYeAf43wCv4YSFubwfk2/akax8b03DahzGmFCxJDWLQQntaBEe/NmgAk0cvwAmicgOEdkB/AP4eS37DAO2OjWUImAKcHWlbW4HJqnqIQBVTXfKk4F5qlqiqnnAamCM854CviQSBewN8BpOWHhYxeG4WXm+GoclDmNM48srLGHNnmyGdw9+MxUEPqpqlaoOAgYCA1V1CHBBLbvFA7v9Xqc5Zf56A71FZIGILBYRX3JYBYwRkZYiEoN3BFcX573bgJkikob3JsS/BHIN9eFrqipxahyH8nw1DmuqMsY0vuU7D1FapgxLCn7HONRxBUBVzXHuIAd4oAHOHwb0AkYB1wOviEg7VZ0DzAQWAu8CiwDf2Nf7gctUNQFvB/3fqjqwiNzhGwWWkZFRryB9icPXVJVpNQ5jTAhZsj0Tt0s4o1vw+zegfkvH1nZ3yR6O1RIAEpwyf2k4d6Kr6nZgM95Egqo+qaqDVXW0c67NIhILDFLVJc7+/wXOqerkqjpZVVNUNSU2NrZOF1aZp9JwXF+NI9c6x40xIeC77Vn0j4+idURA89bWW30SR23zbywFeolIkoiEAxOAGZW2mY63toHTJNUbSBURt4hEO+UD8TaRzQEOAVEi0tvZfzSwoR7XEJDwSneO23BcY0yoKCguZdXu7JMyDNenxvQkIrlUnSAEaFHTvqpaIiJ3A5/hXfTpVWctj4nAMlWd4bx3sYisx9sU9ZCqZopIJDDfuWU+B7hRVUucmG4H3heRMryJ5GeBX+6JCSufq6oMVeVQvpM4rKnKGNPIVuw6RFFp2UnrGIdaEoeqtqnPwVV1Jt6+Cv+yR/2eK96+kgcqbVOAd2RVVcf8EPiwPnHVlX8fR87REkqd6dWtc9wY09iWpGbhEkhJPHmJoz5NVc1GePmoKiXLqW10bBthNQ5jTKNbtC2T5NPa0jbSc9LOaYkjAOVTjpSWld/D0S26FUUlZRQU20SHxpjGsTU9l+92ZHFJcqeTel5LHAHwn3Ik84i3xpEU7Z3j0WodxpjG8tqCHYSHufjx8K4n9byWOAJwrI/jWMd4t5iWgA3JNcY0jsP5Rby/Io1rBscT3brh1xWviSWOAHj8RlX5huKW1zhsSK4xJsj2Hj7KnW8tZ8fBvPKyd7/bTUFxGT89N/Gkx2OJIwD+KwAeyisi0uMirq03w9sMucaYYPtuexaz1u7nx68sZndWPsWlZby5aAcjekYHfdGmqljiCECYy6lxlCmZeUVEt4ooH8FgNQ5jTLBl5HoH5eQWlHD9K4t59dvt7Msu4GcjkholHkscARARwt0uiku9NY72rTy0beEkDuscN8YEWXpuARFhLt6+fTjZR4v586yNJMW04vzT4xolHkscAfK4heKSMrLyiuhQocZhTVXGmOBKzy0krm0EAxPa8ebPhhHTOoJ7LuiJy1XblIHBcXJmxDoFeMK8NY6s/CK6x7Ym0uPC4xZyrcZhjAmy9JxC4tpEAjCka3uW/u5CnCmZGoXVOAIU5nJRVKpkHSmifctwRIS2kR5rqjLGBF16bgGxfkNuGzNpgCWOgIW7hbzCEvKKSoluHQ5A2xYea6oyxgRdhtNUFSoscQTIE+biQE4BAO1behNHm8gwq3EYY4KqoLiUnIIS4tpY4mhyPG4X6c6QuA6tnBpHpMeG4xpjgso3FNfXxxEKLHEEyON2sT/bW+MoTxwtwuwGQGNMUKXnev/uxFqNo+kJdwtHnZlwrcZhjDlZ0nO8NQ5LHE1QmPvYR3WsxuGxSQ6NMUGVccRpqmouneMiMkZENonIVhF5uJptxonIehFZJyLv+JU/JSJrncd4v/L5IrLSeewVkenBvAYf30SHLoEo567xtpFhHC0upchZi9wYYxpaek4hLoHoVqGTOIJ2A6CIuIFJwGggDVgqIjNUdb3fNr2AR4ARqnpIROKc8suBocBgIAL4WkRmqWqOqo702/994KNgXYM/30SH7VqG43bu1vRNO5JbUHzSpzU2xjQP6bkFxLSOKP+7EwqCWeMYBmxV1VRVLQKmAFdX2uZ2YJKqHgJQ1XSnPBmYp6olqpoHrAbG+O8oIm2BC4DpwbuEY3zLx/qaqcA7HBdshlxjTPCk5xaGVP8GBDdxxAO7/V6nOWX+egO9RWSBiCwWEV9yWAWMEZGWIhIDnA90qbTvWOBLVc2p6uQicoeILBORZRkZGfW9lvIah3/iaKgZchenZrJi16Eat8nOL2bwxDnMXru/XucyxjQt3ulGmk/iCEQY0AsYBVwPvCIi7VR1DjATWAi8CywCKi/ufb3zXpVUdbKqpqhqSmxsbL0D9YQ5iaOlX+JooBlyH/1oLRMmL+bbLQer3WbJ9kwO5xcze+2+ep3LGNO0ZBwpDKl7OCC4iWMPFWsJCU6ZvzRghqoWq+p2YDPeRIKqPqmqg1V1NCDOewA4tZBhwKdBjL8Cj9O+2KH18TWO+o6s2ne4gKKSMm57cymLtmVWuc3SHVkALE7NQlXrdT5jTNNQWqZkHgmt6UYguIljKdBLRJJEJByYAMyotM10vLUNXzLoDaSKiFtEop3ygcBAYI7ffj8CPlHVgiDGX0F5U1WFGofTx1GPpqq8whJyC0u47dwkurRvya1vLC1PEv6W7vA2Ze3PKWBHZv4Jn88Y03RkHimkTGk+TVWqWgLcDXwGbACmquo6EZkoIlc5m30GZIrIemAu8JCqZgIeYL5TPhm40TmezwRqaKYKBk+YU+Ooqo+jHk1Vvvmvkk9ry9u3D6dTVCQ//8/yCkN884tKWLsnmzH9OgFUWysxxpxafNMcNafOcVR1pqr2VtUeqvqkU/aoqs5wnquqPqCqyao6QFWnOOUFTlmyqp6lqisrHXeUqs4OZuyVVdU53jLcjdsl9Zohd7+TODq1jSSuTSS/vbQvWXlFLNx2rL9j5a7DlJQp48/sQlybCBanWuIwpjk4Nt1I8+njOKVUNRxXROo9Q65vOoG4tt5fjJG9Y2gTEcbMNcc6wb/bkYUIDO3WnrO6R7MoNdP6OYxpBo5NcNiMahynkqpqHFD/+arKaxxR3sQREebmouSOzFl/gOJSb3PV0h1Z9OnUlqgWHs7uEU1GbiHbMvJO+JzGmKYhFOepAkscAQtzH9/HAfWfIXd/dgGtI8JoHXHsJv7LBnTmcH4xC7dlUlxaxve7DnNmYnsAzu4eDWDNVcY0A+m5hUS18BDpcTd2KBVY4ghQdKtwWoa7q6xx1Gfd8fTcguOG2o3sFUPriDBmrt7H+r055BeVcmZiBwC6RbekU9tIFlniMOaUl55bEHK1DQjiXFWnmnFndmHU6XHHZf62kR62HzzxZqP92QV0alux4yvS4+aivnF8tn4/SbGtABiW5E0cIsLZPaKZvyUDVW30tYeNMcGTnht6d42D1TgCFhHmpkuHlseVe5uq6jMct/C4xAHHmqv+NX87XTu0pKPfNmd3j+bgkSK2ph854fMaY0JfhiWOU1ObenSOl5Wp01R1fOI4r3csrcLdHDxSSIrTv+FzltPPYc1Vxpy6VNVb46ji70Njs8RRT20jPeQVlVJSWvc1ObLyiyguVTpVMZ1ApMfNhX07AjDM6d/w6dKhBV07tOS1BTvKh+sZY04tOUdLKCopsxrHqcg37ciJzFflW8PcNxS3snEpXYj0uBjRM6ZCuYjw13GD2J9dwI3/WsKhvKI6n9sYE9pCca1xH0sc9VSfiQ59vxjVVUXP7RXDuj+MqbJv5czEDvzr5hS2Z+Zx06vf1XuGXmNMaAnV6UbAEke91Wdq9f3Z3l+MqjrHfWpa9WtEzxheunEoG/fncPc739f5/MaY0HXsrnHr4zjltI088RlyD+QUIFK/bxQX9OnIXaN6Mm9zhvV3GHMKOdYiYTWOU06besyQeyCngOhWEeXTmZyoC/vGAbBga/ULQRljmpa0Q0dpFe6mTUTo3W5niaOejq3JcQKd4zkFdIqq/7eJfqdF0b6lh3lb6r9ErjEmNKzZk02/06JC8iZfSxz1VJ8+jupu/qsrt0sY0TOGb7cctFlzjTkFFJeWsX5vDgMSoho7lCpZ4qin1uFhtI0M49/fbmfuxvQ67Xsgp+qb/07EyF4xpOcWsvmA3U1uTFO3aX8uhSVlDGyOiUNExojIJhHZKiIPV7PNOBFZLyLrROQdv/KnRGSt8xjvVy4i8qSIbBaRDSJybzCvoTYul/DGz4bRJjKMn76+lHvf/Z7MI7V3UheWlJKVV9QgNQ6Ac3vFAjDfmquMafLW7MkGYFBCu8YNpBpBSxwi4gYmAZcCycD1IpJcaZtewCPACFXtB9znlF8ODAUGA8OBB0WkrbPbLUAXoI+q9gWmBOsaAjWka3s+uWck913Ui1lr9/Gb91fXuo9vnv2ODTRiIr5dC3rEtmLeFusgN6apW512mLaRYXSLPv4erlAQzBrHMGCrqqaqahHeP/BXV9rmdmCSqh4CUFVfW08yME9VS1Q1D1gNjHHeuxOYqKpllfZpVOFhLu67qDdXDjqNtXtyat3et9Z4xwach2Zkr1iWpGZSUFzaYMc0xpx8q9OyGZjQLiQ7xiG4iSMe2O33Os0p89cb6C0iC0RksYj4ksMqYIyItBSRGOB8vLUMgB7AeBFZJiKznFpLyOge04r9OQXkFdY8yuqAU+OobrqRE3Fe7xgKS8pYvvNQgx3TGHNyFRSXsml/bsj2b0Djd46HAb2AUcD1wCsi0k5V5wAzgYXAu8AiwPc1OgIoUNUU4BXg1aoOLCJ3OMllWUbGyWv3T4ppDcCOzJrX6PAtGduxAe8KHZ4UjcctNizXmCZs/b4cSsqUgSHavwHBTRx7OFZLAEhwyvylATNUtVhVtwOb8SYSVPVJVR2sqqMBcd7z7fOB8/xDYGBVJ1fVyaqaoqopsbGxDXJBgUiK8S68VNviTgdyCggPc9GupafBzt0qIoyhXdszf7P1cxjTVK1J83aMN9cax1Kgl4gkiUg4MAGYUWmb6XhrGzhNUr2BVBFxi0i0Uz4Qb3KY47fP+c7zH3AsoYSExBhvZ9aOABJHp7aRDd6GeV7vWNbvy2Hv4aMNelxjzMmxKu0wMa0j6NyAzdgNLWiJQ1VLgLuBz4ANwFRVXSciE0XkKmezz4BMEVkPzAUeUtVMwAPMd8onAzc6xwP4C3CtiKwB/gzcFqxrOBEtw8Po1DaS1FoSx/7sggYbUeXvqkGn4XYJry3Y3uDHNsYE3+q0bAYlhOYd4z5BnQRFVWfi7avwL3vU77kCDzgP/20K8I6squqYh4HLGzrWhpQU0yqgpqr+8Q1fFe3SoSWXD+jMO0t2cfcFvYhq0XBNYcaY4DpSWMK2jCNcMbBzY4dSo8buHD8lJcXWnDhUtcGmG6nKHed1J6+olLcW7wzK8Y0xwbF2TzaqoXvjn48ljiDoHtOKw/nF1a7Mtze7gKPFpcS3bxGU8/ePj2JkrxheW7DD7ukwpgnxdYyH6hxVPpY4gqB8ZFU1Q3J9c1qdW2lJ2Ib0ix/04OCRQj5YUXkgmzEmVK1KO0x8uxbEtA69NTj8WeIIgkRf4sioPnF06dCCnnGtgxbDOT2iGRAfxSvzUyktsxlzjQl1hSWlzN9ykGFJHRo7lFpZ4giCLu1b4nZJlf0cBcWlLNh2kAtOjwvqqAkR4ec/6M72g3l8ueFA0M5jjGkYczemk320mLFDKk+wEXoscQRBeJiLLu1bVJk4Fm3LpKC4jPP7xAU9jjH9OhHdKpyPVu4N+rmMMfXz/oo9xLWJYESP6MYOpVaWOIIkKaZVlfdyfLUxnRYeN2d1D/4vR5jbxWUDOvPlxgMcqWXuLGNM48nKK2LuxnTGDoknrJ5LSZ8MoR9hE5UU05odB/MqrMinqny1MZ0RPWOI9LhPShxXDjqNguKyJtVcFegqhll5RRSVlAU5GmOC7+NVeykpU344NPSbqSDINwA2Z0mxrThaXOq9X8OZOmDzgSPsOXyUX57f86TFkdKtPZ2jIvl41V6uHtw0fil/8dZyXCI8f/0QPNV8+9qwL4er/7EAgD6d29A/Popbzkmkd8c2JzNUYxrEByvSSO7clj6d2ta+cQiwGkeQdHdGVqUePLaU61fOMNwLTkL/ho/LJVwxsDPfbM7gcP6x+0qW78ziq42hVwvZcTCPz9YdYNba/fz2gzVV1j7KypTffbiG1pFh/HREIq0jwpj+/R7ufmcFZTaCzDQxW9NzWZWW3WRqG2CJI2gSq5gl96uNB0ju3LZB1+AIxJWDTqO4VPls3X4A1u3N5sZ/fcfd73wfcjcIfvj9HkTghuFdeW95Gn//cstx27y3fDcrdh3mkUv78MhlfXnn9rN48pr+bD5whK83139dr437c5j48Xryi0K7Xyi3oJh3luwK+ThNzT5YsQe3S7hq8GmNHUrALHEESee2kUSEucpnyT2UV8TynYdOam3DZ0B8FN2iW/Lxqn0cPFLIHW8uRwTyi0r5etPJW0DxcH4RX244wPaDeVXeW6KqTF+5h3N6RPPE2P786IwEnvtiC28t3lle88jKK+LPszYyLLEDPzojoXzfKwaexmlRkbz0TWq9YiwtUx747ypeXbCd+6asDOl7YJ6avZHffriGH/5zYa2zMZvQVFamTP9+D+f1iiGuAdfmCTZLHEHickn5ZIffbc/i2hcXosCY/p1OeiwiwlWDTmPhtoPc+vpSDh4p5O3bhtOhVTifrtl/UmJQVX41ZSW3vrGM85/5muRHZ3PtiwtJO5Rfvs2KXYfZmZnP2MHxiAh//uEARvaK4X+nr+Xy57/lo5V7+NPMDRwpKOGPY/tXuA/G43Zx68jufLc9ixW7TnwFxLeX7GT9vhzG9OvEnPUH+MusDbVe15YDufzjqy1M/Hg9xaUnp7N+a3ou7363m5G9YtifU8CV//i2SQ2AMF7f7z7E3uyCJtP/6GOJI4iSYloxb8tBxr28iOKyMv7zs+FBmRE3EFcOOo0yhVVp2Tz9o4EM6dqeS/p14ssNB05Kc9XMNfv5ZnMGd43qwdM/GshNZ3dj0/5cHnxvVXm/xIffpxHpcZUnV4/bxb9vPpOnrx1IUWkZv5qykmnL07j13CRO73R8J/iEM7sQ1cLD5BOsdRw8Usgzn23i3J4xvHjjUG4+uxuvzN9e7WSRH6/aywV//YbRz87jmTmbeXXBdp6evfGEzl1Xf5m1kZYeN8+NH8zHd59L1w4tufWNZbz49baAR6WZxjdrzX7C3S4u7HvyWyLqw0ZVBVH/+Chmr9vPz0Yk8eAlvWkZ3ngfd++Obbh2aAK9O7Yu/3ZzxcDOvPvdLr7elM6Y/sGbxjm3oJiJn6yj32lteWB07/Jx6r3i2vDr91fz2sId/OSsbnyyeh+jkzvRJvLYVPDhYS7GndmFH52RwFcb01mcmsm9F1a9zHyriDB+clY3Jn29ldSMI3SPrduULk/N2sjR4lIev6ofIsLvr0hmV1Y+j81YR9/ObTij27GpILKPFvPbD9YQ374FT4ztz+jkjvxz7lZemb+dM7q1D+rnuXDbQb7YkM5vxvQhunUE0cD7d57DQ9NW89Tsjew5nM8fruqP2xW66zkYb2111tr9jOwVU+F3vimwGkcQ3TYyicWPXMijVyY3atLw+eu4Qfz8Bz3KXw9P6kCHVuF8snpfUM/7t883k55byJPXDKhwc9N1KQlc2CeOp2dv5N/fbudwfjE/rGa6BZdLuCi5I/97RTKtIqr/LG8ZkYjH7eKV+XWrdSzdkcV7y9P42blJ5XOIhbldvPDjoUS3CueJTzdU+Cb/5sId5BaW8Ndxg7jxrG50bBvJby/vy6Au7XjovdW1rsdyosrKlD/N3EB8uxb8dERieXmkx83fxw/m5z/ozluLd/Hz/yznaFFoDXwwFa1Oy2bP4aNcOiC0196oiiWOIIoIc9MxSGtuNIQwt7dZ6KuN6UH7I7N2TzZvLNzBjcO7MbhLuwrviQh/vnYALcPdPDV7I9GtwhnZq34zBse0juCawfF8tHIvhSU1X5OqsiQ1kzvfWs6EyYvpHBXJvRdUrM20jgjjgdG9+X7X4fJRaXmFJfx7wXYu7BNHv9OONT1GhLn55w1DcbuFO99afsKjnXIKirnr7eVMXbb7uGan1xbuYO2eHB665PTjbiJ1uYRHLu3LH67qx5cbD3DNPxewcX/OCcVggm/W2v2EuYTRfTs2dih1FtTEISJjRGSTiGwVkYer2WaciKwXkXUi8o5f+VMistZ5jPcrf11EtovISucxOJjXcKq7fEDnOo2uUlU27MvhhS+38OSn62u8b+LrTenc8eYyOrSK4MFLTq9ym7g2kTx5zQDA2w/TENMtXNK/I/lFpXy3PavabQqKS7n+lcWMn7yYRamZ3DYyiQ/vGlFlbeZHZyTQM641T8/eRHFpGe8s2cXh/GJ+ecHxN3LGt2vBc+MHs/lALre9sazO/UdlZcoD/13JzDX7+fW01Tz43mryi0rILSjm/v+u5I+frGdkrxiuGlT90M2bz0nk1VvO5OCRQq56YQH/mp9q97eEGG8z1T7O6RlDVMum1UwFQezjEBE3MAkYDaQBS0Vkhqqu99umF/AIMEJVD4lInFN+OTAUGAxEAF+LyCxV9X19ekhVpwUr9uZkeFIHoluF8+mafbVWmT9auYenZ29iz+Gj5WWDurTjioEV/4hl5RUx8eN1TF+5l55xrXnxukE1LmF72YDOvHpLCkO7tq/fxTjO7h5DRJiLrzamM7JXbJXbPPbROhanZvH7K5L58bCutAivfgqYMLeL34zpw+1vLuPNRTuZPD+VET2jq4131OlxPHPdIP7nvVXc/uYyXrkpJeApZp7/agtfbEjn91ckk3O0mOe/2sLqtMMUlJSy93AB91/Um1+e3wNXLf0X558ex2f3ncfDH6zhiU83MGvtfu4+vyc/6B1b674m+Nbvy2FnZj53+jUdNyXBbHgfBmxV1VQAEZkCXA2s99vmdmCSqh4CUFXf195kYJ6qlgAlIrIaGANMDWK8zVKY28Ul/Tvx4Yo9ZB4pJLqaBWQKikt5bMY6OrWN5KlrBzDq9Dhu/NcS/v7FFi7t37m8I/bgkUIuf34+WXlF3HthL355fg8iwmr/o3lBn4arrrcId3N2j2jmbkznsSv7Hff+f5fu4r/LdvPL83tw67lJAR3zor5xnJnYnic+XY8qPD9hSI3b/3BoAiVlyq+nrebOt5Zz+3nd2Xe4gP05BXRoFc7o5I7HLdbz+foDPPfFFn44NJ6fjUhERDijW3vu++9KWoa7mfrzsyp00NcmunUEk39yBu8tT+PZzzfz09eX0iuuNbef151rhsRXO52LCb7Za/fjEhid3PSaqSC4iSMe2O33Og0YXmmb3gAisgBwA4+r6mxgFfCYiPwVaAmcT8WE86SIPAp8CTysqoWVTy4idwB3AHTt2rVBLuhUdcs5iUxbnsZD01bz75tTqlwnZNbafRzOL2bSj4cywlm58FcX9eLud77nk9XeebBUld9+sIZDecVM+8U5DKrUp3EyXdAnjkc/Wnfc6Ko1adn8/qN1nNszhgdGV918VhUR4eFL+3Ltiws5o1t7zupe+x/wcSldKC1THvlgDXM3ZVR473cfrmF4UjQpie3JPlpM5pEivtmcwYD4KP50zYDyf4Pzescy79fnE+aSE5oYU0QYl9KFa4bE8+nqfbw8L5VfT1vNP+du5f7Rvbly4GlWAznJVJVP1+zjrO7R1X5RC3WNPdQnDOgFjAISgHkiMkBV54jImcBCIANYBPgaix8B9gPhwGTgN8DEygdW1cnO+6SkpFgDbw16d2zDby/tw+Mfr+eNhTu4ZcTx38LfXryLpJhWnO03Hfxl/Ttzeset/P3LLVwx8DSmf7+HOesP8NvL+jRq0gBvUw2s46uN6eWJI7egmDvfXk5Mq3Cev35InYerntGtPc+OH8SA+KiAF+G6flhX+p8WRW5BMZ3btaBT20h2ZOYxa80+Pl2zjxe+2kpUCw/RrcNJSWzPk9cMOC5BtK5hFFmgPG4XY4fEc/Xg0/hyQzrPzNnEr6as5MWvt3HnqB5c2r8z4WFWAzkZtqYfITUjj59W8f+sqQhm4tgDdPF7neCU+UsDlqhqMbBdRDbjTSRLVfVJ4EkAp9N8M4Cq+saOForIa8CDwbuE5uPmcxKZv+Ugf5q1keHdo+nb+dgsnZv257Js5yF+d1nfCt9OXS7hvot6cefbK3jpm2289M02hiV24NZzuzfGJVTQpUNLesW1Zu6mdG4b6Y3nb59vZs/ho0z7xTl0aBV+Qse9ZkhC7RtVMiCh4k2ffTu3pW/ntjxw8emUlJad1PUXRLzDmi/oE8fHq/fy9y+28KspK/lT2w385Kxu3DC8G+1P8LMxgflms7f2eWEjTD/UUIL5G7sU6CUiSSISDkwAZlTaZjre2gYiEoO36SpVRNwiEu2UDwQGAnOc152dnwKMBdYG8RqaDRHh6R8NpF0LD/e8+z15fgs/vbNkJ+FuF9eecfwfzUv6daJv57b832ebKC1TnrluUMjceHZ+nzi+257FkcIS1u31Dgu+YXhXzujWMJ3wDaGxFu1xuYSrB8fzxQM/4NVbUujdsQ3PzNnMuU99xf99tpFDeUW1H6QG+UUlbE0/UvuGgVg9FZ7tD4+38/5c3bS7OhdsPUj3mFac1q5FY4dywoL2W+t0bN8NfAZsAKaq6joRmSgiVzmbfQZkish6YC7e0VKZgAeY75RPBm50jgfwtoisAdYAMcATwbqG5ia6dQR/GzeY1IwjjHt5Efuyj5JfVMIHK/Zw2YBOVX5Ld7mEBy/ujQj87+XJdI1u2QiRV+380+MoLlXmb87g99PX0r5lOA9d3KexwwopLpdwQZ+O/OfW4Xx233mc3yeOf369jZFPz+Wp2RvZn11Q7b6FJaVMmruVf81PJT3Xu11pmTLlu12M+r+vuehv3/D4jHX1m9Jm9VT4+F7I3g2o9+fH9zbZ5FFcWsaS7Vnl/YRNlTSHeW1SUlJ02bJljR1GkzF3Yzr3vPs9LcLdXD6gM68v3MF7vzibMxOr7xA+eKTwuFFCja24tIyhf/yc1hFh7Msu4JnrBlWYUddUbdP+XJ7/aguz1uzDJd71XG49t3uFJrfs/GJ+/tYyFqd675VxCYzsFcv+7AI2HchlaNd2nN6pDe9+t5sesa34+4QhAc/Tpqq8Mj+VF7/exmy9i46acfxGUV3g/qbX2LB0RxbXvbSIl248o1EmPK0rEVmuqimVyxu7c9yEoPP7xPHBXedw6xtLeX3hDnrFtSalluadUEsa4O0QPq9XLJ+u2ceZie25tgktlNOYTu/Uhkk/HsquzHxeX7iDqct2M33lXs7o1p6bz0lkQHwUt7+5jJ2ZeTw3fjD946P48Ps0pn+/lwiPixdvGMqY/p0QES7t35mHpq1i7KQFPHPdIMZWM6WMT/bRYh58bxWfrz/AuT1jiEs7WOV2mp1GaDSI1s23Ww7iEioMMmmKrMZhqpWVV8QfP1nPlYM6N+h9FifTrDX7uH/qSqb/ckSTWZYz1OQUFPPesjTeXLSDnZneafDbRobx8k9SOLtH7X8AD+cX8Yu3lrNkexZPjO3PDcO7HbdNYUkpczdm8OTM9ew7XMBvL+vLT0ckIs8NcJqpKkori+HeTm9yy4gkLu3fKWTuSdlz+Chr0rJJO5RP2qGjDOnarsKU6de9tJCikjI+uvvcRowycNXVOCxxmFOaqlJYUnZC90CYisrKlG82ZzBn/X5uPTeJnnGBr+9eUFzKXW+v4KuN6TxyaR9uOKsbOzPz2JWZz9ebMpi5dh+5BSXEt2vB89cPOTaAwdfHUXxstgINa8G8Pr/nse3J7MjMJ6F9Cx69IpnRyR2rHCa9cX8OM1fvo1t0K64ZEl8+MlBVeWvxTj5ds49Hr+hH8mn1+2KxdEcWN/5rCYUl3jVZ3C7B7RLmPjiK+HYtOFJYwuA/zOH287rzmzFNo6/NEoclDmMaVVFJGfdPXcmnlWZjbhXu5pL+nbh6cDwjekQfP9Js9VT4ciJkp0FUAlz4KAwcR1mZMndTOk/P3sSmA7mMOj2Why/tgyqkHTrK1vQjfLxqL+v3HZvo8UznXpk2kWH8etpq5m85SHiYC7cIfxs3qHzanR0H83h36S4iwtyc2zOGwV3a1Xify5YDuVz74kJiWkfwt/GDSYxuSV5RKec/8zVXDTqNZ64bxFcbD/Cz15fx9m3Dm0znuCUOSxzGNLrSMu+3/PyiUrpFt6Rrh5b0jGtdrxphcWkZby7aybOfb+ZIYcUZiQcmRHHt0ASuGNiZLzem8+eZG8gtKKGFx01JmfK7y/tycXJHfvHWclbsOszPRiSRdiifzzccwC1CmSplCi3D3QxL6sC5PWM4t1cMp3dsU1672Zd9lGv/uZDiMuWDO8+hS4djIwv/NHMDr8xPZdavRjJ1aRpvL9nJqscubjI1YEscljiMOaWl5xQwe91+oltF0KVDCxLatzxuCHlWXhFPzdrIvpwCJl7Vj8SYVoC3j+X309cydVka7Vp6uHF4N246uxsRYW4WpWaycNtBvt16kNQM7zorUS08dI6KJK5tJDsz88g8UsSUO846buTY4fwiznt6LimJHdh7+CjRrcN5+7azTs4H0gAscVjiMMbUQFVZlZbN6R3bVDtb8p7DR1mw9SCrdh/mQE4h6bkFFBSX8viV/Tinmuanl77Zxl9meZcU/vWY07lr1PHT8YcqG45rjDE1EJHjFhurLL5dC8aldGFcSpcat/N3yzmJvL5gB/tzCji3ifRt1CY0xrAZY8wpKtLj5g9X9+P802MrrBjZlFmNwxhjguySfp24pF/o3ykeKKtxGGOMqRNLHMYYY+rEEocxxpg6scRhjDGmTixxGGOMqRNLHMYYY+rEEocxxpg6scRhjDGmTprFXFUikgHsdF5GAdk1PK/8MwaoehmyqvkfM5D3KpdV97qmWBszxlD/DAOJ1VPH+E5GjKH+GVYus8/w1PwMu6lq7HFHVtVm9QAm1/S8ip/LTvT4gbxXuay617XE2mgxhvpnGEisdY3PPsPjy+wzPDU/w+oezbGp6uNanlf+WZ/jB/Je5bLqXtcUa101ZIyh/hkGGmtdNffPsLr368I+w8DOVZOT8X/lOM2iqao+RGSZVjGtcCgJ9RgtvvoL9RhDPT4I/RhDPT5/zbHGUVeTGzuAAIR6jBZf/YV6jKEeH4R+jKEeXzmrcRhjjKkTq3EYY4ypE0scxhhj6sQShzHGmDqxxFEPIjJSRF4SkX+JyMLGjqcyEXGJyJMi8oKI3NzY8VRFREaJyHzncxzV2PFURURaicgyEbmisWOpioj0dT6/aSJyZ2PHU5mIjBWRV0TkvyJycWPHUxUR6S4i/xaRaY0di4/ze/eG89nd0Njx+Gu2iUNEXhWRdBFZW6l8jIhsEpGtIvJwTcdQ1fmq+gvgE+CNUIsPuBpIAIqBtIaMrwFjVOAIENnQMTZQfAC/AaY2ZGwNGaOqbnB+D8cBI0IwvumqejvwC2B8Q8bXgDGmquqtDR1bZXWM9YfANOezuyrYsdVJXe9UPFUewHnAUGCtX5kb2AZ0B8KBVUAyMABvcvB/xPntNxVoE2rxAQ8DP3f2nRaKnyHgcvbrCLwdgvGNBiYAtwBXhOJn6OxzFTAL+HEoxufs91dgaKh+hsH6f1KPWB8BBjvbvBPMuOr6CKOZUtV5IpJYqXgYsFVVUwFEZApwtar+GaiymUJEugLZqpobavGJSBpQ5Lwsbcj4GipGP4eAiFCLz2k+a4X3P/JREZmpqmWhFKNznBnADBH5FHgnlOITEQH+AsxS1RUNFVtDxniy1CVWvDXwBGAlIdY61GwTRzXigd1+r9OA4bXscyvwWtAiqqiu8X0AvCAiI4F5wQzMT51iFJEfApcA7YB/BDUyrzrFp6q/AxCRW4CDDZk0alDXz3AU3maNCGBmMANz1PX38B7gIiBKRHqq6kvBDM5R188wGngSGCIijzgJ5mSpLtbngX+IyOWc+JQkQWGJo55U9bHGjqE6qpqPN7GFLFX9AG+CC2mq+npjx1AdVf0a+LqRw6iWqj6P949gyFLVTLx9MCFDVfOAnzZ2HFUJqepPCNgDdPF7neCUhYpQjw9CP8ZQjw9CP8ZQjw+aRow+TSlWwBJHZUuBXiKSJCLheDtFZzRyTP5CPT4I/RhDPT4I/RhDPT5oGjH6NKVYvRq7d76xHsC7wD6ODVW91Sm/DNiMd5TD7yy+phtjqMfXFGIM9fiaSoxNMdaaHjbJoTHGmDqxpipjjDF1YonDGGNMnVjiMMYYUyeWOIwxxtSJJQ5jjDF1YonDGGNMnVjiMM2SiBw5yedrkPVaxLt+SbaIrBSRjSLyTAD7jBWR5IY4vzFgicOYBiEiNc77pqrnNODp5qvqYGAIcIWI1LYGx1i8s/sa0yAscRjjEJEeIjJbRJaLd1XCPk75lSKyRES+F5EvRKSjU/64iPxHRBYA/3FevyoiX4tIqojc63fsI87PUc7705waw9vOtOOIyGVO2XIReV5EPqkpXlU9infK7Xhn/9tFZKmIrBKR90WkpYicg3etjv9zaik9qrtOYwJlicOYYyYD96jqGcCDwD+d8m+Bs1R1CDAF+LXfPsnARap6vfO6D95p4ocBj4mIp4rzDAHuc/btDowQkUjgZeBS5/yxtQUrIu2BXhybMv8DVT1TVQcBG/BOZ7EQ77xHD6nqYFXdVsN1GhMQm1bdGEBEWgPnAO85FQA4trBUAvBfEemMd4W27X67znC++ft8qqqFQKGIpONd2bDykrjfqWqac96VQCLe5XNTVdV37HeBO6oJd6SIrMKbNJ5T1f1OeX8ReQLv2iatgc/qeJ3GBMQShzFeLuCw03dQ2QvA31R1hrNo0uN+7+VV2rbQ73kpVf8fC2SbmsxX1StEJAlYLCJTVXUl8DowVlVXOQtPjapi35qu05iAWFOVMYCq5gDbReQ68C53KiKDnLejOLY+ws1BCmET0N1vWdHxte3g1E7+AvzGKWoD7HOax27w2zTXea+26zQmIJY4THPVUkTS/B4P4P1je6vTDLQO77rP4K1hvCciy4GDwQjGae66C5jtnCcXyA5g15eA85yE83tgCbAA2Oi3zRTgIadzvwfVX6cxAbFp1Y0JESLSWlWPOKOsJgFbVPXZxo7LmMqsxmFM6Ljd6Sxfh7d57OXGDceYqlmNwxhjTJ1YjcMYY0ydWOIwxhhTJ5Y4jDHG1IklDmOMMXViicMYY0ydWOIwxhhTJ/8PzX90rNa5yV0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e30ade1d-125f-42d3-9eb9-2bf54bf2bde0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.251922</td>\n",
       "      <td>0.257991</td>\n",
       "      <td>0.892286</td>\n",
       "      <td>0.957853</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.248150</td>\n",
       "      <td>0.243493</td>\n",
       "      <td>0.899114</td>\n",
       "      <td>0.963003</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.231296</td>\n",
       "      <td>0.237253</td>\n",
       "      <td>0.900543</td>\n",
       "      <td>0.964258</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit(3, 3e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07caba6f-b7ec-4ec4-8dca-a8fef8a527b1",
   "metadata": {},
   "source": [
    "This model is much worse than the pretrained ResNet34 model that we fine-tuned in the previous lecture. We can also see that we've started overfitting quite strongly after the first epoch (because the training and validation losses are diverging). Let's now see if we can make the model more accurate and train it even faster by using some of the techniques that the factory learners (like `vision_learner()`) provide under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3ba0b8-0187-4820-85da-46d0a4e1c9bd",
   "metadata": {},
   "source": [
    "## Debugging training with activation statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40fa303d-a681-4990-9de9-897568a25b6c",
   "metadata": {},
   "source": [
    "One way to train a more accurate CNN is simply double the number of filters in each stride-2 layer. However, increasing the number of filters also requires us to increase the kernel size. For example, if our first layer increases the number of filters from 4 to 8, then a $3 \\times 3$ kernel will not force the CNN to learn any useful features. To handle that, we'll increase the kernel size to $5\\times 5$ and adjust the CNN architecture as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7514c2ef-8197-4d4e-a4a9-a3e8cded781c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_cnn():\n",
    "    return nn.Sequential(\n",
    "        conv(1, 8, ks=5),  # 20x20\n",
    "        conv(8, 16),  # 10x10\n",
    "        conv(16, 32),  # 5x5\n",
    "        conv(32, 64),  # 3x3\n",
    "        conv(64, 128),  # 2x2\n",
    "        conv(128, 2, act=False),  # 1x1\n",
    "        Flatten(),\n",
    "    ).to(\"cuda\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f3f54fa-beea-4cee-adef-7b1860241292",
   "metadata": {},
   "source": [
    "Since we'll be training this network in several ways, let's wrap the creation of the `Learner` and training in a `fit()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "31d9faea-10ff-4f5c-b1b3-ee931e9b76c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(epochs=1, lr=0.06):\n",
    "    learn = Learner(\n",
    "        dls,\n",
    "        simple_cnn(),\n",
    "        metrics=[accuracy, RocAucBinary()],\n",
    "        loss_func=F.cross_entropy,\n",
    "        cbs=ActivationStats(with_hist=True),\n",
    "    )\n",
    "    learn.fit(epochs, lr)\n",
    "    return learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07fbbea4-ea40-4cac-a19e-7977ddb0a6b3",
   "metadata": {},
   "source": [
    "To give a sense for what can go wrong when training CNNs, we've used a much larger learning rate than before. We've also use the `ActivationStats()` callback, which records the statistics of the activations in each layer - this will be handy for debugging. Let's see how well this model performs after 1 epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "91b843fb-a642-4a09-a0b5-2857f6c980dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lewis/miniconda3/envs/dl4phys/lib/python3.9/site-packages/fastai/callback/core.py:67: UserWarning: You are shadowing an attribute (modules) that exists in the learner. Use `self.learn.modules` to avoid this\n",
      "  warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.694601</td>\n",
       "      <td>0.693960</td>\n",
       "      <td>0.500343</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>00:19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5252429-272e-41af-9fd0-b6c96e6dd591",
   "metadata": {},
   "source": [
    "Okay, it seems the large learning rate has given us a model than doesn't perform better than random chance. To diagnose the problem, we can use the `plot_layer_stats()` method that the `ActivationStats` callback has tracked. For example, here are the mean and standard deviation of the activations in the first layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "26a2013b-032b-4eb6-9690-0a4d957f5527",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAADSCAYAAACMwpyAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuo0lEQVR4nO3de7hdVX3v//dnrX3LhZCExBAIIcEGIV7RDcjxxpFbvIFPT6VAbaMHfxz7q9pqj4pHqxS1T2p7bB/PQ62p5UgvSKm2JeeIcgKC/noUTUBAEoSECCQhJJEkkPvea63v748519ozO2tfVtZOdubcn9fzrOx5GXPOsebeI/M7xxxjTEUEZmZmZmY2eqXxzoCZmZmZWd44iDYzMzMza5GDaDMzMzOzFjmINjMzMzNrkYNoMzMzM7MWOYg2MzMzM2uRg2gzswlO0g2S/mG882FmlicOos3MCswBstnoSfpLSTsl/VjSvMzyayR9ZTzzZscfB9FmZmY24Uk6D3gdcDLw78D16fITgY8Dnxm/3A2Q1HEU910+WvsuIgfRE5ikpyR9XNIjkvZK+ltJcyR9V9JuSXdLmpGmfb2kH0naJelhSRdm9vN+SY+l22yQ9F8y6y6UtEnSH0raJmmLpPcf+29rVnySPilpc1oWH5f0DuC/Ab8paY+kh9N0CyX9IE23Epg1rhk3Oz4sBP49Ig4C9wBnpMu/CPxZRLw43MaSviHpJknfScvWTyS9NLP+LEkrJe1Iy+eVmXXvkPQzSS9K2ijphsy6BZJC0rWSngG+3+TY/yst4/VPTdL7RnHcb0j6qqQ7Je0F/qOksyXdl17v10i6/AjO5YTgINr+E3AJcCbwLuC7JBfd2SR/Hx+RdCrwHeALwEzgvwLfljQ73cc24J3ANOD9wF9Iem3mGCcDJwKnAtcCN9WDczMbG5JeBnwIODciTgAuA34B/AnwTxExNSJenSa/FXiAJHj+PLB0HLJsdrxZA7xJ0iTgImCNpF7gZRFx6yj3cRXwx8AMYD1JAI6kKcBKkrL3kjTdX0lanG63F/gdYDrwDuB3Jb170L7fApxNUrYPERHvSsv4VOA9wHPAPaM4LsA1aT5PAH4C/C/g/6TpPwz8Y/r/iw3iINr+R0RsjYjNwP8H/CQifhYRB4B/Bc4B3gvcGRF3RkQtIlYCq4G3A0TEdyLiyUj8gKTwvSlzjH7gxojoj4g7gT2AC6TZ2KoC3cBiSZ0R8VREPDk4kaT5wLnAH0XEwYj4IclF02xCi4hHgW8D9wPzgS8BXyGpTPqIpB9K+kdJ04fZzb9GxE8jogL8I/CadPk7gaci4n9GRCUifpYe6z3pse+LiJ+n19hHgG+SBM1ZN0TE3ojYP9TBJZ0J3AJcGREbRzpu6o6I+L8RUUvzOxVYFhF9EfF94H8DVw/znScsB9G2NTO9v8n8VOB04D3po51dknYBbwTmAkh6m6T700dFu0iC6+zj4efT/1Dq9qX7NbMxEhHrgT8AbgC2SbpN0ilNkp4C7IyIvZllTx/9HJod/yLiLyLi1RHxm8CVwA9JYqXrSGqnHyNtKz2E5zLT2Wvd6cD5g66jv0XypBZJ50u6V9J2SS8AH+TwZlYbh8t72nb7DuAzEfHvozluk/2eAmxMA+q6p0meJNsgDqJtNDYCfx8R0zOfKRGxTFI3yV3tnwNzImI6cCegccyv2YQUEbdGxBtJLpwB/Gn6M2sLMCN9zFs3/xhl0SwXJM0hCZxvBF4BPBIR/cAq4FVHsMuNwA8GXUenRsTvputvBVYAp0XEicBfc/h1dHBZzua3lO7j3ohY3sJxB+/3WeC0dH9184HNrXzZicJBtI3GPwDvknSZpLKknrTD4Dygi+QR8nagIultwKXjmVmziUjSyyS9Nb2xPUDyJKlG8nRpQf2iGBFPkzTH+mNJXZLeSNIfwswGfJmk+cQ+4JfAuZKmAhcCG45gf/8bOFPSb0vqTD/nSjo7XX8CsCMiDigZJeSaFvf/RWAK8PstHnewn5DUoH8iTXshyf8Pt7WYnwnBQbSNKG1XdQVJh8PtJHe2HwdKEbEb+AhwO7CTpOCvGKesmk1k3cAy4Fckj5RfAnwK+Od0/fOSHkynrwHOB3YAnwP+7thm1ez4JemtwPSI+FeAiPgpSef6jcB/JClnLUmvlZeSdOx7lqSM/ilJuQX4f4EbJe0GPktyTW3F1cDrgZ2ZETp+axTHHZzPPpKg+W0k/5f8FfA7EfGLFvMzIShiyKcDZmZmZmbWhGuizczMzMxa5CDazMzMzKxFDqLNzMzMzFrkINrMzMzMrEUOos3MzMzMWtQx3hk4ErNmzYoFCxaMdzbMjhsPPPDAryJi9njnoxmXV7NDubya5cdw5TWXQfSCBQtYvXr1eGfD7Lgh6bh9bbPLq9mhXF7N8mO48urmHGZmZmZmLXIQbWZmZmbWIgfRZmZmZmYtchBtZmZmZtaitjoWSpoJ/BOwAHgKuDIidjZJtxT4TDr7hYi4JV3+ReB3gBkRMbWdvJi1IiKIgEot6Ooo8cL+fiSo1YJf7emjq1wiCA5Wauw9WOGkKd2USlCpBpt27ufpHXs5feYUOspi1S93cMbsqfzyV3s4beZkJLH3YIVHN7/AeQtncrBS4+9+/BRXnTuf/mqN3QcqbN99kPkzJ7P7QD9f+f56zpk/nd7TZ7Bjbz/3b3ieOdO6kcT5C2dy0dlzeN3pM8b7lB0z1Vrw8W89zL88uLmx7IxZU3jnq+byle+vp6Mk/uidi5ncVeYf7n+a375gAc/u2s/uA/1M6+nkrLnTqNaCzrLo7igzubvMrCnd9FVrTO4q091RoqezTLkkKmm6zlKJUknj+K3N8mHH3j7+adVGPviWM5BcZmxiU0Qc+cbSl4AdEbFM0vUkwfAnB6WZCawGeoEAHgBeFxE7Jb0eeBpY10oQ3dvbG+49fHyp1YIA9vVV2Lb7IHOm9VBLA9W1z77I1hcP8ML+fjbv2s8Zs6bwf9ZuZcbkLr736Bauf/vZfPXe9fR0ldn6wgEWzTmBhzbuAuDVp03n4Y27mNbTwYsHKsPmYWp3B3sODqQ5dfokNu/afxS/9bHx2Xcu5j+/ceGwaSQ9EBG9xyhLLWmlvNZqwfu/sYofPLH9KOfqcF3lEn3VGgCzT+imJNj64kF6OkucNmMy67btYVpPB6fNnExJoqczeZC36qmdvGnRLMolUZYol0RHWZRLJTpKyfyB/io9nWV6Okt0lJLtJnWVKUtIIKBUEpM6y1TTctPdUaKzXEKCcikJ9Ov7HpzvIElfLomOUolaRLKdhCRKSvZfEum8kmOmxy9JlErpz6HSpNuXS0IMpBNQLovujhKlIYKqUnpeLJHn8nrtN1Zxzy+28e3fvYDXnT7zGObMbHwMV17bHeLuCuDCdPoW4D7gk4PSXAasjIgdaWZWAkuAb0bE/emyNrMxsdUD2L5KjWoE1VpyY/TcCwd4cvse9vdV+fnmFzj9pMn8+V2Pc9bcaTzw9GEPDMbNH/3bo4fM1wNogIfT6ZECaIA9Byt0pLWLAC/u7z9k/cnTeqjUavxqTx8XnHESO/b28cyOfVzZO4/vrXmOrS8ePCT9lK4yl73iZP7lwc1c+8aF/NpLpiLgmR37eHL7Ht7+yrl0d5T52cadnHPaDJ7cvoez554AwNTuTsol6Oks01UucbBSY8aULg70V+nuKDG1uyMJRiT291U5cVJnEuhIBJCNN9q4z82d+57Yxg+e2M6vv/ZUrl9yFl0dJSTRV6lREuzrqzZqjw/011jz7Au8/JQTuX/D82zffZATejqYNqmTksSOvQfZ11dlUmeZKd0dbH3xAPNmTGLH3n729VXo6SwTEax8bBsCzl84k40793HipE4g+Zv7ziNbOG/hSXSVxSnTJ7Frfz/TJ3VSqdXorwT7+iucNKWLXfv6KSl5slHNfOrzm3ftZ0pXme7OMvv6Khzor1EuqVFWJ4J60F2WCKIR3NcD8lJ6N5EE6fXAPpkGkKCjVEqeMnQkgXwtLRyNbUoigsaNgerr0muM0v0oPW59v0on6usH0g4sa+REHLKskT6dP3POCXz2XYuP7skcR7vTior+6sT52zUbSrtB9JyI2JJOPwfMaZLmVGBjZn5Tuqwlkq4DrgOYP39+q5uPq4iBi+mDT+/k5BN7eGLrbvqqwT+v3sj+viqrj2FQe6wC6GvOn8+OPX18b81znDp9Eq889UQe2bSLi86ewyObX+Cis17Cdx7Zwg2Xv5wntu5m0ZypbNqxn7nTe9jXV+WNvzaLAGoR1GrRCG7qF8mxvPn64yteMeS6L1/5mmG3XfKKk9s69tTu4YvhRLrHXPHQs8yc0sWf/qdX0Vk+vMvGSYPmf+0lUw/5eSQ+9NZFQ6676Zoj3u2wIpIgsloLBFQj6E9rwesqtaCvUqO/WkOISq1GtRZp8FIPYJIbjCCoVKNxE12tRXJDFkEtaPysRTSeENUGrRspTUSSz0qy4JBtqrXgYKU65PftryZ5CoJqLfmbrkVAeoz6viMzn80HJPOVajSa4URE46Yzu4+yRDX9PyO73/TMkx628XsIaCzLPplNlqXpo7H1QNoaBLVM2mT7A8OcBzMrlhGDaEl3A82ihE9nZyIiJB21W9OIWA4sh+Rx09E4Rn+1RkdJ/GpPH1tfPMBJU7v4xXO7eXTTC8w/aTL/7V9+zt6+/PwH+dr503nwmV2N+Xe9+hR6Okps2pk0cZh7Yg+LT5nGBS89idlTu6lF8hh794F+TpzUidKL8NF8UvCRi5IA5oKXpuHRS0feZiIFlRPNA8/s5IIzTmoaQBdJvUzVmziUUOG/s5lZ0YwYREfExUOtk7RV0tyI2CJpLrCtSbLNDDT5AJhH0uzjmKnVgm89sIlPfPuRY3nYlr1p0SwWzprCvr4qr5p3IrOmdhMBb3nZbKZ0lRvNFDoyjyePhumTuxrTbmpjx8rBSpVNO/fz6+fMG++smNkIJlIzM7OhtNucYwWwFFiW/ryjSZq7gD+RVB9e4FLgU20ed9R++ssdXPm1H4/5fk/o7mD3wQp/fPnLeetZL+HFA/3MmzEZAlSCyZ1lOsa4Zqmz7IDWiuuZ5/cRAWfMnjLeWTGzIfgqZDag3SB6GXC7pGtJRtm4EkBSL/DBiPhAROyQ9HlgVbrNjZlOhl8CrgEmS9oEfD0ibmgzTw0R0TSAfsuZs5k+uZN1W/fw8lOm8fElL2PWlG5qEWMe+JrZ6Gz41V4AFs5yEG1mZse/toLoiHgeuKjJ8tXABzLzNwM3N0n3CeAT7eRhOI9t2d2YfmrZO0ZMX/I9ttm42b47GR1lzrSecc6JmZnZyApd7fr39z8FwCeWvGx8M2JmI9q1rw+A6ZM7xzknZmZmIyt0EN3dUQbgg28exZAPZjaudu7rT8ZSTsutmZnZ8azQQfT6bXt49bwT/TpfsxzYubfvkJFhzCwhaYmkxyWtT98OPHj9fEn3SvqZpEckvT1dvkDSfkkPpZ+/Pva5NyuudjsWHte2vLCfl518wnhnw8xGYee+PmZMcVMOsyxJZeAm4BKSl5WtkrQiItZmkn0GuD0ivippMXAnsCBd92REvOYYZtlswih0TfQL+/tds2XWxChqtj4o6edp7dW/pxfm+rpPpds9LumyscrTzn39zHB5NRvsPGB9RGyIiD7gNuCKQWkCmJZOnwg8e7QzFXigaLPCBtERwa59/Uyf5Jots6xMzdbbgMXA1dkgOXVrRLwyrcH6EvDldNvFwFXAy4ElwF+l+2vb3oOVEV+BbjYBnQpszMxvSpdl3QC8Nx0q9k7gw5l1C9NmHj+Q9KZ2M+P3b5kNKGwQvbevSqUW7ulvdrgRa7Yi4sXM7BRoVDtdAdwWEQcj4pfA+nR/bdvfX2VSpzsVmh2Bq4FvRMQ84O3A30sqAVuA+RFxDvAx4FZJ0wZvLOk6Saslrd6+ffsxzbhZnhU2iG4MlzXJj4fNBhlNzRaSfk/SkyQ10R9pcduWL8oH+qv0dDmINhtkM3BaZn5euizrWuB2gIj4MdADzEpvdp9Plz8APAmcOfgAEbE8Inojonf27NlH4SuYFVOBg+h+AE50TbTZEYmImyLipcAnSToutbJtyxfl/X2uiTZrYhWwSNJCSV0kzalWDErzDOmLzySdTRJEb5c0u97cStIZwCJgwzHLuVnBFbYB4gv7kyDabaLNDjOamq2s24CvHuG2oxIRbs5h1kREVCR9CLgLKAM3R8QaSTcCqyNiBfCHwN9I+ihJ06v3RURIejNwo6R+oAZ8MCJ2jNNXMSucwgbR9Zpoj85hdphGzRZJAHwVcE02gaRFEbEunX0HUJ9eQdKu8svAKSQ1Wz9tN0N91Rq1gEluzmF2mIi4k6TDYHbZZzPTa4E3NNnu28C3j3oGzSao4gbR+/0KYbNmRlmz9SFJFwP9wE5gabrtGkm3A2uBCvB7EVFtN08H+moArok2M7PcKG4QXW8T7eYcZocZRc3W7w+z7ReBL45lfvb3J3G4a6LNcsLDRJsVt2Phiwf66SqX6HHNltlxrxFEu7yaHdeEB4o2qytsEH2wv0ZPZ2G/nlmh7O9Lgmjf9JqZWV4UNso80F+l2xdks1xwcw4zM8ubwgbRByuuiTbLiwNpEN3T4TJrlgduEm1W4CD6QH+V7g7XapnlQV81GZ2jy0G0mZnlRGGvWK6JNsuP/koSRHeWXWbN8sDdC80KHEQf6K/S45pos1yo1JKHww6izcwsLwp7xTpYqdHtmmizXOiv1muiXb9lZmb5UNgo0zXRZvnR5+YcZrnijoVmBQ6iXRNtlh9uzmGWD/LDIrOGwl6xDlaqdPmCbJYLbs5hZmZ5U9gos1oNOhxEm+VCvTmHy6yZmeVFYa9Y/bVwrZZZTtSbc/jpkZmZ5UVhr1iVao2OUmG/nlmhDIwT7RtfMzPLh8JGmZVq0OELslku1NtEl0sus2Zmlg9tBdGSZkpaKWld+nPGEOmWpmnWSVqaLpss6TuSfiFpjaRl7eRlsP5azT39zXKivxZ0lUvIXf/NzCwn2o0yrwfuiYhFwD3p/CEkzQQ+B5wPnAd8LhNs/3lEnAWcA7xB0tvazE9DpRp0uFbLLBf6KzU35TDLkfBA0WZtB9FXALek07cA726S5jJgZUTsiIidwEpgSUTsi4h7ASKiD3gQmNdmfkj3R6XmINosL/qrNY/MYZYDflhkNqDdq9aciNiSTj8HzGmS5lRgY2Z+U7qsQdJ04F0ktdlNSbpO0mpJq7dv3z5spqppT39flM3yIRlNx+XVzMzyY8SrlqS7JT3a5HNFNl1EBEfwJlBJHcA3ga9ExIah0kXE8ojojYje2bNnD7vPSiOI9i2zWR70V2p0ubyaNSVpiaTHJa2X1KzZ5HxJ90r6maRHJL09s+5T6XaPS7rs2ObcrNg6RkoQERcPtU7SVklzI2KLpLnAtibJNgMXZubnAfdl5pcD6yLiL0eT4dFovP3MQ9yZ5YKbc5g1J6kM3ARcQvIkd5WkFRGxNpPsM8DtEfFVSYuBO4EF6fRVwMuBU4C7JZ0ZEdVj+y3Miqndq9YKYGk6vRS4o0mau4BLJc1IOxRemi5D0heAE4E/aDMfh6hUXRNtlif9NQ9JaTaE84D1EbEh7T90G0l/pKwApqXTJwLPptNXALdFxMGI+CWwPt2fmY2BdoPoZcAlktYBF6fzSOqV9HWAiNgBfB5YlX5ujIgdkuYBnwYWAw9KekjSB9rMD5AMbwduE23WzCgeDX9M0tr0sfA9kk7PrKumZfUhSSvGKk8RQdk9lsyaGbFfEXAD8F5Jm0hqoT/cwrZmdoRGbM4xnIh4HrioyfLVwAcy8zcDNw9Kswk4KlfNRsdCj85hdohRPhr+GdAbEfsk/S7wJeA303X7I+I1Y52vWg1KDqLNjtTVwDci4r9LugD4e0mvGO3Gkq4DrgOYP3/+UcqiWfEUsqq20ZzDQbTZYCM+Go6IeyNiXzp7P2M09ORwqhEeOsusuc3AaZn5eemyrGuB2wEi4sdADzBrlNu21HG/sU3r4wiYFU4hg+hGx0I35zAbrNXHu9cC383M96RDTd4v6d1DbdTKkJSQNufwTa9ZM6uARZIWSuoi6Sg4uCnVM6RPhSWdTRJEb0/TXSWpW9JCYBHw03Yyo6PzANksl9pqznG88hB3Zu2T9F6gF3hLZvHpEbFZ0hnA9yX9PCKeHLxtRCwnGXmH3t7eEausauHmHGbNRERF0odIOuSXgZsjYo2kG4HVEbEC+EPgbyR9lKST4fvSYWfXSLodWAtUgN/zyBxmY6eQQXS9JrrDQ9yZDTaqx7uSLibp+PuWiDhYXx4Rm9OfGyTdB5wDHBZEt6paC1wRbdZcRNxJ0mEwu+yzmem1wBuG2PaLwBePagbNJqhCRpn1NtGdrok2G2zER8OSzgG+BlweEdsyy2dI6k6nZ5FctLMdEo9YLYKSo2gzM8uRQtZE15tzuI2l2aFG+Wj4z4CpwD8raWLxTERcDpwNfE1SjeQGfNmgUT3ayJebc5iZWb4UM4h2x0KzIY3i0XDTt5RGxI+AVx6NPNXCzTnMzCxfChllVjxOtFmuVGuBXBNtZmY5UsggutGx0DXRZrkQgd9YaJYj4WGizYoZRLtjoVm+JB0LxzsXZjYS3+uaDSjkZcsdC83ypRrhjoVmZpYrhQyia+Eg2ixP/LIVMzPLm2IH0b4om+VCeHQOs1xxk2izggbR1bQ5h3v7m+VDzc05zMwsZwoZRNd7Dbs5h1k+VGv4jYVmOeLSalbQILpeE+1rslk+uDmHmZnlTSGD6HqbaD8eNssHN+cwM7O8KXYQ7aots1yo1hxEm+WJOxaaFTSITl9Y6NE5zHIiwje9ZmaWL4UMogdqosc5I2Y2KjW3iTYzs5wpZJjpNtFm+eKXrZiZWd4UM4iu+WUrZnniNtFmZpY3hQyiq2mPB1+UzfLBQ9yZmVneFDKIDreJNssVN+cwM7O8KWSYOfCyFV+UzfKgFuGbXjMzy5VCXrZqfu23Wa74ZStmQ5O0RNLjktZLur7J+r+Q9FD6eULSrsy6ambdinbzUr+uVupjyZpNYB3jnYGjoT46h6/JZvng5hxmzUkqAzcBlwCbgFWSVkTE2nqaiPhoJv2HgXMyu9gfEa8Zq/xM7U7Chj0HK2O1S7PcKmRNdNWjc5jliseJNhvSecD6iNgQEX3AbcAVw6S/Gvjm0crMtEmdALx4wEG0WVtBtKSZklZKWpf+nDFEuqVpmnWSlmaWf0/Sw5LWSPrr9I67bR4n2ixfqrXwGwvNmjsV2JiZ35QuO4yk04GFwPczi3skrZZ0v6R3t5uZE9Ka6N0H+tvdlVnutVsTfT1wT0QsAu5J5w8haSbwOeB8kjvqz2WC7Ssj4tXAK4DZwHvazA8wME60L8pmzY2ijeXHJK2V9Iike9KLc31d05vidoSbc5iNhauAb0VENbPs9IjoBa4B/lLSSwdvJOm6NNBevX379mEPUL+upnVVZhNau0H0FcAt6fQtwLubpLkMWBkROyJiJ7ASWAIQES+maTqALmBMimUt3KnQbCiZNpZvAxYDV0taPCjZz4DeiHgV8C3gS+m2w90UHzE35zAb0mbgtMz8vHRZM1cxqClHRGxOf24A7uPQ9tL1NMsjojciemfPnj0WeTabENoNoudExJZ0+jlgTpM0wz6KknQXsA3YTXKxbqqVO+WqL8hmwxmxjWVE3BsR+9LZ+0ku3DDMTXE7PDqH2ZBWAYskLZTURRIoHzbKhqSzgBnAjzPLZkjqTqdnAW8A1g7ethUupWYDRgyiJd0t6dEmn8EX3eAIapIj4jJgLtANvHWYdKO+U/YF2WxYo25jmboW+G4r27Zy0wtQq7n5lVkzEVEBPgTcBTwG3B4RayTdKOnyTNKrgNsiDmlocTawWtLDwL3AsuyoHm3mayx2Y5ZrIw5xFxEXD7VO0lZJcyNii6S5JDXKg20GLszMzyN5pJQ9xgFJd5DUhq0cRb6HVas5iDYbC5LeC/QCb2llu4hYDiwH6O3tHfFq6+YcZkOLiDuBOwct++yg+RuabPcj4JVjmRdfWs0GtNucYwVQ71i0FLijSZq7gEvTx0ozgEuBuyRNTQNvJHUA7wB+0WZ+ALeJNhvBqNpYSroY+DRweUQcbGXbVvnpkZmZ5U27QfQy4BJJ64CL03kk9Ur6OkBE7AA+T9KuaxVwY7psCrBC0iPAQyS12H/dZn6AZLgsX4/NhjRiG0tJ5wBfIwmgs0+Ymt4Ut5OZiPDLVsxyxq05zNp8Y2FEPA9c1GT5auADmfmbgZsHpdkKnNvO8YdSi3BNtNkQIqIiqd7GsgzcXG9jCayOiBXAnwFTgX9WEtw+ExGXR8QOSfWbYhi4KW4jP8lPB9Fmxz+5a6FZQ2Ff++0LstnQRmpjOVxfiGY3xe0YeDnSWO3RzI42V0SbFfa1367VMsuLWr0m2lG0mZnlSCGD6IigXMhvZlY8AzXRDqLNzCw/ChlqVj3EnVluuDmHWX740mo2oJBBtHv6m+VHzR0LzXLHo3OYFTaIDkqF/GZmxdOoiXZVtNlxz6XUbEAhQ81aBGXXapnlQq3m5hxmeRMen8OsmEG020Sb5Yebc5jliMupWUMhg+gIPxo2ywt3LDTLH7eJNitoEJ3URI93LsxsNBrNOVxozcwsR4oZRPuNhWa54eYcZvnj4mpW0CA6edmKS7hZHrg5h1n+uDmHWUGDaHcsNMsPv7HQLD9cSs0GFDKIrrljoVlu1GrJTwfRZvnhimizwgbR7lholhcDL1sZ54yY2Yh8r2s2oJCXLb9sxSw/3JzDLIfcKNqsmEG020Sb5YeDaLP8kFtFmzUUMohO2kSPdy7MbDQ8xJ2ZmeVRIUPNmmuizXLDQ9yZDU/SEkmPS1ov6fom6/9C0kPp5wlJuzLrlkpal36WHtOMmxVcx3hnYKxt332Q1U/vHO9smNkoNUbncBRtdhhJZeAm4BJgE7BK0oqIWFtPExEfzaT/MHBOOj0T+BzQSzKgxgPptm1fJN0i2qyANdHhzg5mueI20WbDOg9YHxEbIqIPuA24Ypj0VwPfTKcvA1ZGxI40cF4JLGknMy6mZgMKF0SbWb64OYfZsE4FNmbmN6XLDiPpdGAh8P1WtpV0naTVklZv3759VJlyfZVZEYNoX4jNcsUdC83GzFXAtyKi2spGEbE8Inojonf27NnDpnUpNRtQvCDazIY1ik5Kb5b0oKSKpN8YtK6a6cC0YizyM/CyFV+ezZrYDJyWmZ+XLmvmKgaacrS6rZm1yEG02QSS6aT0NmAxcLWkxYOSPQO8D7i1yS72R8Rr0s/lY5GnWs3NOcyGsQpYJGmhpC6SQPmwG1hJZwEzgB9nFt8FXCpphqQZwKXpsraFuxaaFW90Dg8EbzasRiclAEn1TkrZnv5PpetqxyJDbs5hNrSIqEj6EEnwWwZujog1km4EVkdEPaC+CrgtMr3rI2KHpM+TBOIAN0bEjmOZf7MiK1wQbWbDatbR6PwWtu+RtBqoAMsi4t+aJZJ0HXAdwPz584fdoUfnMBteRNwJ3Dlo2WcHzd8wxLY3AzcftcyZTWBuzmFmrTg9InqBa4C/lPTSZola6ajk5hxm+eF7XbMBbQXRkmZKWpm+CWll2uaqWbph35gkaYWkR9vJy8C+xmIvZoXVVkejiNic/twA3Ef6Uod2NJpzOIo2yw0PcWfWfk309cA9EbEIuCedP0TmjUnnk7TH/Fw22Jb068CeNvNhZqMzqk5KzaSdk7rT6VnAG8i0pT5SHifaLD/kmiqzhnaD6CuAW9LpW4B3N0kz5BuTJE0FPgZ8oc18mNkoREQFqHdSegy4vd5JSdLlAJLOlbQJeA/wNUlr0s3PBlZLehi4l6RN9BgG0b44m+WFK6LN2u9YOCcitqTTzwFzmqQZ7o1Jnwf+O7BvpAONtqOSL8Nmwxupk1JErCJp5jF4ux8Brxzr/DiINjOzPBoxiJZ0N3Byk1Wfzs5EREga9c2ppNcAL42Ij0paMFL6iFgOLAfo7e31TbBZQdTSgfQcRJvlh9tEm40iiI6Ii4daJ2mrpLkRsUXSXGBbk2SbgQsz8/NIOiRdAPRKeirNx0sk3RcRF9IGt9cyy5eBNxaOc0bMbES+xJoNaPeytQKoj7axFLijSZqmb0yKiK9GxCkRsQB4I/BEuwG0meWPm3OYmVketRtELwMukbQOuDidR1KvpK9D8sYkkrbPq9LPMXljkq/HZvngNxaamVketdWxMCKeBy5qsnw18IHM/LBvTEpfM/yKdvJSV78M+4Jslg8e4s7MzPKosK0Qyw6izXLBL1sxy5/wIHdmxQ2iHUOb5cPAa79daM2Od/JAsmYNhQui69dhX4/N8sHNOcxyyBXRZsULout8t2yWD+5YaJYfLqZmAwobRJtZPgyME+2rs1leuCLarIBBtGugzfJloE30OGfEzMysBYULouv8yMksH+rNOTyijpmZ5Ulhg2gzy4d6cw45iDY77rmUmg0oXhDtEm6WKx6dwyx/Itwq2qx4QbSZ5YrHiTbLDxdTswGFC6JdwM3yxW8sNMsfV0SbFTCIrvPl2Cwf3JzDbHiSlkh6XNJ6SdcPkeZKSWslrZF0a2Z5VdJD6WdF23nx1dWsoWO8M2BmE9tAEO2Ls9lgksrATcAlwCZglaQVEbE2k2YR8CngDRGxU9JLMrvYHxGvOZZ5NpsoClcT7cuwWb40hrhzVbRZM+cB6yNiQ0T0AbcBVwxK8/8AN0XEToCI2HaM82g2IRUuiDazfBkY4m6cM2J2fDoV2JiZ35QuyzoTOFPS/5V0v6QlmXU9klany9/d7ACSrkvTrN6+ffuYZt6syArbnMNjzprlg0fnMGtbB7AIuBCYB/xQ0isjYhdwekRslnQG8H1JP4+IJ7MbR8RyYDlAb2/vqLoMul+hmWuizWycNUbncBBt1sxm4LTM/Lx0WdYmYEVE9EfEL4EnSIJqImJz+nMDcB9wTjuZcTE1G1C4INp3x2YjG6m3v6Q3S3pQUkXSbwxat1TSuvSztN28rHn2BcCjc5gNYRWwSNJCSV3AVcDgUTb+jaQWGkmzSJp3bJA0Q1J3ZvkbgLWMAQ9xZ1bg5hxm1txoevsDzwDvA/7roG1nAp8DeknuWR9It915pPm5a83W+r6PdBdmhRURFUkfAu4CysDNEbFG0o3A6ohYka67VNJaoAp8PCKel/QfgK9JqpFUmi0bVM7NrA0Oos0mnkZvfwBJ9d7+jYtrRDyVrqsN2vYyYGVE7EjXrwSWAN88+tk2m5gi4k7gzkHLPpuZDuBj6Seb5kfAK49Knvzc16yAzTnScu06LbMhjaa3/9HY1szMrDAKF0Sb2fjzkFlmxeY20WYOos0motH09m9r24hYHhG9EdE7e/bsI86omR1ffuN18wCY0l0e55yYjT8H0WYTz2h6+w+l3oFphqQZwKXpMjObAKZP7kLykJRm4CDabMKJiApQ7+3/GHB7vbe/pMsBJJ0raRPwHpLe/WvSbXcAnycJxFcBN9Y7GZrZxODw2SxRvNE56u20XMrNhjSK3v6rSJpqNNv2ZuDmo5pBMzOz45xros3MzKwl7lho5iDazMzMWuAXI5kl2mrOkb697J+ABcBTwJXN3lyWvhr4M+nsFyLilnT5fcBcYH+67tKI2NZOnhrHHIudmJmZ2SGmdndw92NbmTapg8ldSRgRmarpWjpZTSc27dzP1J4OpvV0ZNIEEUkLzI6SqNQCkczX1wHUaslrXSLS5cnBGi03+6tBuZR0dIyAagT9lRrlspJtMzXmAZQE1Rp0pOv7q7VGnkvK5D2CzpKoRfJimWotDulMGUClWqMWUJbSNFCpJcuyMYiUzFfTY9S/d/2cReOf5Fj1PEcMvNSmvk19mqbLB/bX2Meg3032fNTPc7mkxnEbv5dMfga+h1D6fbKqtWict4HjDPzeguT7l1T//SZpyun5rudP0qF5SpfXYuCY2fxLA38X9d9NLeKQtLVI/r7q+x28n86y+Nv3ncusqd0ciXbbRF8P3BMRyyRdn85/MptgFK8J/q2IWN1mPhp6upLK9avOmz9WuzSzo+h1p8/ggaeP+K3hZnaM/cHFi/jS9x7nT+78xbjloR4MZYMnSAKmkpQEvSVRzkR8UpKuo1SiUqshRDWCSZ3lZF0t6CgnMUQS9EYaNCoN0uOQALKcHqtSC0qCjlKJzrIaQV8MCszrQWQ9z2r8Q+M4A9P15TrkmE3TNEmrzIpD9zcQrJYkSoIDlUiD3OT7ICiVkv1BEmALNW5iDmnKE8n3HkhPI/1A/nRIIF+SkmC6lpzPJGBOfjflUrKXSuampX7esr/vepBez3cSPKtxI1Y/ZZKo1mqUpMbvJHvO+qtBZ/nIG2W0G0RfAVyYTt8C3MegIJpj/Jrg7o4yj39hCV1tnBQzO3Zu/y8XNC6AZnb8e/8bFvK+/7CAPQcr7DlYAZJAph4UltMgqaMk+qtBRBLQdnWUGoFlPeCp1AOpRsCkQwLkei1uYzoTRNXSGsv6PLipiR1b7QbRcyJiSzr9HDCnSZqRXhP8PyVVgW+TNPVoejWVdB1wHcD8+cPXMnd3eBB4s7wol0TZDbDMckUSJ/R0ckJP57gdv9ykhtbsWBoxiJZ0N3Byk1Wfzs5EREhqtTrptyJis6QTSILo3wb+rlnCiFgOLAfo7e11tZWZmZmZjZsRg+iIuHiodZK2SpobEVskzQWadQrczECTD0jGnr0v3ffm9OduSbcC5zFEEG1mZmZmdrxot+HwCmBpOr0UuKNJmqavCZbUIWkWgKRO4J3Ao23mx8zMzMzsqGs3iF4GXCJpHXBxOo+kXklfh2FfE9xNEkw/AjxEUmP9N23mx8zMzMzsqGurY2FEPA9c1GT5auADmfnDXhMcEXuB17VzfDMzMzOz8aAhBsM4rknaDjw9QrJZwK+OQXaOZz4HiYlwHk6PiNnjnYlmXF5HzecgMRHOg8trMfg8TIxzMGR5zWUQPRqSVkdE73jnYzz5HCR8Ho5//h35HNT5PBz//DtK+Dz4HPiNJGZmZmZmLXIQbWZmZmbWoiIH0cvHOwPHAZ+DhM/D8c+/I5+DOp+H459/Rwmfhwl+DgrbJtrMzMzM7Ggpck20mZmZmdlRUbggWtISSY9LWi/p+vHOz1iTdLOkbZIezSybKWmlpHXpzxnpckn6SnouHpH02sw2S9P06yQtbXas45Wk0yTdK2mtpDWSfj9dPqHOQxG4vBb/79TltViKXGZdXl1eWxYRhfkAZeBJ4AygC3gYWDze+Rrj7/hm4LXAo5llXwKuT6evB/40nX478F1AwOuBn6TLZwIb0p8z0ukZ4/3dWjgHc4HXptMnAE8Aiyfaecj7x+V1YvydurwW51P0Muvy6vLa6qdoNdHnAesjYkNE9AG3AVeMc57GVET8ENgxaPEVwC3p9C3AuzPL/y4S9wPTJc0FLgNWRsSOiNgJrASWHPXMj5GI2BIRD6bTu4HHgFOZYOehAFxeJ8DfqctroRS6zLq8ury2qmhB9KnAxsz8pnRZ0c2JiC3p9HPAnHR6qPNRmPMkaQFwDvATJvB5yKmJev4n7N+py2vuTcTfwYT9O3V5HVnRgugJL5LnKBNiyBVJU4FvA38QES9m102k82D5NZH+Tl1eLe8m0t+py+voFC2I3gyclpmfly4ruq3p4xPSn9vS5UOdj9yfJ0mdJAX8HyPiX9LFE+485NxEPf8T7u/U5bUwJuLvYML9nbq8jl7RguhVwCJJCyV1AVcBK8Y5T8fCCqDe83UpcEdm+e+kvWdfD7yQPo65C7hU0oy0h+2l6bJckCTgb4HHIuLLmVUT6jwUgMvrBPg7dXktlIlYZifU36nLa4vGu2fjWH9Ieoo+QdKD+NPjnZ+j8P2+CWwB+knaGF0LnATcA6wD7gZmpmkF3JSei58DvZn9/Gdgffp5/3h/rxbPwRtJHiU9AjyUft4+0c5DET4ur8X/O3V5LdanyGXW5dXltdWP31hoZmZmZtaiojXnMDMzMzM76hxEm5mZmZm1yEG0mZmZmVmLHESbmZmZmbXIQbSZmZmZWYscRJuZmZmZtchBtJmZmZlZixxEm5mZmZm16P8HIA/vFa46NYoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8babf04d-4c04-49f7-bc8c-241548fe2dbe",
   "metadata": {},
   "source": [
    "Here we can see that the mean and standard deviation have a somewhat dramatic spike / drop in the early stages of training. A bigger problem is that the fraction of activations that are near zero is almost 1 after a few iterations. This is a problem because activations that are zero in one layer, will propagate zeros to the next layer, and so on. As a result, these parts of the network are effectively turned off, which makes the learning process suboptimal. \n",
    "\n",
    "Let's now have a look at the last convolutional layer in the CNN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d4f4304b-708d-4869-8c3c-3a6c5f6828d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAADSCAYAAACMwpyAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAArtElEQVR4nO3de5RcZZnv8e+vu5MOkkCuBEwIiRKU4AW0AV0qIteAQpgZkIAjwYOLcUZ0RtQRBg9glBkYPaKeYZyJEAEREVEOGYlmQgBvA5hwCwSEhIiQEEjMhTsJSZ7zx34rKYrq7qpUVXfXrt9nrVq9L+/e9XSl3+yn9n4vigjMzMzMzKxybf0dgJmZmZlZs3ESbWZmZmZWJSfRZmZmZmZVchJtZmZmZlYlJ9FmZmZmZlVyEm1mZmZmViUn0WZmLU7ShZKu6e84zMyaiZNoM7Mcc4JsVjlJ35K0XtIdksYXbT9V0nf6MzYbeJxEm5mZWcuTdBDwbmB34LfAOWn7rsAXgS/3X3TbSepo4LnbG3XuPHIS3cIkPS7pi5IWS3pR0hWSxkr6haTnJd0iaUQq+x5J/yNpg6T7JR1adJ5PSHo4HbNc0t8U7TtU0gpJn5e0WtIqSZ/o+9/WLP8kfUnSylQXH5H0YeCfgJMlvSDp/lRukqRfpXLzgdH9GrjZwDAJ+G1EbAQWAG9K2y8Cvh4Rz/V0sKQrJV0m6eZUt+6S9Oai/W+VNF/SulQ/P1q078OS7pX0nKQnJV1YtG+ipJB0hqQngFvLvPd/pTpeeG2VdHoF73ulpO9KmivpReBDkvaVdHu63i+RdPwOfJYtwUm0/RVwJLAPcBzwC7KL7hiyv4/PShoH3Ax8DRgJfAH4qaQx6RyrgY8AuwCfAC6V9K6i99gd2BUYB5wBXFZIzs2sPiS9BTgLODAihgFHA38A/hn4cUQMjYh3puLXAneTJc9fBWb0Q8hmA80S4AOSdgIOB5ZI6gLeEhHXVniO6cBXgBHAMrIEHEk7A/PJ6t5uqdy/S5qSjnsROA0YDnwY+FtJJ5Sc+4PAvmR1+zUi4rhUx4cCJwFPAwsqeF+AU1Ocw4C7gP8C/juV/wzww/T/i5VwEm3/NyKeiYiVwG+AuyLi3oh4BbgROAD4a2BuRMyNiK0RMR9YBBwLEBE3R8RjkfkVWeX7QNF7vArMjIhXI2Iu8ALgCmlWX1uATmCKpEER8XhEPFZaSNIE4EDgf0fExoj4NdlF06ylRcSDwE+BO4EJwL8C3yG7mfRZSb+W9ENJw3s4zY0R8fuI2Az8ENg/bf8I8HhEfD8iNkfEvem9TkrvfXtEPJCusYuBH5ElzcUujIgXI+Ll7t5c0j7AVcBHI+LJ3t43uSkifhcRW1O8Q4GLI2JTRNwK/Bw4pYffuWU5ibZnipZfLrM+FNgLOCk92tkgaQPwfmAPAEnHSLozPSraQJZcFz8eXpv+Qyl4KZ3XzOokIpYB/wBcCKyWdJ2kN5Yp+kZgfUS8WLTtT42P0Gzgi4hLI+KdEXEy8FHg12S50plkd6cfJrWV7sbTRcvF17q9gINLrqMfI3tSi6SDJd0maY2kZ4FP8fpmVk/2FHtqu30T8OWI+G0l71vmvG8EnkwJdcGfyJ4kWwkn0VaJJ4EfRMTwotfOEXGxpE6yb7XfAMZGxHBgLqB+jNesJUXEtRHxfrILZwCXpJ/FVgEj0mPeggl9FKJZU5A0lixxngm8DVgcEa8CC4F37MApnwR+VXIdHRoRf5v2XwvMAfaMiF2B/+D119HSulwcb1s6x20RMauK9y0971PAnul8BROAldX8sq3CSbRV4hrgOElHS2qXNCR1GBwPDCZ7hLwG2CzpGOCo/gzWrBVJeoukw9IX21fIniRtJXu6NLFwUYyIP5E1x/qKpMGS3k/WH8LMtvsmWfOJl4A/AgdKGgocCizfgfP9HNhH0sclDUqvAyXtm/YPA9ZFxCvKRgk5tcrzXwTsDPx9le9b6i6yO+j/mMoeSvb/w3VVxtMSnERbr1K7qmlkHQ7XkH2z/SLQFhHPA58FrgfWk1X8Of0Uqlkr6wQuBv5M9kh5N+Bc4Cdp/1pJ96TlU4GDgXXABcDVfRuq2cAl6TBgeETcCBARvyfrXP8k8CGyelaVdK08iqxj31NkdfQSsnoL8HfATEnPA+eTXVOrcQrwHmB90QgdH6vgfUvj3ESWNB9D9n/JvwOnRcQfqoynJSii26cDZmZmZmZWhu9Em5mZmZlVyUm0mZmZmVmVnESbmZmZmVXJSbSZmZmZWZWcRJuZmZmZVamjvwPYEaNHj46JEyf2dxhmA8bdd9/954gY099xlOP6avZarq9mzaOn+tqUSfTEiRNZtGhRf4dhNmBIGrDTNru+mr2W66tZ8+ipvro5h5mZmZlZlZxEm5mZmZlVyUm0mZnZACVptqTVkh7sZr8kfUfSMkmLJb2raN8MSUvTa0bfRW3WGpxEm5mZDVxXAlN72H8MMDm9zgS+CyBpJHABcDBwEHCBpBENjdSsxTRlx8Le3HjvCibvNoy3jdu1v0Mxs1489NRzPPLMc/zFAeP7OxSzAScifi1pYg9FpgFXR0QAd0oaLmkP4FBgfkSsA5A0nywZ/1Et8Ty48llO+d6dvHP8cPbYdUgtpzIbED79ob2ZOHrnHTo2l0n05358PwCPX/zhfo7EzHpz7Hd+A+Ak2mzHjAOeLFpfkbZ1t/11JJ1JdhebCRMm9PhmK9a/zPOvbOa3y/7MG51EWw6c9t6JO3xsXZJoSVOBbwPtwOURcXHJ/k7gauDdwFrg5Ih4PO07FzgD2AJ8NiLm1SMmM+teBXX2EOBbwDuA6RFxQ9G+CcDlwJ5AAMcW6rOZNZ+ImAXMAujq6opKjhk3fCd+d85hDY3LbKCruU20pHbgMrJ2WVOAUyRNKSl2BrA+IvYGLgUuScdOAaYD+5E9Zvr3dD4za5AK6+wTwOnAtWVOcTXw9YjYl6yt5erGRWtmvVhJ9oW2YHza1t32upDqdSaz5lWPjoUHAcsiYnlEbAKuI2ujVWwacFVavgE4XJLS9usiYmNE/BFYls5nZo3Ta52NiMcjYjGwtXh7SrY7ImJ+KvdCRLzUR3Gb2evNAU5Lo3S8B3g2IlYB84CjJI1IHQqPStvqos1ZtFldmnOUa3d1cHdlImKzpGeBUWn7nSXHlm2zVY0xwzo5Yt+xtZ7GLK8qqbPd2QfYIOlnwCTgFuCciNhSXKiaNpZm1j1JPyLrJDha0gqyETcGAUTEfwBzgWPJbkK9BHwi7Vsn6avAwnSqmYVOhvWJq15nMmteTdOx0BdlswGhA/gAcABZk48fkzX7uKK40I60sTSz14uIU3rZH8Cnu9k3G5jdiLh8J9qsPs05Kml3ta2MpA5gV7IOhhW32YqIWRHRFRFdY8aMqSAsX7fNulFLW8kVwH2pKchm4P8B7+r5EDPLG+fQZvVJohcCkyVNkjSYrKPgnJIyc4DCbEknAremb89zgOmSOiVNIhss/ve1BuS6bdajSupsT8cOl1T4JnsY8FADYjSzAcx3os3qkESnu1FnkXVYeBi4PiKWSJop6fhU7ApglKRlwNnAOenYJcD1ZBfhXwKfLm1baWb1VUmdlXRgan95EvCfkpakY7cAXwAWSHqA7Dvr9/rj9zCz/uMU2qxObaIjYi5Z54bibecXLb9CdjEud+xFwEX1iOO15633Gc3yo4I6u5CsmUe5Y+eTjR9tZi2mcAPad6LN6tOcY8Bx3TYzM2scX2fNcppEm5mZWeP4TrRZjpNoN+cwMzNrjLbcZg9mlctlNZC7PJiZmTWM70Sb5TSJNjMzs8aRk2iz/CbR4clWzMzMGqLNObRZPpNof0E2MzNrHF9mzXKaRJuZmVn9FZJnt4k2cxJtZmZmVXplsycXNsttEu0h7szMzOprw0uvAvDgyuf6ORKz/pfLJNoPmczMzOrvuVde7e8QzAaMXCbRZmZmeSFpqqRHJC2TdE6Z/XtJWiBpsaTbJY0v2rdF0n3pNafWWDZv9WNes4LcJtGu5mbdq+CifIikeyRtlnRimf27SFoh6d/6JmKz1iSpHbgMOAaYApwiaUpJsW8AV0fEO4CZwL8U7Xs5IvZPr+NrjcdD25ltl8sk2oPAm3WvwovyE8DpwLXdnOarwK8bFaOZbXMQsCwilkfEJuA6YFpJmSnArWn5tjL768ajcphtl8sk2sx61OtFOSIej4jFwNbSgyW9GxgL/HdfBGvW4sYBTxatr0jbit0P/GVa/gtgmKRRaX2IpEWS7pR0Qq3B+CaV2Xa5TaI9OodZtyq5KJclqQ34P8AXeil3ZrpwL1qzZs0OB2pmFfkC8EFJ9wIfBFYChTHo9oqILuBU4FuS3lx6cDX11c05zLbLbRJtZg3xd8DciFjRU6GImBURXRHRNWbMmD4KzSyXVgJ7Fq2PT9u2iYinIuIvI+IA4Ly0bUP6uTL9XA7cDhxQ+gY7Ul9HDx1c9S9iljc1JdGSRkqaL2lp+jmim3IzUpmlkmYUbb89dW4q9BzerZZ4zKwivV6Ue/Be4CxJj5N1ZjpN0sX1Dc/MiiwEJkuaJGkwMB14zSgbkkanp0QA5wKz0/YRkjoLZYD3AQ/VI6iPvOON9TiNWVOr9U70OcCCiJgMLEjrryFpJHABcDBZW8wLSpLtjxX1HF5dYzzbhMfnMOtOrxfl7kTExyJiQkRMJHuEfHVEvK7em1l9RMRm4CxgHvAwcH1ELJE0U1JhtI1DgUckPUrWX+GitH1fYJGk+8k6HF4cETUl0W4qabZdR43HTyOrvABXkT0q+lJJmaOB+RGxDkDSfGAq8KMa37tb7vdg1r2I2CypcFFuB2YXLsrAooiYI+lA4EZgBHCcpK9ExH79GLZZy4qIucDckm3nFy3fANxQ5rj/Ad7e8ADNWlStSfTYiFiVlp8m+wZcqrdOTN+XtAX4KfC1CH/PNWu0Ci7KC8maefR0jiuBKxsQnpmZ2YDXaxIt6RZg9zK7ziteiYiQVG0C/LGIWClpGFkS/XHg6m7iOBM4E2DChAm9n9mpuJmZmZk1SK9JdEQc0d0+Sc9I2iMiVknaAyjXpnkl25t8QHZ36/Z07kKv4eclXUvWZrpsEh0Rs4BZAF1dXT2myG7OYWZmVn++P2W2Xa0dC+cAhdE2ZgA3lSkzDzgq9RIeARwFzJPUkXoLI2kQ8BHgwRrjMTMzswbzzSqz2pPoi4EjJS0FjkjrSOqSdDlA6lD4VbIRARYCM9O2TrJkejFwH9kd6+/VGM82/rZsZmbWGO69ZFZjx8KIWAscXmb7IuCTReuzSeNWFm17EXh3Le/fHeGvyGZmZvXmq6vZdp6x0MzMzCriG9Bm2zmJNjMzs6q4TbRZjpNoDzdtZmZmZo2SyyTa35DNzMzqzzeozLbLZRJtZmZmZtZIuU2i/V3ZzMysMTwKlllOk2hXbTMzMzNrpFwm0WbWM0lTJT0iaZmkc8rsP0TSPZI2SzqxaPv+ku6QtETSYkkn923kZmZmA0NNk60MVI+vfYkhg9r7OwyzAUlSO3AZcCSwAlgoaU5EPFRU7AngdOALJYe/BJwWEUslvRG4W9K8iNjQ+MjNzMwGjlwm0QB/ePr5/g7BbKA6CFgWEcsBJF0HTAO2JdER8Xjat7X4wIh4tGj5KUmrgTHAhoZHbWZmNoC4OYdZ6xkHPFm0viJtq4qkg4DBwGNl9p0paZGkRWvWrNnhQM2souZXe0lakJpY3S5pfNG+GZKWpteMvo3cLN+cRJtZ1STtAfwA+EREbC3dHxGzIqIrIrrGjBnT9wGa5URR86tjgCnAKZKmlBT7BnB1RLwDmAn8Szp2JHABcDDZE6gLJI3oq9jN8s5JtFnrWQnsWbQ+Pm2riKRdgJuB8yLizjrHZmavta35VURsAgrNr4pNAW5Ny7cV7T8amB8R6yJiPTAfmFqPoDypmZmTaLNWtBCYLGmSpMHAdGBOJQem8jeS3fW6oYExmlmmkuZX9wN/mZb/AhgmaVSFx7r5ldkOchJt1mIiYjNwFjAPeBi4PiKWSJop6XgASQdKWgGcBPynpCXp8I8ChwCnS7ovvfbv+9/CzIp8AfigpHuBD5I9WdpS6cFufmW2Y3I5Osfuuwyhvc3Pmsy6ExFzgbkl284vWl5I1syj9LhrgGsaHqCZFfTa/CoiniLdiZY0FPiriNggaSVwaMmxtzcyWLNWkss70ZPHDmW3XTr7OwwzM7Na9dr8StJoSYXr+bnA7LQ8DzhK0ojUofCotM3M6qCmJFrSSEnz09A587vr9Svpl5I2SPp5yfZJku5Kw/b8OP0HUbM2ia1RjzOZmZn1n0qaX5HdbX5E0qPAWOCidOw64KtkifhCYGbaZmZ1UOud6HOABRExGViQ1sv5OvDxMtsvAS6NiL2B9cAZNcYDQJsgwlm0mZk1v4iYGxH7RMSbI6KQIJ8fEXPS8g0RMTmV+WREbCw6dnZE7J1e3689llrPYJYftSbR04Cr0vJVwAnlCkXEAuA1UwhKEnAYUOjh3+3x1cruRLumm5mZNYJ7HZnVnkSPjYhVaflpssdIlRoFbEiPqmAHZ00rRxJbXzf9g5mZmZlZffQ6OoekW4Ddy+w6r3glIkJSw27/SjoTOBNgwoQJPZZtE74TbWZmZmYN02sSHRFHdLdP0jOS9oiIVWka4NVVvPdaYLikjnQ3usdZ0yJiFjALoKurq8cM2c05zMzM6i/wtdWsoNbmHHOAGWl5BnBTpQdG1vPvNuDEHTm+J+1tHp3DzMysUTztt1ntSfTFwJGSlgJHpHUkdUm6vFBI0m+AnwCHS1oh6ei060vA2ZKWkbWRvqLGeNL7uTmHmZlZo/gSa1bjjIURsRY4vMz2RcAni9Y/0M3xy4GDaomhnDbJFdzMzKzO5HE5zLbJ5YyF7lhoZmZWf24TbbZdTpNodyw0MzNrFLeJNstpEu1xos16JmmqpEckLZP0uplGJR0i6R5JmyWdWLJvhqSl6TWj9FgzM7NWkMsk2tN+m3VPUjtwGXAMMAU4RdKUkmJPAKcD15YcOxK4ADiYrD/DBZJGNDpmMzOzgSanSbSHuDPrwUHAsohYHhGbgOuAacUFIuLxiFgMlD7TORqYHxHrImI9MB+Y2hdBm1n/8/0ps+3ymUS3uWOhWQ/GAU8Wra9I2xp9rJnlhNwo2iyfSbR8J9qsX0k6U9IiSYvWrFnT3+GYmZnVXU3jRA9UbhNt1qOVwJ5F6+PTtkqPPbTk2NtLC0XELGAWQFdXV1WV8cWNm7lz+VpWPfsKm7dsZWv4yZI1jzcM7uDUgyf0dxhm1gdymkR7iDuzHiwEJkuaRJYUTwdOrfDYecA/F3UmPAo4t16BLXnqWWbM/j1/fmFTvU5p1qfG7tLpJNqsReQ4ie7vKMwGpojYLOkssoS4HZgdEUskzQQWRcQcSQcCNwIjgOMkfSUi9ouIdZK+SpaIA8yMiHX1iu3cnz1Am8Q1ZxzM3rsNpbOjjTYJteF50qwpNKKtsKSpwLfJ6uvlEXFxyf4JwFXA8FTmnIiYK2ki8DDwSCp6Z0R8qu4BmrWoXCbR8oyFZj2KiLnA3JJt5xctLyRrqlHu2NnA7HrHtPSZ51m84lkuPG4K7588ut6nN2tKRUNSHknWkXehpDkR8VBRsS8D10fEd9NwlXOBiWnfYxGxfx+GbNYyctmxsE3yMDxmTeaO5WsBOGLK2H6OxGxA6XVISiCAXdLyrsBTjQrGl1az7XKaRPtOtFmzeXjV8+y60yDGDd+pv0MxG0gqGVbyQuCvJa0guwv9maJ9kyTdK+lXkj5Q7g12ZDQdN68yy20S7Y6FZs3mkaef4y27D/P4s2bVOwW4MiLGA8cCP5DUBqwCJkTEAcDZwLWSdik9OCJmRURXRHSNGTOmTwM3a2a5TKI9TrRZ83lqwytMGPmG/g7DbKCpZEjKM4DrASLiDmAIMDoiNkbE2rT9buAxYJ+GR2zWInKZRHucaLPm8+cXNrLbsM7+DsNsoNk2JKWkwWRDUs4pKfMEcDiApH3Jkug1ksakjolIehMwGVheSzC+tJptl8vROTzEnVnz2bw1nESblahkSErg88D3JH2OrO/f6RERkg4BZkp6FdgKfKqeQ1KatbqcJtHuWGjWjHbbZUh/h2A24FQwJOVDwPvKHPdT4Kf1jMVdFsy2q6k5h6SRkuZLWpp+juim3C8lbZD085LtV0r6o6T70mv/WuIpOi8RbtJh1mxGD/WdaDMzaw61tok+B1gQEZOBBWm9nK8DH+9m3xcjYv/0uq/GeICsOQe47ZZZs3nD4Pb+DsHMeuDrqtl2tSbR08imGiX9PKFcoYhYADxf43tVrC09btri2m7WVHZyEm3WHNysw6zmJHpsRKxKy08DOzLV2EWSFku6VFK3z3KrGQy+LWXRbhdt1lyGDHISbWZmzaHXJFrSLZIeLPN6zbSjkTVArjZrPRd4K3AgMBL4UncFqxkM3s05zJrTTk6izQa08MTfZtv0mkRHxBER8bYyr5uAZyTtAZB+rq7mzSNiVWQ2At8HDtqRX6JUoTmH70SblSdpqqRHJC2T9Lq+DJI6Jf047b9L0sS0fZCkqyQ9IOlhSefWMy4n0WbNQW7PYVZzc445wIy0PAO4qZqDixJwkbWnfrDGeIDtd6K3eLBos9dJky9cBhwDTAFOkTSlpNgZwPqI2Bu4FLgkbT8J6IyItwPvBv6mkGDXQ2dHLud/MjOzHKr1inUxcKSkpcARaR1JXZIuLxSS9BvgJ8DhklZIOjrt+qGkB4AHgNHA12qMB4D2NifRZj04CFgWEcsjYhNwHVkn4WLFnYZvIKu7ImuytbOkDmAnYBPwXL0CK/RnMLOBzc06zGqcbCUi1pKmGi3Zvgj4ZNH6B7o5/rBa3r87g9qzC/GrW1zJzcoYBzxZtL4COLi7MmnGtGeBUWQJ9TRgFfAG4HOeAc3MzFpRLp+dtrdlv5bvRJvV3UHAFuCNwCTg85LeVFqomtF0zKz5uE20WU6T6I5td6K39nMkZgPSSmDPovXxaVvZMqnpxq7AWuBU4JcR8WpErAZ+B3SVvkE1o+mYmZk1o1wm0YXmHJt9J9qsnIXAZEmTJA0GppN1Ei5W3Gn4RODWNIzlE8BhAJJ2Bt4D/KFPojYzMxtAcplEb2/O4TvRZqUiYjNwFjAPeBi4PiKWSJop6fhU7ApglKRlwNlAYRi8y4ChkpaQJePfj4jFffsbmFl/8cixZtvV1LFwoBrU5o6FZj2JiLnA3JJt5xctv0I2nF3pcS+U225mrUVuEm2WzzvRHe3Zr7XZSbSZmZmZNUA+k+i2QptoN+cwM7PmVsEMoxMk3SbpXkmLJR1btO/cdNwjRXM01MzNOsxy2pyjwx0LzcwsB4pmGD2SbEz3hZLmRMRDRcW+TNa34btp9tG5wMS0PB3Yj2xYylsk7RMRW/r2tzDLp5zeic5+LQ9xZ2ZmTa6SGUYD2CUt7wo8lZanAddFxMaI+COwLJ2vZm4TbZbXJLrd036bNZvC0JRm9hrlZhgdV1LmQuCvJa0guwv9mSqONbMdlM8kutAm2h0LzZrG4PZc/ndk1hdOAa6MiPHAscAPJFVcoTzDqNmOyeVVa1C7m3OYNZvBHbn878isVpXMMHoGcD1ARNwBDAFGV3jsDs0w6udGZjlNotvb3JzDrNkM8p1os3IqmWH0CeBwAEn7kiXRa1K56ZI6JU0CJgO/77PIzXIul6NzFNpWvuok2qxpdA5yEm1WKiI2SyrMMNoOzC7MMAosiog5wOeB70n6HFknw9MjIoAlkq4HHgI2A5/2yBxm9ZPLJLrD036bNZ1/Ombf/g7BbECqYIbRh4D3dXPsRcBFdYylXqcya3q5vPXT7mm/zZrO3rsN7e8QzMzMKpbLJHqQp/02MzOrO3mAaLNtakqiJY2UNF/S0vRzRJky+0u6Q9KSNB3pyUX7Jkm6K01J+uPUaaJm28eJdnMOs3IqmEa4M9XJZamOTiza946iOv2ApCF9GryZmdkAUOud6HOABRExGViQ1ku9BJwWEfsBU4FvSRqe9l0CXBoRewPryYbpqVmHm3OYdatoGuFjgCnAKWl64GJnAOtT3byUrK4iqQO4BvhUqtOHAq/WJ656nMXMGsltos22qzWJngZclZavAk4oLRARj0bE0rT8FLAaGKPsmdBhwA09Hb8jOgrNOXwn2qycSqYRLq7bNwCHpzp7FLA4Iu4HiIi19ert3+Ys2qxpuLqa1Z5Ej42IVWn5aWBsT4UlHQQMBh4DRgEbImJz2t3jdKTVzKi0bcZCD3FnVk4lUwFvK5Pq6LNkdXYfICTNk3SPpH+sV1CFDsFmZmbNoNch7iTdAuxeZtd5xSsREZK6zVol7QH8AJgREVur7ZwQEbOAWQBdXV09Zsee9tusYTqA9wMHkjXVWiDp7ohYUFxI0pnAmQATJkyo6MS+E21mZs2k1yQ6Io7obp+kZyTtERGrUpK8uptyuwA3A+dFxJ1p81pguKSOdKer7HSkO6J9WxLt5hxmZVQyFXChzIrUDnpXsjq7Avh1RPwZQNJc4F1kfSK2qeZLb0Gb70SbDXhuEm22Xa3NOeYAM9LyDOCm0gJpxI0bgasjotD+mTSb0m3AiT0dvyMkMahdbs5hVl4l0wgX1+0TgVtTnZ0HvF3SG1Jy/UGy2dBq9oZB7fU4jZn1AeEvvWa1JtEXA0dKWgockdaR1CXp8lTmo8AhwOmS7kuv/dO+LwFnS1pG1t7yihrj2aa9zUm0WTnpyU9hGuGHgesL0whLOj4VuwIYlerm2aSRdyJiPfBNskT8PuCeiLi5lniO3i/rSjFi57qMcGlmfSDw9dWspmm/I2ItcHiZ7YuAT6bla8iGxCp3/HKykQLqblBbG6+6OYdZWRVMI/wKcFI3x3Zbp3fEzoM7GD9ip3qdzszMrE/kcsZCgEEdTqLNmoX7FJo1FzfnMMtxEj24vY1Nm51Em5mZmVn95TeJ7nASbWZmZmaNkdskurOjjY1Oos3MzOrG3QnNtsttEu070WZmZo3hfgxmOU6iOzva2OSOhWZm1uQkTZX0iKRlks4ps//SoiFkH5W0oWjflqJ9pePBm1kNahribiAb3NHGxledRJuZWfOS1A5cBhxJNmPoQklzImLbJEcR8bmi8p8BDig6xcsRsX8fhWvWUnJ8J7qdjb4TbTbguY2lWY8OApZFxPKI2ARcB0zrofwpwI/6JDKzFpfbJDq7E72lv8Mwswp4zFmzbo0DnixaX5G2vY6kvYBJwK1Fm4dIWiTpTkkndHPcmanMojVr1tQpbLP8y20S7TbRZmbWYqYDN0RE8R2kvSKiCzgV+JakN5ceFBGzIqIrIrrGjBnTV7GaNb3cJtEencPMzHJgJbBn0fr4tK2c6ZQ05YiIlenncuB2Xtteumrh9ldm2+Q2ie7saPc40WZm1uwWApMlTZI0mCxRft0oG5LeCowA7ijaNkJSZ1oeDbwPeKj02B3hBlhmOR6do9N3os3MrMlFxGZJZwHzgHZgdkQskTQTWBQRhYR6OnBdxGvuFe8L/KekrWQ3zS4uHtXDzGqT2yR6cEcbGze7Y6FZOZKmAt8muyhfHhEXl+zvBK4G3g2sBU6OiMeL9k8gu6N1YUR8o6/iNmtFETEXmFuy7fyS9QvLHPc/wNsbGpxZC8txcw7fiTYrp2jc2WOAKcApkqaUFDsDWB8RewOXApeU7P8m8ItGx2pmA0t4UEqzbXKbRA9ub2NrwKseocOsVCXjzk4DrkrLNwCHS9lEv2mYrD8CS/omXDMzs4Ent0n0ToPbAXjZY0Wblapk3NltZSJiM/AsMErSUOBLwFd6eoNqxp0Nd/c3axoe091su5qSaEkjJc2XtDT9HFGmzP6S7pC0RNJiSScX7btS0h8l3Zde+9cST7FhQ7Lm3s+/srlepzQzuBC4NCJe6KlQtePOytdlMzNrMrXeiT4HWBARk4EFab3US8BpEbEfMJVssPfhRfu/GBH7p9d9NcazzdDOQQC84CTarFQl485uKyOpA9iVrIPhwcC/Snoc+Afgn9LIAWbWAtwm2my7WkfnmAYcmpavIhvI/UvFBSLi0aLlpyStBsYAG2p87x4V7kS/sPHVRr6NWTPaNu4sWbI8nWw2s2JzgBlkY86eCNyahs76QKGApAuBFyLi3/oiaDMbQPz4yKzmO9FjI2JVWn4aGNtTYUkHAYOBx4o2X5SaeVxaGBS+m2MrbmMJMNTNOczKSm2cC+POPgxcXxh3VtLxqdgVZG2glwFnU/4pk5mZWcvq9U60pFuA3cvsOq94JSJCUrfPeSTtAfwAmBERhSEzziVLvgcDs8juYs8sd3xEzEpl6Orq6vV50rDOwp1oJ9FmpXobdzYiXgFO6uUcFzYkODMzsybQaxIdEUd0t0/SM5L2iIhVKUle3U25XYCbgfMi4s6icxfuYm+U9H3gC1VF34PCnWi3iTYzM6sPD6Zjtl2tzTkK7SZJP28qLSBpMHAjcHVE3FCyb4/0U8AJwIM1xrPN0E435zAzM2sEt4g2qz2Jvhg4UtJS4Ii0jqQuSZenMh8FDgFOLzOU3Q8lPQA8AIwGvlZjPNvsPLiD9jax/qVN9TqlmZmZgcfoMKPG0TkiYi1weJnti4BPpuVrgGu6Of6wWt6/J21tYrdhnTzz3MZGvYWZ1YEvxmZm1oxyO2MhwNhdhvDMc6/0dxhm1gs/GjZrLq6zZjlPonffZQhPO4k2MzMzszrLdxK96xBWbXiZcHdiMzMzM6ujXCfRb9l9GC9u2sIT617q71DMzMx2iKSpkh6RtEzS6yY+SpOVFTruPyppQ9G+GZKWpteM0mOr5VtSZtvVOu33gPb2cbsCcN+TG9hr1M79HI2ZmVl1JLUDlwFHAiuAhZLmRMRDhTIR8bmi8p8BDkjLI4ELgC6y/PfudOz62uOq9QxmzS/Xd6LfuvswRg/tZM59T/V3KGZmZjviIGBZRCyPiE3AdcC0HsqfAvwoLR8NzI+IdSlxng9MbWi0Zi0k13eiO9rb+Ph79uLSWx7l7Ovv4x3jduUNnR3behUrfZXevt4vYZr16M1jhvLOPYf3dxhm1j/GAU8Wra8ADi5XUNJewCTg1h6OHdeAGM1aUq6TaIC/+9CbWffiRn5y9wp+ds/K/g7HrGqfeN9EJ9FmVonpwA0RsaWagySdCZwJMGHChB7LvvdNo/jOgqUcPGnUDgdplhe5T6IHtbfxlWlv4/zj9uPZl1/lxY2vnQa8MHBHuLuEDVDDhgyq+zklTQW+DbQDl0fExSX7O4GrgXcDa4GTI+JxSUeSzUw6GNgEfDEibqUGX/7wFDZt2VrLKczybCWwZ9H6+LStnOnAp0uOPbTk2NtLD4qIWcAsgK6urh4vhu998yge+dpUOjvae4vbLPdyn0QXtLeJkTsPZuTOg/s7FLN+VUlHJeAMYH1E7C1pOnAJcDLwZ+C4iHhK0tuAedT4eHjMsM5aDjfLu4XAZEmTyJLi6cCppYUkvRUYAdxRtHke8M+SRqT1o4Bzaw3ICbRZJtcdC82srEo6Kk0DrkrLNwCHS1JE3BsRhZ66S4Cd0l1rM2uAiNgMnEWWED8MXB8RSyTNlHR8UdHpwHVRNDFCRKwDvkqWiC8EZqZtZlYHLXMn2sy2qaSj0rYyEbFZ0rPAKLI70QV/BdwTERtL36CaNpZm1rOImAvMLdl2fsn6hd0cOxuY3bDgzFqY70SbWdUk7UfWxONvyu2PiFkR0RURXWPGjOnb4MzMzPqAk2iz1lNJR6VtZSR1ALuSdTBE0njgRuC0iHis4dGamZkNQE6izVrPto5KkgaTtaWcU1JmDlCYIvhE4NaICEnDgZuBcyLid30VsJmZ2UCjoj4ITUPSGuBPvRQbzWvbb7YifwaZVvgc9oqIittNSDoW+BbZEHezI+IiSTOBRRExR9IQ4Adk0wevA6ZHxHJJXybr3b+06HRHRcTqHt7L9bVy/hxa4zOoqr72JdfXqvhzaI3PoNv62pRJdCUkLYqIrv6Ooz/5M8j4cxj4/G+U8efgz6AZ+N8o48/Bn4Gbc5iZmZmZVclJtJmZmZlZlfKcRM/q7wAGAH8GGX8OA5//jTL+HPwZNAP/G2X8ObT4Z5DbNtFmZmZmZo2S5zvRZmZmZmYNkbskWtJUSY9IWibpnP6Op94kzZa0WtKDRdtGSpovaWn6OSJtl6TvpM9isaR3FR0zI5VfKmlGufcaqCTtKek2SQ9JWiLp79P2lvoc8iLPddb11fU1b1xf8/936jpbhYjIzYtszNvHgDcBg4H7gSn9HVedf8dDgHcBDxZt+1eyyS8AzgEuScvHAr8ABLwHuCttHwksTz9HpOUR/f27VfEZ7AG8Ky0PAx4FprTa55CHV97rrOur62ueXq6vrfF36jpb+Stvd6IPApZFxPKI2ARcB0zr55jqKiJ+TTb5RbFpwFVp+SrghKLtV0fmTmC4pD2Ao4H5EbEuItYD84GpDQ++TiJiVUTck5afBx4GxtFin0NO5LrOur66vuaM62sL/J26zlYub0n0OODJovUVaVvejY2IVWn5aWBsWu7u88jN5yRpItmsenfRwp9DE2vFf4OW/Tt1fW16rfhv0NJ/p66zPctbEt3yInuG0hJDrkgaCvwU+IeIeK54Xyt9Dta8Wunv1PXVml2r/Z26zvYub0n0SmDPovXxaVvePZMenZB+rk7bu/s8mv5zkjSIrHL/MCJ+lja33OeQA634b9Byf6eur7nRiv8GLfl36jpbmbwl0QuByZImSRoMTAfm9HNMfWEOUOj1OgO4qWj7aann7HuAZ9OjmHnAUZJGpN61R6VtTUGSgCuAhyPim0W7WupzyIlWrLMt9Xfq+porrq8t8HfqOluF/u7ZWO8XWS/RR8l6EJ/X3/E04Pf7EbAKeJWsfdEZwChgAbAUuAUYmcoKuCx9Fg8AXUXn+V/AsvT6RH//XlV+Bu8ne4y0GLgvvY5ttc8hL68811nXV9fXvL1cX/P/d+o6W/nLMxaamZmZmVUpb805zMzMzMwazkm0mZmZmVmVnESbmZmZmVXJSbSZmZmZWZWcRJuZmZmZVclJtJmZmZlZlZxEm5mZmZlVyUm0mZmZmVmV/j9dHSJrN5xr1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23443277-57ef-4bcd-9a88-5b6f2ba35965",
   "metadata": {},
   "source": [
    "Here we see a similar pattern, with the fraction of near zero activations reaching almost 1 in fewer iterations. Let's see if we can imprve the situation by increasing the batch size."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7a40e92-d146-45fc-a639-3977ec5408dc",
   "metadata": {},
   "source": [
    "### Increasing the batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f5a9d8d-a53e-4a56-b59d-b8c8f6ff98d4",
   "metadata": {},
   "source": [
    "If you're GPU can handle it, increasing the batch size is one technique that can sometimes stabilise training. This is because a larger batch size implies more accurate gradients, so SGD can potentially train more efficiently. Let's try increasing our batch size by a factor of 4, training the model and investigating the activation statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3a18217e-2008-45d2-a621-595be005c9e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lewis/miniconda3/envs/dl4phys/lib/python3.9/site-packages/fastai/callback/core.py:67: UserWarning: You are shadowing an attribute (modules) that exists in the learner. Use `self.learn.modules` to avoid this\n",
      "  warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.693562</td>\n",
       "      <td>0.693991</td>\n",
       "      <td>0.499657</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAADSCAYAAAC8To2hAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsH0lEQVR4nO3de5xcdX3/8dd7Zm+5X0gIIXcgXFIvXFbEgoICGlCJv6qYoBX9oWltqbYiLdaWUvqzxdqK2gda8/CGFkHEVlJNpRhQqtWQIIIm4RIikISQhJAAue7t8/vjnN3MTnaT3czszsyZ9/PxmMecy3fO93N288189nu+53sUEZiZmZmZ2cFylQ7AzMzMzKxaOVk2MzMzM+uHk2UzMzMzs344WTYzMzMz64eTZTMzMzOzfjhZNjMzMzPrh5NlM7M6Iek6Sf9W6TjMzGqJk2UzswxwImw2MJI+K2mHpJ9Lml6w/TJJn69kbFadnCybmZlZXZB0JnAGcAzwU+CadPs44GrgryoX3QGSGobw2PmhOnZWOVmuA5KelHS1pIcl7Zb0FUlTJP2XpJck/UjShLTsWZL+V9JOSQ9JOq/gOO+XtDb9zHpJf1Cw7zxJGyVdJWmrpM2S3j/8Z2uWfZL+QtKmtC0+KunNwF8C75K0S9JDabk5kn6SlrsbmFTRwM0qbw7w04jYDywHjku3fxL4dES8eKgPS/q6pJsk/SBtVyskHV+w/2RJd0t6Pm2blxbse7OkByW9KGmDpOsK9s2WFJKukPQ0cE8fdf9n2r67X12S3jeAer8u6YuSlknaDbxe0imSfpx+16+WdMkR/CzrhpPl+vF24ELgROCtwH+RfLlOJvl38GFJ04AfAP8PmAh8DPiupMnpMbYCbwHGAu8HbpR0ekEdxwDjgGnAFcBN3Um4mZWHpJOAK4FXRcQY4E3AI8DfA9+OiNER8cq0+LeAB0iS5L8DLq9AyGbVZDXwWkkjgPOB1ZJagZMi4lsDPMZC4G+BCcA6kkQbSaOAu0na3dFpuS9Impd+bjfwXmA88GbgQ5LeVnTsc4FTSNp1LxHx1rR9jwbeCTwLLB9AvQCXpXGOAVYA/wn8d1r+T4Bb0v9brA9OluvHv0TElojYBPwPsCIiHoyIfcB/AKcB7wGWRcSyiOiKiLuBVcDFABHxg4h4IhI/IWlory2oox24PiLaI2IZsAtw4zMrr06gGZgnqTEinoyIJ4oLSZoJvAr464jYHxH3kXxBmtWtiPgN8F3gF8BM4B+Bz5N0GH1Y0n2SbpE0/hCH+Y+IuD8iOoBbgFPT7W8BnoyIr0VER0Q8mNb1zrTuH0fEr9Pv14eBW0mS40LXRcTuiNjbX+WSTgRuBi6NiA2Hqzd1Z0T8LCK60nhHAzdERFtE3AN8H1h0iHOua06W68eWguW9fayPBmYB70wvy+yUtBM4B5gKIOkiSb9IL/PsJEmiCy/rbk//8+i2Jz2umZVJRKwD/hS4Dtgq6TZJx/ZR9FhgR0TsLtj21NBHaFbdIuLGiHhlRLwLuBS4jyQfWkzS27yWdCxzP54tWC78npsFvLroO/TdJFddkfRqSfdK2ibpBeAPOXho1IZDxZ6Orb4T+KuI+OlA6u3juMcCG9LEudtTJFeFrQ9Olq3QBuCbETG+4DUqIm6Q1Ezyl+o/AVMiYjywDFAF4zWrSxHxrYg4h+RLMoBPpe+FNgMT0ku03WYOU4hmVU/SFJIE+XrgZcDDEdEOrARecQSH3AD8pOg7dHREfCjd/y1gKTAjIsYB/8rB36HF7bgw3lx6jHsjYskg6i0+7jPAjPR43WYCmwZzsvXEybIV+jfgrZLeJCkvqSW9cW860ERy6Xcb0CHpIuCNlQzWrB5JOknSG9I/YPeRXBnqIrlaNLv7CzAiniIZRvW3kpoknUNyv4KZJT5DMuxhD/Bb4FWSRgPnAeuP4HjfB06U9PuSGtPXqySdku4fAzwfEfuUzMpx2SCP/0lgFPCRQdZbbAVJj/ifp2XPI/m/4bZBxlM3nCxbj3Ts0wKSG/+2kfy1ejWQi4iXgA8DtwM7SBr50gqFalbPmoEbgOdILgcfDXwc+E66f7ukX6bLlwGvBp4H/gb4xvCGaladJL0BGB8R/wEQEfeT3OC+AXg9SRsblPR78o0kN9g9Q9I+P0XSZgH+CLhe0kvAtSTfp4OxCDgL2FEwI8a7B1BvcZxtJMnxRST/j3wBeG9EPDLIeOqGIvrt8TczMzMzq2vuWTYzMzMz64eTZTMzMzOzfjhZNjMzMzPrh5NlMzMzM7N+OFk2MzMzM+tHQ6UD6M+kSZNi9uzZlQ7DrGo88MADz0XE5ErH0R+3WbPeqrnNur2a9Xao9lq1yfLs2bNZtWpVpcMwqxqSqvpRxW6zZr1Vc5t1ezXr7VDt1cMwzMzMzMz6UZZkWdJ8SY9KWifpmj72v07SLyV1SHpHOeo0MzMzMxtqJSfLkvLATSSPTZwHLJI0r6jY08D7gG+VWp+ZmVm9kPRVSVsl/aaf/ZL0+bSz6mFJpw93jGZZV46e5TOBdRGxPn3e+G3AgsICEfFkRDwMdJWhPjMzs3rxdWD+IfZfBMxNX4uBLw5DTGZ1pRw3+E0DNhSsbwReXYbjHlJE8OX/+S0LTj2Wo8e2DHV1Zlainzy2jZzgtXOrcnIAs6oUEfdJmn2IIguAb0REAL+QNF7S1IjYXM44Ht/yEl/56W/p7IqebQ150dGZrDc25PjI+XOZcpjv46/89Lc8svnFcoZmdliTxzTz5/NPPuLPV9VsGJIWk/xlzMyZMw9Z9oltu/jksrXctfpZ7vjQ7w5HeGZWgsu/ej8AT97w5gpHYpYpfXVYTQMOSpYH8x1b7NsrN/DtVRuYmibD23e3sb+ji5FNeca0NLDlxf18a8XTjGjM93uMjq4u2tPk+thx7uSy4TNj4siSPl+OZHkTMKNgfXq6bdAiYgmwBKC1tTUOVba7wb20r+NIqjIzM6srg/mOLfbU83uYe/Ro/vvPzgXgH5at5Uv3redjbzyJ/3vOHO781SZWP3P4HuPmhhwffN1xjG1pPIIzMKuMciTLK4G5kuaQJMkLgcvKcNxDirSZS0Ndk5mZWdUqW4dVoQeeep69bV2cM3cSO3a3cfeaLfzeadN69n/0jSfyttOmMffo0QAsOHUaC06d1t/hzGpayTf4RUQHcCVwF7AWuD0iVku6XtIlAJJeJWkj8E7gS5JWl1qvmZmZsRR4bzorxlnAC+UYr/zFH6/n75etBeDSL/0cgLOOO6pnf3NDnlOmjqUh78c1WPaVZcxyRCwDlhVtu7ZgeSXJX7tlEwzqCpKZmVnNkXQrcB4wKe10+hugESAi/pXku/diYB2wB3h/ueoO4NxP38tT2/cAcP4pR5fr0GY1papu8BuMA8MwPA7DzMyyKSIWHWZ/AH9c7nq7v1q7E2WAo0Y3l7sas5pQ89dPnCqbmZmVX8SBK7iLX3dcBSMxq6yaT5bNzMysvIo7oj507vEVicOsGtR8suxRGGZmZuXX3nngobtjR3iqN6tfNZsse+o4MzOzobOnrbNnOZ/zl63Vr9pNltPZMORRy2ZmZmUlwa79yUO/po0fUeFozCqrZpNlMzMzGzrdT8j9m7fOq3AkZpVVs8lyeJplMzOzIVF41XbK2JYKRmJWebWbLKfvHrNsZmY2dCaOaqp0CGYVVbvJcnSPWTYzM7NyKuyIyvnmPqtzNZssd7lr2czMbMj5W9bqXc0my+FBy2ZmZkOisB/KfVJW72o2We7s8jAMMzOzoeYpWq3e1Wyy3OWHkpiZmQ0JJ8hmB9RssuxhGGZHTtJ8SY9KWifpmj72z5R0r6QHJT0s6eJKxGlmledOKat3NZss9/QsVzYMs5ojKQ/cBFwEzAMWSSp+6sBfAbdHxGnAQuALwxulmVWU+lw0q0s1myx3umfZ7EidCayLiPUR0QbcBiwoKhPA2HR5HPDMMMZnZtXE2bLVuZpNlru651n29SGzwZoGbChY35huK3Qd8B5JG4FlwJ/0dSBJiyWtkrRq27ZtQxGrmVWAei37e9bqW80my34oidmQWgR8PSKmAxcD35R00P8XEbEkIlojonXy5MnDHqSZDT33SVm9q9lkuaur0hGY1axNwIyC9enptkJXALcDRMTPgRZg0rBEZ2ZmVkVqNlnu7BmGUeFAzGrPSmCupDmSmkhu4FtaVOZp4HwASaeQJMseZ2FWJwqHOPpr1updzSbLB4ZhuBmbDUZEdABXAncBa0lmvVgt6XpJl6TFrgI+KOkh4FbgfeH5Gs3qku8NsnrXUI6DSJoPfA7IA1+OiBuK9jcD3wDOALYD74qIJ0ups8tf22ZHLCKWkdy4V7jt2oLlNcDZwx2XmVUH9bNsVo9K7lke4JytVwA7IuIE4EbgU6XW2z0bhluxmZnZ0HHHstW7cgzDGMicrQuAm9PlO4DzVeJ1nU53LZuZmQ0J9XooibNlq2/lSJYHMmdrT5l0vOQLwFHFBxrMnK1jWpIRJM0NNTvs2szM7LAG8Hj6WZKWp4+m/7Gk6ZWI0yyrqirTHMycrW84eQrHjG3h2HEjhik6MzOz4TXAoY7/BHwjIl4BXA/8Q8n19rtiVn/KkSwPZM7WnjKSGkgen7u91IqlgrHLZmZm2TOQoY7zgHvS5Xv72F8Sj1m2eleOZHkgc7YuBS5Pl98B3FOOaahyEk6VzcwswwYy1PEh4PfS5f8DjJF00FDHwfA8y2YHlJwsD3DO1q8AR0laB3wUOGjM1ZFwz7KZmRkfA86V9CBwLsnV3M7iQoO5L6joc2UL1KwWlWWe5QHM2boPeGc56iokgXNlMzPLsMMOdYyIZ0h7liWNBt4eETuLDxQRS4AlAK2trYf89vQ8y2YHVNUNfoOVk/BDxczMLMMOO9RR0iRJ3d/nHwe+Ws4A3LFs9a6mk2XhJ/mZmVl2DXCo43nAo5IeA6YAnyy5YifIZj3KMgyjUnyDn5mZZd0AhjreQfLAryHhh5JYvavtnmXf4GdmZjakPAzD6l2NJ8ses2xmZlZu7k02O6Cmk+WGnHhq+55Kh2FmZpZZ7lm2elfTyfLJx4xxsmxmZlZmhVdt3cts9a6mk+XJY5rp9HQYZmZmZbXqqR2VDsGsatR0spzLiU6PWTYzMyurcSMae5Y9DMPqXU0ny3mJLvcsm5mZlVWuIEF2rmz1rraTZfcsm5mZDSm5a9nqXE0ny8njrvH0cWZmZkPEqbLVu5pOlvPpdSLf5GdmZjY03LFs9S4bybJ7ls3MzIaEh2FYvavpZDmXNuCurgoHYmZmliHugjI7oKaT5XwavXuWzQZH0nxJj0paJ+mafspcKmmNpNWSvjXcMZqZmVWDhkoHUIrunmWPWTYbOEl54CbgQmAjsFLS0ohYU1BmLvBx4OyI2CHp6MpEa2ZmVlk13rPcPQzDybLZIJwJrIuI9RHRBtwGLCgq80HgpojYARARW4c5RjMzs6qQiWTZwzDMBmUasKFgfWO6rdCJwImSfibpF5LmD1t0ZmZmVSQTwzDcs2xWdg3AXOA8YDpwn6SXR8TOwkKSFgOLAWbOnDnMIZrZUHEflNkBNd2z3JW25i/8+IkKR2JWUzYBMwrWp6fbCm0ElkZEe0T8FniMJHnuJSKWRERrRLROnjx5yAI2MzOrlJKSZUkTJd0t6fH0fUI/5X4oaaek75dSX7E9bZ0AfP1/nyznYc2ybiUwV9IcSU3AQmBpUZnvkfQqI2kSybCM9cMYo5mZWVUotWf5GmB5RMwFlqfrffk08Psl1nUQz4JhNngR0QFcCdwFrAVuj4jVkq6XdEla7C5gu6Q1wL3A1RGxvTIRm5mZVU6pY5YXkPY+ATcDPwb+orhQRCyXdF7x9lK1d/ppJGZHIiKWAcuKtl1bsBzAR9OXmZlZ3Sq1Z3lKRGxOl58FppR4vEFp63CybGZmZmZD57A9y5J+BBzTx65PFK5EREgqaVzEYO+sd8+ymZlZ+YUfeG3W47DJckRc0N8+SVskTY2IzZKmAiU9uCAilgBLAFpbWw/bUts73ZjNzMzMbOiUOgxjKXB5unw5cGeJxxuUxryGszozMzMzqzOlJss3ABdKehy4IF1HUqukL3cXkvQ/wHeA8yVtlPSmEusF4E8vOBGAl08bV47DmZmZVR1J8yU9KmmdpINmnZI0U9K9kh6U9LCkiysRp1lWlTQbRjqV1Pl9bF8FfKBg/bWl1NOfUc0NvOl3pvDU9j1DcXgzM7OKkpQHbgIuJHlY0EpJSyNiTUGxvyKZAvKLkuaRzHQze9iDNcuomn6CH8CIxnzPw0nMzMwy5kxgXUSsj4g24DaSaVsLBTA2XR4HPDOM8ZllXqnzLFfciKYG9rY7WTYzs0yaBmwoWN8IvLqozHXAf0v6E2AUybBIMyuTTPQs73XPspmZ1a9FwNcjYjpwMfBNSQd9v0taLGmVpFXbtm075AHDk02Z9aj5ZHlkU5697Z2EW7aZmWXPJmBGwfr0dFuhK4DbASLi50ALMKn4QBGxJCJaI6J18uTJQxSuWfbUfLI8oilPZ1fQ5geUmJlZ9qwE5kqaI6kJWEgybWuhp0lvtpd0CkmyfOiuYzMbsNpPlhvzAOxrc7JsZmbZEhEdwJXAXcBaklkvVku6XtIlabGrgA9Kegi4FXhf+HKrWdlk4Aa/JFne297JOBorHI2ZmVl5RcQykungCrddW7C8Bji7vHWW82hmtS0zPct72joqHImZmZmZZU3tJ8sFPctmZmZmZuVU+8ly2rPs6ePMzMzMrNxqPlke6Z5lMzMzMxsiNZ8st/SMWXaybGZmZmblVfPJcnfP8j73LJuZmZlZmdV8stx9g597ls3MzMys3Go/WfYNfmZmZmXlaZbNDqj9ZNk3+JmZmZnZEKn5ZLkpnyMn9yybmZmZWfnVfLIsiZFNDR6zbGZmVibh512b9aj5ZBmS6eM8DMNs4CTNl/SopHWSrjlEubdLCkmtwxmfmZlZtchEsjyyKe+p48wGSFIeuAm4CJgHLJI0r49yY4CPACuGN0IzM7PqkYlkeURjnj1tHZUOw6xWnAmsi4j1EdEG3AYs6KPc3wGfAvYNZ3BmZmbVJBPJcktTnr3tXZUOw6xWTAM2FKxvTLf1kHQ6MCMifjCcgZmZmVWbkpJlSRMl3S3p8fR9Qh9lTpX0c0mrJT0s6V2l1NmXkY159rpn2awsJOWAzwBXDaDsYkmrJK3atm3b0AdnZmY2zErtWb4GWB4Rc4Hl6XqxPcB7I+J3gPnAZyWNL7HeXkY0+QY/s0HYBMwoWJ+ebus2BngZ8GNJTwJnAUv7uskvIpZERGtEtE6ePHkIQzYzM6uMUpPlBcDN6fLNwNuKC0TEYxHxeLr8DLAVKOu36oimvKeOMxu4lcBcSXMkNQELgaXdOyPihYiYFBGzI2I28AvgkohYVZlwzczMKqfUZHlKRGxOl58FphyqsKQzgSbgiX72H9El3RGNefY5WTYbkIjoAK4E7gLWArdHxGpJ10u6pLLRmZmZVZeGwxWQ9CPgmD52faJwJSJCUr+zmEuaCnwTuDwi+rwbLyKWAEsAWltbBzwj+simPHs8DMNswCJiGbCsaNu1/ZQ9bzhiMjMzq0aHTZYj4oL+9knaImlqRGxOk+Gt/ZQbC/wA+ERE/OKIo+3HiMa8H3dtZmZWZn/wuuMqHYJZxZU6DGMpcHm6fDlwZ3GBdEzkfwDfiIg7SqyvTy2NefZ3dNHZ5cdzmpmZlcvFL59a6RDMKq7UZPkG4EJJjwMXpOtIapX05bTMpcDrgPdJ+lX6OrXEensZ1ZwH8IwYZmZmZSRVOgKzyjvsMIxDiYjtwPl9bF8FfCBd/jfg30qp53BGNiWnsXt/B6ObSzolMzMzM7MemXiCX3eCvHu/H0xiZmZWLsJdy2aZSJZH9STLHoZhZmbZImm+pEclrZN00MO/JN1YMMzxMUk7S60zfAuQWY9MjFnoHrO8yz3LZmaWIZLywE3AhcBGYKWkpRGxprtMRPxZQfk/AU4rX/3lOpJZ7cpGz3KTh2GYmVkmnQmsi4j1EdEG3Eby9Nz+LAJuHZbIzOpENpLl7mEYbU6WzcwsU6YBGwrWN6bbDiJpFjAHuKef/Uf0lFyzepeJZHm0xyybmZktBO6IiD6/DCNiSUS0RkTr5MmTD3mgIBm07GEYZhlJlrvHLHsYhpmZZcwmYEbB+vR0W18WUuYhGJ4NwywjyXL3PMu+wc/MzDJmJTBX0pz0ibgLSZ6e24ukk4EJwM+HOT6zzMtEspzPiZFNeSfLZmaWKRHRAVwJ3AWsBW6PiNWSrpd0SUHRhcBtEeWd9M3DMMwyMnUcwFGjm3hu1/5Kh2FmZlZWEbEMWFa07dqi9euGom4ny2YZ6VkGmDKmhS0v7qt0GGZmZmaWIdlJlse1sOVF9yybmZmVi2/wM8tQsnzM2BaefWEfZR6uZWZmZmZ1LDPJ8pSxzext7+TFvb7Jz8zMrBTudzI7IDPJ8pxJowFYt21XhSMxMzMzs6zITLJ88jFjAHjk2RcrHImZmZmZZUVmkuXpE0YwurmBR599qdKhmJmZ1bTuURieOs4sQ8myJE46ZgyPbHaybFbNHnhqR6VDMDMzG7DMJMsA86aOZc3mF+nq8p0JZtXq7V/830qHYGZmNmCZSpZfOWM8u/Z38IRv8jMzMyuZR2GYZSxZPnXGOABWPunLvGaHImm+pEclrZN0TR/7PyppjaSHJS2XNKsScZqZmVVaScmypImS7pb0ePo+oY8ysyT9UtKvJK2W9Iel1Hkox08ezayjRrLs15uHqgqzmicpD9wEXATMAxZJmldU7EGgNSJeAdwB/OPwRmlmZlYdSu1ZvgZYHhFzgeXperHNwGsi4lTg1cA1ko4tsd4+SeLtp0/np+ue46ENO4eiCrMsOBNYFxHrI6INuA1YUFggIu6NiD3p6i+A6cMco5mZWVUoNVleANycLt8MvK24QES0RcT+dLW5DHUe0vvPns3RY5q54uaVbNq5dyirMqtV04ANBesb0239uQL4r752SFosaZWkVdu2bStjiGZmZtWh1MR1SkR0j3l4FpjSVyFJMyQ9TPIF/amIeKbEevs1pqWRG991Ks/tauPsG+7h9lUbDv8hM+uTpPcArcCn+9ofEUsiojUiWidPnjy8wZnZkAk/79qsx2GTZUk/kvSbPl7Fl22DA/OYU7RvQzr28QTgckn9JdVl6aU6+4RJ/N5pSUfZdx/YeMTHMcuoTcCMgvXp6bZeJF0AfAK4pODqkJmZWV1pOFyBiLigv32StkiaGhGbJU0Fth7mWM9I+g3wWpKbhor3LwGWALS2tpb0Z+0/X/pKRjTl+e4vN9Le2UVjPlMTf5iVYiUwV9IckiR5IXBZYQFJpwFfAuZHxCHbtZmZWZaVmkEuBS5Ply8H7iwuIGm6pBHp8gTgHODREus9LEmcOWci+9q7PO+yWYGI6ACuBO4C1gK3R8RqSddLuiQt9mlgNPCddCabpRUK18wqyI+7NhtAz/Jh3ADcLukK4CngUgBJrcAfRsQHgFOAf5YUJPOb/1NE/LrEegdk3tSxAKzd/CInHzN2OKo0qwkRsQxYVrTt2oLlfq8omZmZ1ZOSkuWI2A6c38f2VcAH0uW7gVeUUs+RmjNpFE0NOR7Z/BKcVokIzMzMzKyWZXogb0M+x4lTRrNm84uVDsXMzKxmeC4MswMynSwDnHzMWNZufqnSYZhZmW15cR9tHV2VDsMs4zxo2SzzyfIpU8fy3K79PLfLM1+Z1bqtL+5j+dotXHX7Q5z1D8t567/8lBf2tFc6LDMzy7BSb/CreidOGQ3AY1teYtLo5gpHY2aDFRHcev8Gltz3BE9uT57APbq5gfNOnMz/PP4cV976Sz638DQmjmqqcKRmZpZFdZAsjwFg3dZd/O7xkyocjZkNxq79Hfzlv/+apQ89Q+usCbznrFm8bNo4Tp0xnpbGPLfd/zSf+N5vOO/T9/L2M6bzhpOP5hXTxzNuRGOlQzcrG0nzgc8BeeDLEXFDH2UuBa4jGW78UERcVlzGzI5M5pPlo8c0M6algce2eNyyWS3Z197J739lBQ9t2MnVbzqJD517PLlc7/GTC8+cyRmzJnDjjx7jll88zdd+9iQAE0c1MXVcC1PGtjC2pYExLY2MHZG8j25uYERjnhFN6asxf2C9MU9LwXI+5/GaVlmS8sBNwIXARmClpKURsaagzFzg48DZEbFD0tElV+w7/Mx6ZD5ZlsSJU8bw2BY/mMSsVnR1BVd95yF+tWEnN112Ohe/fGq/ZedOGcMX3n0GL+1r56ENL/DrTS+wYcceNu/cy7Mv7OPxre28tK+Dl/Z10Nk1uAygqSHXK5luaczT0pijuSFHS2Oe5oYczQ353uuNybaWxgP7mhtztDTke/Z1f66nTMExm/K5g/4osLp2JrAuItYDSLoNWACsKSjzQeCmiNgB4KdumpVX5pNlSIZiLPv1ZiIC+XFEZhW3+pkX2PD8Xua/7BgAOruCG+9+jPXP7eKv3zKPW1c8zQ8e3szHLzr5kIlyoTEtjZwzdxLnzO17uFVEsKetk137O9jX3sne9k72tqWvgvUD+7rY256ut3Wyp72TvW0d7O/oYn97Fzt2t7GvvYv9HZ3Jto4u9qXlB5mTH6Qpn0uT7CSxbsyLpoYcjfnk1dSQoynfe3tTur2x8L2/zzWIpnyexrxobMjRnM/R2Os4ojGfoyGfoyGn5NW9nBeNOSf0w2gasKFgfSPw6qIyJwJI+hnJUI3rIuKHwxOeWfbVRbJ82ozx3Hr/06x/bjfHTx5d6XDM6t6bP/9TAB786wvJ5cSffftX3PPIVhrz4q7VW+jsCi5tnc7i1x1XtjolMaq5gVHNQ//fXkdnV68EOlnuZH974XqyrSfhbu9jW5qYt3cmr7aOLtrS5T1tHbR3xkHb2zq6aO8M2tLloSJBYy5HvjuBzifLjTmRTxPqhrzI55KkPkm6u7elyXj62YZcwXJPgp58Lp9LXjkVL0NOyXHzOZHLibwOvBdu6/5c92fyhWVzB9Z713OgbE5JbIXH777SUCUagLnAecB04D5JL4+InYWFJC0GFgPMnDlzQAd2/5JZnSTLp88aD8ADT+1wsmxDprMrDkp62gqSogPbe/dG7m/vpK2zi0tbZ9TdjC2XfXkFL+xpY+tL+/m7t72M154wiU/98BFmHjWSqy48qWavBDWkvbKjKvzrjAg6unon1N3JdK8ku1eyHT3bOrq66OgKOjojfS9cT47T2fNeuC05fvKebO8siKMjLdv3cQv3Je+dEUSVjaH949cfz9VvOnk4qtoEzChYn55uK7QRWBER7cBvJT1GkjyvLCwUEUuAJQCtra1V9hM1q151kSwfN2k0Y1sa+OVTO7i0dcbhP2A1rTtp3VfQi1f43mtbe9ehy6bJ7L7CHsLiY3QcSABK8bq5k+smWZ7/O8dw7kmTuWXFU8w6ahT/ctlpnDFrIgBffM8ZFY4uOyQlQy3yOUbW+Mx6EUkS3hlBVxd0dq+nr644eDl5h46url6f6dmfHu/g45DWU1hnksh3f/YV08cN16mvBOZKmkOSJC8Eime6+B6wCPiapEkkwzLWD1eAZllXF8lyLid+9/hJ3PvoVrq6wmPthllEsL+j6xBjQzt7jQ3d256MFd3fT4K7r733perCy9j7Ojpp7zzypDUn0pu48rQUjBntvnlr/IhGWsY297o5q/BGrqaG3mNNmxoKb+gqWG/M9bk/q6KoW/Cai05m9qRRLDpzYJeCzZQOhaiLL60CEdEh6UrgLpLxyF+NiNWSrgdWRcTSdN8bJa0BOoGrI2J7SfWWGrhZhtTN/ztv/J0p/HD1szy0cSenzZxQ6XCqRntnehNTUSJ7IHnt6tleXKbPZLdnX1evfUeiO4HsTl4PLOcY2dTAxFG9k9nu2QZaGnrPWFD42eKZCloac732N+azm7BWk9mTRlU6BLOaERHLgGVF264tWA7go+nLzMqsbpLl80+eQkMuuXmo1pLlzq5gT1tHckd+Wye70+Xdbcnd+bv3H7hTf/f+JDndvb+7TAd70s/t6S7fdiDxPZKhA035HC2NuT7npR3b0khL0dy1Ld3L6WdaGvuf17Z7u6fPypbCjuUPnXd85QIxMzMbpLpJlseNbOTsEybxvQc3cdUbTxyyHsT9HZ3s3p8kq7sKXrv3d7Bnfyd70mR1bx9J756iBLg76d0/yDvaWxpzjGpqYERTvud9ZFOeCSObGNmUZ1RzkqCO7CdZLU52C8u0NCQ3LpkdqT+9YG6lQzAzMxuwukmWAd77mllccfMqlv16MwtOnTbgz+3v6OTxLbtY/9xunnpuN8/t2s/23W08n766k+Hd+5NZDQYinxMjG/OMbM4zsil5otio5jzjRjRy7LiWngS3OOkd1ZxnRGNDT9I7orEheW86cBw/dcyqTXfH8qIzZ9LcUDXTbZmZmR1WXSXLrz/paE4+Zgyf/MFazj1xMuMPcXt4e2cX//WbZ7njgY3c/9vt7Gs/kASPbWngqNHNTBzVxPQJIxnb0tAzf+uYlgZGNeUPLKfbRzc39Ep+mxtyNTstltmRmjqupdIhmNkAdN+U628pszpLlnM58U/vfCVvu+lnXHvnaj638NQ+E9ZfbdjJx77zEOu27mL6hBEsfNVMWmdP4ISjRzP7qFHVNBG9WU0ong3DzMysVtRVsgzwsmnj+PD5c/nM3Y8xdVwLHz5/bq8net29ZgtXfuuXTBrdzL++5wzeOG+KbzQzKxO3JDMzqzV1lywDXPn6E9j8wl6+dN96blnxNAtOPZZ3v3oWk0Y38ZHbHuSkY8bw1fe9qm4eEGE21NyvbFabPFzQrE6T5VxO/MPvvYJ3nDGdW1Y8zR0PbOSWFU/TmE/+U/iXRac5UTYbAv7eNTOzWlPXc4CdMWsin7n0VO7/ywt45xnTae8Mrn7TScw6yg9MMDOz+vX206cDMHFUjT8n3awMSupZljQR+DYwG3gSuDQidvRTdiywBvheRFxZSr3lNm5kI59+5yv5y4tPYYL/Y7A6IGk+8DmSx+d+OSJuKNrfDHwDOAPYDrwrIp480voacuInV5/H+BFuX2a14Mo3nMDic4/zVI9mlN6zfA2wPCLmAsvT9f78HXBfifUNKSfKVg8k5YGbgIuAecAiSfOKil0B7IiIE4AbgU+VWCezjhrFuJGNpRzGzIaJJCfKZqlSk+UFwM3p8s3A2/oqJOkMYArw3yXWZ2alOxNYFxHrI6INuI2kLRcqbNt3AOfLd/qYmVkdKjVZnhIRm9PlZ0kS4l4k5YB/Bj52uINJWixplaRV27ZtKzE0M+vHNGBDwfrGdFufZSKiA3gBOGpYojMzM6sihx2zLOlHwDF97PpE4UpEhKS+Zoj6I2BZRGw8XMdURCwBlgC0trZ6timzKidpMbAYYObMmRWOxszMrPwOmyxHxAX97ZO0RdLUiNgsaSqwtY9irwFeK+mPgNFAk6RdEXGo8c1mNnQ2ATMK1qen2/oqs1FSAzCO5Ea/XvwHrpmZZV2pwzCWApeny5cDdxYXiIh3R8TMiJhNMhTjG06UzSpqJTBX0hxJTcBCkrZcqLBtvwO4J/zMajMzq0Mq5ftP0lHA7cBM4CmSqeOel9QK/GFEfKCo/PuA1oFMHSdpW3rMQ5kEPHcksVexrJ1T1s4HKndOsyJicjkOJOli4LMkU8d9NSI+Kel6YFVELJXUAnwTOA14HlgYEesPc8x6bLNZOx/I3jlV8nzK1mbLze01M7J2TlXZXktKlitN0qqIaK10HOWUtXPK2vlANs9puGTtZ5e184HsnVPWzmc4Ze1nl7XzgeydU7WeT10/wc/MzMzM7FCcLJuZmZmZ9aPWk+UllQ5gCGTtnLJ2PpDNcxouWfvZZe18IHvnlLXzGU5Z+9ll7Xwge+dUledT02OWzczMzMyGUq33LJuZmZmZDZmaTZYlzZf0qKR1kmpi3mZJMyTdK2mNpNWSPpJunyjpbkmPp+8T0u2S9Pn0HB+WdHplz6BvkvKSHpT0/XR9jqQVadzfTufyRVJzur4u3T+7ooH3Q9J4SXdIekTSWkmvqfXfUaW5vVaXLLVZt9eh4TZbPbLUXqE222xNJsuS8sBNwEXAPGCRpHmVjWpAOoCrImIecBbwx2nc1wDLI2IusDxdh+T85qavxcAXhz/kAfkIsLZg/VPAjRFxArADuCLdfgWwI91+Y1quGn0O+GFEnAy8kuTcav13VDFur1UpS23W7bXM3GarTpbaK9Rim42ImnuRPEL7roL1jwMfr3RcR3AedwIXAo8CU9NtU4FH0+UvAYsKyveUq5YXyaOSlwNvAL4PiGRC8Ybi3xVwF/CadLkhLadKn0PR+YwDflscVy3/jir9cnutfOxF55GZNuv2OmQ/V7fZKnllqb2mcdVkm63JnmVgGrChYH1juq1mpJdHTgNWAFMiYnO661lgSrpcC+f5WeDPga50/ShgZ0R0pOuFMfecT7r/hbR8NZkDbAO+ll72+rKkUdT276jSav5nlKH2Ctlqs26vQ6Pmf04ZarOfJTvtFWq0zdZqslzTJI0Gvgv8aUS8WLgvkj+damKKEklvAbZGxAOVjqWMGoDTgS9GxGnAbg5cDgJq63dkpctKe4VMtlm3VztIVtpsBtsr1GibrdVkeRMwo2B9erqt6klqJGnEt0TEv6ebt0iamu6fCmxNt1f7eZ4NXCLpSeA2kstEnwPGS2pIyxTG3HM+6f5xwPbhDHgANgIbI2JFun4HScOu1d9RNajZn1HG2itkr826vQ6Nmv05ZazNZq29Qo222VpNllcCc9M7QpuAhcDSCsd0WJIEfAVYGxGfKdi1FLg8Xb6cZJxV9/b3pneDngW8UHCZouIi4uMRMT0iZpP8Du6JiHcD9wLvSIsVn0/3eb4jLV9Vfz1GxLPABkknpZvOB9ZQo7+jKuH2WiWy1mbdXoeM22wVyFp7hRpus8M9SLpcL+Bi4DHgCeATlY5ngDGfQ3Jp4WHgV+nrYpIxRcuBx4EfARPT8iK5I/kJ4NdAa6XP4RDndh7w/XT5OOB+YB3wHaA53d6Srq9L9x9X6bj7OZdTgVXp7+l7wIQs/I4q/DN1e62yV1barNvrkP1c3War6JWV9prGWXNt1k/wMzMzMzPrR60OwzAzMzMzG3JOls3MzMzM+uFk2czMzMysH06WzczMzMz64WTZzMzMzKwfTpbNzMzMzPrhZNnMzMzMrB9Ols3MzMzM+vH/AfcdxHdW6EbGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls = get_dls(512)\n",
    "learn = fit()\n",
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc94691b-b8db-45b5-ae4a-77fb9a0e32bd",
   "metadata": {},
   "source": [
    "Okay, increasing the batch size hasn't helped much. Clearly the problem is that our learning rate is too large, which is causing training to diverge. One way to handle this is to use a _dynamic_ learning rate that is adjusted from low to high, and back to low values during training itself! Let's take a look at 1-cycle training and finally understand what's going one when we call `Learner.fit_one_cycle()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f6da557-50e8-473c-89fd-380cfed9c2ce",
   "metadata": {},
   "source": [
    "### 1-cycle training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88fcdcbf-3f96-4e27-b36e-03c0a31db576",
   "metadata": {},
   "source": [
    "The basic idea behind 1-cycle training is to split training into two phases:\n",
    "\n",
    "* **Warmup:** grow the learning rate from some minimum to maximum value. By starting with a small learning rate, we can avoid hopping out of a local minimum in the loss and diverging.\n",
    "* **Annealing:** decrease the learning rate from the maximum value to the minimum. By gradually lowering the learning rate, we avoid skipping a good local minimum towards the end of training.\n",
    "\n",
    "This technique is one of the main aspects that allows fastai to train fast and accurate models. As we've done in previous lectures, we can use the `fit_one_cycle()` method, so let's adjust our `fit()` function and train again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cc0358c8-1d38-40df-8bb8-364d3ef80228",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lewis/miniconda3/envs/dl4phys/lib/python3.9/site-packages/fastai/callback/core.py:67: UserWarning: You are shadowing an attribute (modules) that exists in the learner. Use `self.learn.modules` to avoid this\n",
      "  warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.255070</td>\n",
       "      <td>0.257523</td>\n",
       "      <td>0.892029</td>\n",
       "      <td>0.957344</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def fit(epochs=1, lr=0.06):\n",
    "    learn = Learner(\n",
    "        dls,\n",
    "        simple_cnn(),\n",
    "        metrics=[accuracy, RocAucBinary()],\n",
    "        loss_func=F.cross_entropy,\n",
    "        cbs=ActivationStats(with_hist=True),\n",
    "    )\n",
    "    learn.fit_one_cycle(epochs, lr)\n",
    "    return learn\n",
    "\n",
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f112f556-0480-48b2-8f61-bae857b0e6a2",
   "metadata": {},
   "source": [
    "Nice, we've finally got a model that performs better than random! All fastai `Learner`s come with a `Recorder()` callback, which tracks various aspect of training. Here we can use it to inspect the evolution of the learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5b2f1967-9cc4-417c-9d28-3644735fa181",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtkAAAD4CAYAAAA5IvokAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABSOElEQVR4nO3dd3hUZfr/8fedXkhvQDoQegkQggiIXWxgQ0Fdy+q6ruu6tt3Vn2tZy+7qqmsvrNgbYkVFWQuK0kMJvQQIkFDSK6Q/vz9y8JuNKEFmcs5M7td1zcWZM2eYTzJwcufM/TyPGGNQSimllFJKuY6P3QGUUkoppZTyNlpkK6WUUkop5WJaZCullFJKKeViWmQrpZRSSinlYlpkK6WUUkop5WJ+dgdwldjYWJOWlmZ3DKWU+kWWL19eYoyJsztHZ9LztlLKU3XknO01RXZaWho5OTl2x1BKqV9ERHbYnaGz6XlbKeWpOnLO1nYRpZRSSimlXEyLbKWUUkoppVxMi2yllFJKKaVcTItspZRSSimlXEyLbKWUUkoppVzMrUW2iEwUkU0ikicitx3i8UARmWk9vkRE0to8NlREFonIOhFZIyJB7syqlFJKKaWUq7ityBYRX+Bp4HRgIDBNRAa2O+wqoNwY0wf4N/Cg9Vw/4HXgWmPMIOB4oNFdWZVSSimllHIld86TnQ3kGWO2AYjI28BkYH2bYyYD91jb7wJPiYgApwKrjTG5AMaYUjfmVJYt+6pZml9GSXUDPgJB/r5EhvgTFRJATLcA0mJCiQoNsDumUqqL2rS3mk/X7CElOoTj+sYSH6YfcCqlDs8YQ1ltA7vKD7Cn4gDVdU1U1zdRU9dERLAfV4xNd8vrurPITgR2tblfAIz+qWOMMU0iUgnEAH0BIyJzgTjgbWPMQ+1fQESuAa4BSElJcfkX0FVs2lvNPbPXsWjb4X+XiQj2p1dcKEMTIxieEkVmciSpMSG0/m6klFLus35PJU9+vQVjwNdHOHd4Iref3p+YboF2R1NKOYQxhq3FNazYUcHqwgrWFFaxZV81+xuaD3l8n/huHllkHw0/YBwwCtgPfCUiy40xX7U9yBgzHZgOkJWVZTo9pReYlbOLOz5YS7cgP/7fGf2ZOKgHPSODMMCBxmYqahupONBAUVU9+aW1bC+pJa+ohlnLC3hlUetiR4mRwRzfL44T+sUztk8swQG+9n5RSimvdO7wJM4c0pNtJTXMying1UX5fLelmBmXj2JwYoTd8ZRSNtnf0MS8jcV8s6mI77aUsLeqDoBugX4M6hnOhVnJpESHkBwdQs/IICKC/QkL9Cc00Bc/X/cNT3RnkV0IJLe5n2TtO9QxBVYfdgRQSutV7/nGmBIAEZkDjAC+QrnMi99v595P1jM+I5bHLsr80dUgf18fwoP8SSHkR89tbjFs3lfN8h3lzN9czAcrC3ljyU5CA3w5bXB3zhuexJjeMfj66BVupZTrBPj50L97OHeeNZDzRyRx9SvLuOj5Rbx9zRiGJGmhrVRX0djcwryNRXy8eg9frt/HgcZmwoP8GJcRy/iMOLLTo0mPCcXHxjrEnUX2MiBDRNJpLaanAhe3O2Y2cDmwCLgA+NoYc7BN5M8iEgI0ABNoHRipXOTj3N3c+8l6Jg7qzhPThhPgd2S/yfn6CAN6hDOgRziXHpNKfVMzy7aX88nq3Xy6Zg/vryikR0QQlx6TyrTsFKK1l1sp5WIDe4bz/nVjueC5hVzx0lI+/P1YkqN/fFFAKeU99lXV8dbSnby1dCf7quqJCvHnvBGJnD2sJ6PSoh11cU+McV+XhYicATwG+AIvGmMeEJF7gRxjzGxrWr7XgOFAGTC1zUDJS4HbAQPMMcb8+edeKysry+Tk5Ljta/EmW4trOOuJ7xmcGM5rV40myN+17R11jc18taGIt5bu5Pu8EgL9fDgnM5Frj+9NemyoS19LKW9htcRl2Z2jM7nqvL2tuIbJTy2gV3w3Zv12zBFfNFBKOV9eUTVPz9vK7NzdtBjDhL5xXDo6lQn94vB3Y8vHT+nIOdutRXZn0iK7YxqaWjjv2QUUlh/g8xuPIyHcvaPzN++r5qUF+by/ooCmFsM5mYn84cQ+pGmxrdT/0CL76MxZs4fr3ljBDSf24eZT+7nk71RK2W/Dniqe/HoLn63dS7C/L9OyU7hsTCqpMfbWER05Zzt14KNyk2e/2crawiqe/9VItxfYAH0TwvjHeUO46ZQMnv92G68v3sGHqwqZOiqZm0/pq7MCKKVc4owhPTgnsyfPfruVSZk96RMfZnckpdRR2FdVx8NzN/HuigK6Bfjx++P78Otx6R7VfqpFdheyq2w/z3yTx5lDe3DaoO6d+trxYUHcedZAfjuhF8/M28pri3cwO3c3N57cl8vGpNryUY9SyrvcceZAvt5YxB0frOXta47RqUWV8kAHGpp57tutTJ+/jaaWFq4el871J2QQEeJvd7QjppVNF3L/p+vxEeGOMwbYliE+LIh7Jg3i8z+OZ3hKFPd9sp7THpvPkg7M0a2UUj8nLiyQP0/sz5LtZXy5ocjuOEqpIzR/czGnPvYtj3+1hRP7x/PVzcdzx5kDPbLABi2yu4zlO8qZu24fvz+hNz0jg+2OQ0ZCGK9cOYoXr8iisbmFi6Yv5q8frqG6rtHuaEopDzZ1VDLpsaE8PHcTLS3eMeZIKW9XWlPPTTNXcdmLS/H38eHta47h6UtGkBLj2bMFaZHdRTz25WZiQgP49Tj3rGr0S4gIJ/ZPYO6Nx3HVuHTeWLKT0/49n2826RUopdQv4+frw02n9GXTvmpm5+62O45S6jA+X7uXkx/9lk9W7+aGE/sw54/jOaZXjN2xXEKL7C5gWX4Z320p4doJvQkJcF4bfkiAH3eeNZD3fncsoYF+XPHSMu79eD31TYdeAlUppX7OWUN60L97GE/Ny9Or2Uo5VG19E395dzXXvr6cxKhgPr1hPDef2s/l0wrbSYvsLuDJr/OI7RbIpcek2h3lZ41IieLjP4zjimPTeHHBds55eiF5RdV2x1JKeRgfH+HaCb3JK6rh283FdsdRSrWTu6uCM5/4jneW7+J3x/fm/d+NpW+C980IpEW2l9uyr5r5m4u54thUggOc/9thkL8v90waxIzLs9hXVcdZT37Pe8sL7I6llPIwZw7tQY+IIKbP32Z3FKWUxRjDG0t2MOW5RTQ0tfDWb47hLxP7e+0CUt75VakfvLQwn0A/H6Zlp9gd5YicNCCBz/44nszkSG6Zlcs9s9fR2NxidyyllIfw9/Xh12PTWbStlLWFlXbHUarLq2ts5k/vruaOD9YypncMn97gPb3XP0WLbC9WXtvA+ysKOHd4okcu+pIQHsTrV43m6nHpvLwwn0v+s4Ti6nq7YymlPMRF2ckE+/vyxpIddkdRqkvbVbaf859dyLvLC7jhpAxevGIUUR60qMwvpUW2F3tr2U7qGlu4cqxzZhQ5Un6+Pvz1rIE8PjWT1YUVnP3k96zbrVellFKHFx7kz1lDezB71W5q6pvsjqNUl7R8RxmTn17ArrL9zLg8i5tP6YuvT9dYKEqLbC/V0mJ4a+lOxvSKoV93zx9MMDkzkfd/NxYfgQufW8Q8neZPKZcTkYkisklE8kTktkM8nioiX4nIahH5RkSS2j0eLiIFIvJU56X+edNGp1Db0MzHOp2fUp3u49zdTPvPEsKD/Pjo+nGcNCDB7kidSotsL7V4eym7yg4wNTvZ7iguM7BnOB/8fizpcaFc/UoOry/Wj4CVchUR8QWeBk4HBgLTRGRgu8MeBl41xgwF7gX+0e7x+4D57s56JIYnR9IvIYy3lu60O4pSXYYxhqfn5fGHt1YyLCmCD64bS3psqN2xOp0W2V5qVk4BYUF+nDaou91RXCohPIiZ14xhQt84/vrhWv4xZ4POg6uUa2QDecaYbcaYBuBtYHK7YwYCX1vb89o+LiIjgQTgv52QtcNEhGnZyawuqNQBkEp1gqbmFm57bw3/mruJczJ78vrVo7tE//WhaJHtharqGpmzZg+TM3t61aTuB4UG+jH9VyO5bEwqz8/fxp/eXU2Tzjyi1NFKBHa1uV9g7WsrFzjP2j4XCBORGBHxAR4Bbj3ci4jINSKSIyI5xcWdM4f1OcMT8fcVPlxZ2Cmvp1RXVd/UzPVvrmRmzi7+cGIf/n1RJoF+3leHdJQW2V7o49zd1De1cGGW97SKtOfn68PfJg3i5lP68t6KAn7/5gpdIVIp97sVmCAiK4EJQCHQDFwHzDHGHHZSe2PMdGNMljEmKy4uzr1pLZEhAUzoG8/Hq3fTrJ98KeUWtfVNXPVyDp+v28tdZw3kllP7IdI1Bjj+FC2yvdB7ywvolxDGkMQIu6O4lYhww0kZ3HXWQOau28fVr+Swv0FnEFDqFyoE2v5mnmTt+4ExZrcx5jxjzHDgDmtfBTAGuF5E8mnt275MRP7ZGaE7anJmT/ZV1bNke6ndUZTyOhX7G7h0xhIWbi3h4SnD+PU4z53VzJW0yPYyhRUHWLGzgkmZPbvMb5C/HpfOvy4YyoK8En41YylVdY12R1LKEy0DMkQkXUQCgKnA7LYHiEis1RoCcDvwIoAx5hJjTIoxJo3Wq92vGmN+NDuJnU4ekEBIgK/OMqKUi5XVNjB1+mLWFVbx7KUjuWBk0uGf1EVoke1lPl3d+gPk7KE9bU7SuaZkJfP0xSNYXVDB5S8upVoLbaWOiDGmCbgemAtsAN4xxqwTkXtFZJJ12PHAJhHZTOsgxwdsCfsLBAf4ctqg7sxZs1dby5RykbLaBi7+z2K2l9Qy44osr5ts4Whpke1lPlm9h6FJEaTEhNgdpdOdPqQHT108gjUFlVzx0jJdfEKpI2SMmWOM6WuM6W2MecDad5cxZra1/a4xJsM65mpjzI+WYDXGvGyMub6zs3fE5MyeVB5oZP7mErujKOXxytsW2JePYnxG54yx8CRaZHuR/JJaVhdUctbQHnZHsc1pg7rz5LThrNpVwZUvLaVWC22llGVsn1jCg/z4fO1eu6Mo5dHKaxu4+IUlbCup5YXLsxiXEWt3JEfSItuLfLpmDwBndrFWkfZOH9KDJ6YOZ8XOCq58eZkOhlRKAeDv68PJAxL4auM+nfZTqV+oYn8Dl7ywhK3FNbxwWZZewf4ZWmR7kY9zdzMiJZLEyGC7o9juzKE9eOyiTHLyy7jm1eXag6mUAuDUQd2p2N/I0u1ldkdRyuPU1jdx5cvLyCuq4T+XZXFcXy2wf44W2V5iW3ENG/dWd/mr2G2dPawn/7pgGN/nlXDTzFU6P65Sigl94wjy92HuOm0ZUepI1Dc1c+3ry8ndVcGTFw9nghbYh+XWIltEJorIJhHJE5EfTeckIoEiMtN6fImIpFn700TkgIissm7PuTOnN/hi/T4AThuUYHMSZzl/ZBJ3njWQOWv28tcP12CMFtpKdWXBAb4clxHH3HX7aNFfvJXqkOYWw00zV/HdlhIePH+oziLSQW4rskXEF3gaOB0YCEwTkYHtDrsKKDfG9AH+DTzY5rGtxphM63atu3J6iy837GNgj3CSorrerCKHc9W4dK4/oQ9vLd3Fv+ZusjuOUspmpw3qzt6qOlYXVtodRSnHM8ZwxwdrWi9WnTmAKV68mrSrufNKdjaQZ4zZZoxpAN4GJrc7ZjLwirX9LnCSdJUVVFyotKae5TvKOXmgXsX+Kbec2peLR6fwzDdb+c/8bXbHUUrZ6KQB8fgIfL1hn91RlHK8h+Zu4u1lu7j+hD5cPb6X3XE8ijuL7ERgV5v7Bda+Qx5jLYRQCcRYj6WLyEoR+VZExh/qBUTkGhHJEZGc4uJi16b3IF9vLKLFwCkDtMj+KSLCfZMHc+bQHjwwZwMfrSo8/JOUUl4pMiSAkalRzNvUdX9uKNURry/ewbPfbOXi0Snccmpfu+N4HKcOfNwDpBhjhgM3A2+KSHj7g4wx040xWcaYrLi4rtuA/+WGfXQPD2Jw4o++RaoNXx/h0QuHkZ0ezZ9mrWbJtlK7IymlbHJ8v3jWFFZSVFVndxSlHOnrjfu466O1nNg/nnsnDUIbDY6cO4vsQqBt406Ste+Qx4iIHxABlBpj6o0xpQDGmOXAVkB/hTqEusZm5m8u4eSB8fofoAMC/XyZ/quRJEUHc81ry9laXGN3JKWUDU7oFw/AN5v1arZS7a0pqOT6N1cysGc4T04bjp+vU6/JOps7v2vLgAwRSReRAGAqMLvdMbOBy63tC4CvjTFGROKsgZOISC8gA9BG2kNYuLWEA43NnKytIh0WGRLAy1dk4+cjXPnSMkpqfrQytFLKyw3oEUZCeCDfbCqyO4pSjlJQvp9fv7KMqJAAXrxiFKGBfnZH8lhuK7KtHuvrgbnABuAdY8w6EblXRCZZh80AYkQkj9a2kIPT/B0HrBaRVbQOiLzWGKMrBxzClxuKCA3wZUzvmMMfrH6QEhPCjCtGUVRdx9Wv5FDXqIvVKNWViAgn9Ivnu80lNOrqj0oBUHmgkStfWkZdYzMvXzmK+LAguyN5NLde/zfGzDHG9DXG9DbGPGDtu8sYM9varjPGTDHG9DHGZBtjtln73zPGDLKm7xthjPnYnTk9lTGGbzcVc2yfWAL9fO2O43EykyN57KLh5BZUcOPbq3TOXKW6mOP7xVNd38TyHeV2R1HKdk3NLfz+jRXkl9by/K9GkpEQZnckj6dNNh5sW0kthRUHdNWlozBxcHfuOGMAn6/by2NfbbE7jlKqE43tE4O/rzBPW0aU4v5PN/B9XgkPnDuEY3vH2h3HK2iR7cG+taaf0iL76Fw1Lp0pI5N44qstfLp6j91xlFKdJCzIn1Fp0T+cS5Xqqt5eupOXF+Zz1bh0LtTFZlxGi2wPNn9LMb1iQ0mO1lUej4aIcP+5gxmREsmts3JZt1tXgVOqqxifEcfGvdUUVetUfqprWrq9jDs/WstxfeO4/fT+dsfxKlpke6i6xmYWbyvlOL2K7RKBfr4896uRRIb4c82ry3XGEaW6iLF9WgeNL9qq8+arrqegfD+/e305yVEhOlWfG+h300Mtyy+jrrFFW0VcKD4siOm/yqKkpp7rXl9BQ5POOKCUtxvUM4KIYH++31JidxSlOlVtfRO/eXU5Dc0t/OfyLCKC/e2O5HW0yPZQ324qJsDXh9G9ou2O4lWGJEXw0AVDWZpfxj0fr7M7jlLKzXx9hGN7x7AgrwRjdIYh1TUYY/jTu7ls2lvFUxePoHdcN7sjeSUtsj3U/C3FZKdHExKgk8S72uTMRH53fG/eXLKTt5butDuOUsrNxvaJZXdlHfml++2OolSnmD5/G3PW7OX20wfoJ+JupEW2B9pdcYDN+2r0P4Yb3XpqP8ZnxHL3R+tYXVBhdxyllBuN69M6Xdn3edoyorzfwq0lPPj5Rs4c0oOrx6fbHceraZHtgb7b0jrdlA56dB9fH+HxqcOJCwvkd6+voLy2we5ISik3SY0JITEymAXal6283N7KOm54ayXpsaE8eMFQRMTuSF5Ni2wPtHBrKXFhgfRN0B4qd4oODeCZS0ZQXF3PjTNX0awrQirllUSEcX1iWbi1RP+fK6/V0NTCdW8s50BDM8//aiTdArXd1N20yPYwxhgWbi3l2N4x+htoJxiWHMndkwby7eZinvxaV4RUylsd2yeGqrom1hbqPPnKOz3w6XpW7KzgoQuG0Sdel0zvDFpke5itxTUUV9dzbO8Yu6N0GRdnp3D+iCQe/2oL3+jyy0p5pYPLSGtftvJGH64s5JVFO7h6XDpnDu1hd5wuQ4tsD7PQWjDh4A8E5X4iwv3nDKZfQhh/fHsVu8p0BgKlvE1cWCD9u4fpojTK62zcW8Vt768mOy2av+iKjp1Ki2wPsyCvhKSoYF1KvZMFB/jy3KUjaTGG695YQX1Ts92RlFIuNjo9muU7ymls1oWolHeorW/iujdWEBbkz1OXDMdfV3TsVPrd9iDNLYbF28q0VcQmabGhPDxlGGsKK/nnZxvtjqOUcrHRvWI40NjM6gLty1be4c6P1rK9pJbHp2YSHxZkd5wuR4tsD7JhTxWVBxq1VcRGpw3qzhXHpvHSgny+WL/P7jhKKRfKTm9dQXfp9jKbkyh19N5dXsD7Kwq54cQMrRtsokW2B1m4tXVAzhi9km2r28/oz+DEcG6dlUthxQG74yilXCS2WyB94ruxZLv2ZSvPlldUw50fruWYXtHccFKG3XG6LC2yPcjCraX0jgslIVw/8rFToJ8vT00bQXOL4Ya3Vmr/pvIaIjJRRDaJSJ6I3HaIx1NF5CsRWS0i34hIkrU/U0QWicg667GLOj+9a4xOjyYnv5wm/X+tPFRdYzPXv7mC4ABfHp86HF8fne7XLlpke4jG5haWbi/Tj3wcIi02lAfOHczyHeX8+4vNdsdR6qiJiC/wNHA6MBCYJiID2x32MPCqMWYocC/wD2v/fuAyY8wgYCLwmIhEdkpwFxvdK4aa+ibW76myO4pSv8i9n6xn495qHr1wmF6Us5kW2R5idUEF+xuaddCjg0zOTGTqqGSe/XYr8zcX2x1HqaOVDeQZY7YZYxqAt4HJ7Y4ZCHxtbc87+LgxZrMxZou1vRsoAuI6JbWLHWP1ZS/Zpn3ZyvN8nLubN5fs5LcTenF8v3i743R5WmR7iIV5rT2Cx/TSIttJ7j57EBnx3bj5nVUUVdfZHUepo5EI7Gpzv8Da11YucJ61fS4QJiL/c1ISkWwgANjqppxuFR8eRHpsqPZlK4+zo7SW299fw/CUSG49tZ/dcRRaZHuMxdtLGdAjnKjQALujqDaCA3x56uIR1NQ3cdPMVbS0GLsjKeVOtwITRGQlMAEoBH6YNF5EegCvAVcaYw7Z1Cwi14hIjojkFBc78xOg7LRolm4vo1n/PysP0dDUwvVvrsRH4MlpOh+2U+i74AEam1tYsaOC7LQou6OoQ+ibEMbfJg1iQV4p//lum91xlPqlCoHkNveTrH0/MMbsNsacZ4wZDtxh7asAEJFw4FPgDmPM4p96EWPMdGNMljEmKy7OmR0lo3tFU1XXxMa92petPMOjX2xmTWElD10wjKQoXazOKbTI9gDrd1dxoLGZUVavoHKeC7OSOX1wdx7+7ybWFupCFsojLQMyRCRdRAKAqcDstgeISKyIHPy5cTvworU/APiA1kGR73ZiZrcYbbXlaV+28gQLt5bw/PytTMtOZuLg7nbHUW24tcjuwHRQgSIy03p8iYiktXs8RURqRORWd+Z0umX5rSf6UWlaZDuViPCP84YQExrIDW+v5ECDLruuPIsxpgm4HpgLbADeMcasE5F7RWSSddjxwCYR2QwkAA9Y+y8EjgOuEJFV1i2zU78AF0qMDCYpKlj7spXjVexv4OaZuaTHhHLnWe0nA1J2c1uR3cHpoK4Cyo0xfYB/Aw+2e/xR4DN3ZfQUy/LLSIkO0al4HC4yJIBHLxzG9pJa7v90vd1xlDpixpg5xpi+xpjexpgHrH13GWNmW9vvGmMyrGOuNsbUW/tfN8b4G2My29xW2filHLXR6TEsyy/HGO3LVs5kjOGOD9ZSUlPPY1MzCQnwszuSasedV7I7Mh3UZOAVa/td4CQREQAROQfYDqxzY0bHM8aQk1+uV7E9xLF9YrlmfC/eWLJTl11XyoONSouirLaBbSW1dkdR6pDeW1HIp2v2cPOpfRmaFGl3HHUI7iyyOzId1A/HWB9VVgIxItIN+Avwt597AU8YpX60thbXUlrbwCgd9Ogxbj61L4N6hvOX91ZTVKXT+inlibKsc+7y/HKbkyj1YztKa7n7o7Vkp0fz2+N62x1H/QSnDny8B/i3Mabm5w7yhFHqRyvnYD+2Dnr0GIF+vjw+NZP9DU3cMitXp/VTygP1iu1GZIg/OTt08KNylsbmFv749ip8fIR/X5Spy6Y7mDuL7MNOB9X2GBHxAyKAUmA08JCI5AM3Av9PRK53Y1bHWppfRkxoAL1iQ+2Ooo5An/gw/nrmQL7bUsLLC/PtjqOUOkI+PkJWahQ5eiVbOcyTX+exalcFfz93CImRwXbHUT/DnUX2YaeDsu5fbm1fAHxtWo03xqQZY9KAx4C/G2OecmNWx8rJLycrLQqrVV15kEtGp3DygHj++dlGNuzR+XaV8jQjU6PZVlJLaU293VGUAlo/3X7q6y2cNyKRs4f1tDuOOgy3FdkdnA5qBq092HnAzcCPpvnryvZV1bGzbL8OevRQIsKD5w8lPNifG99eRV2jTuunlCf5oS97h17NVvarrmvkxpmrSIwK5m+TBtkdR3WAW3uyOzAdVJ0xZooxpo8xJtsY86Pl8owx9xhjHnZnTqdaul3nx/Z0Md0C+deUoWzaV82/v9hsdxyl1BEYkhhBgK+PFtnKEe77ZD27Kw7w2EWZhAX52x1HdYBTBz4qWj8WCgnwZVDPcLujqKNwQr94Lh6dwvTvtv3wi5NSyvmC/H0ZkhTxw4JgStnl6437eCengN9O6M3IVL3w5im0yHawpfnlDE+JxM9X3yZPd8cZA0iOCuGWWauoqW+yO45SqoOyUqNYW1il7V7KNhX7G7jtvTX0SwjjxpMz7I6jjoBWbw5VVdfIxr1V2iriJUID/Xj0wmEUlB/gAV0NUimPMTI1iobmFtYUVtodRXVRd89eR1ltA49cOIxAP1+746gjoEW2Qy3fUY4xkK1FttfISmtdNOCtpbuYt7HI7jhKqQ4Ymdo6+FFbRpQdPl+7h49W7eb6E/swODHC7jjqCGmR7VDLtpfh5yNkpkTaHUW50E2nZNC/exh/fm815bUNdsdRSh1GTLdAesWF6sqPqtOV1tRzxwdrGZwYzu9P6GN3HPULaJHtUDn55QxKjCAkwM/uKMqFAv18efTCTCr2N/DXD9dijK4GqZTTZaVGsXxnua7eqjqNMYa/friW6romHpmSib+OzfJI+q45UH1TM6sKKhhlfUypvMvAnuHcdEpfPl2zh9m5u+2Oo5Q6jKy0aCr2N7KtpMbuKKqLmJ27m8/W7uXGUzLo1z3M7jjqF9Ii24HWFFTS0NTCqHTtx/ZWvz2uNyNTo7jzw7XsrayzO45S6mdk/dCXrS0jyv2Kquq466N1ZCZHcs34XnbHUUdBi2wHWmoNsMnSK9ley9dHeGTKMBqbDX96N1fbRpRysPTYUGJCA8jRIlu5mTGG299fQ11jM49cOEyn8PVw+u450LLtZfSOCyWmW6DdUZQbpcWGcseZA/huSwmvL9lpdxyl1E8QEUakRrF8h84wotxr1vICvtpYxJ8n9qd3XDe746ijpEW2w7S0GHJ2lJOtrSJdwiWjUziubxx//3QD20tq7Y6jlPoJWalR5Jfup6Sm3u4oykvtrjjAfR+vJzs9miuPTbM7jnIBLbIdZtO+aqrrmsjSZVO7BBHhofOH4u8r/GlWLs06e4FSjpSV1tq+t3yHtowo1zPG8Jf3VtNsDA9fMAwfH7E7knIBLbIdJsfqx9Yr2V1H94gg/jZ5EDk7ynlpwXa74yilDmFQzwgCfH1YoUW2coM3luzkuy0l3H7GAFJiQuyOo1xEi2yHWZpfTkJ4IElRwXZHUZ3onMxEThmYwL/mbmJrsU4TppTTBPn7MiQpghwtspWL7Szdz9/nbGBcn1guHZ1idxzlQlpkO4gxhmXbyxiVFo2IflTUlYgID5w7mOAAX27VthGlHGlkahRrCiqpb2q2O4ryEi0trTNM+Yrw4AVD9We/l9Ei20EKyg+wt6pOW0W6qPiwIO6dPJiVOyt44bttdsdRSrUzIiWKhuYW1hZW2R1FeYmXF+azZHsZd541kMRI/QTb22iR7SDLfpgfW4vsrursoT2YOKg7j3yxmS37qu2Oo5RqY2TqwcGPOpWfOnpbi2t48PONnNg/nilZSXbHUW6gRbaDLMsvJyzIT5dQ7cJEhPvPHUy3QD9unZVLU3OL3ZGUUpa4sEBSY0J0hhF11JpbDLfOyiXI35d/njdE20S8lBbZDrIsv4yRqVH46tQ9XVpst0DumzyY3IJKnp+vbSPqyImIr4hMEpEbROTmgze7c3mDkSlRLN9Roau0qqMyff42Vu6s4N7Jg4gPD7I7jnITLbIdoqy2gbyiGkalaauIgjOH9uDMoT147MvNbNyr/Z/qiH0MXAHEAGFtbuoojUyLoqSmnp1l++2OojzUpr3V/PuLzUwc1J1Jw3raHUe5kZ/dAVQrnR9btXff5MEs2VbKrbNy+eC6sfj76u/EqsOSjDFD7Q7hjf6vL7uc1JhQm9MoT9PY3MIts1bRLciP+88drG0iXk5/ajvEsvwyAnx9GJIYYXcU5RDRoQHcf84Q1hZW8ew3W+2OozzLZyJyqt0hvFFGfBhhgX46X7b6RZ6Zt5W1hVU8cM5gYrsF2h1HuZkW2Q6xLL+cYckRBPn72h1FOcjEwd2ZnNmTJ77awrrdlXbHUZ5jMfCBiBwQkSoRqRYR7TtyAV8fYXhqlK78qI7Y2sJKnvx6C5OG9eT0IT3sjqM6gRbZDrC/oYm1hZVkaT+2OoR7zh5EVGgAt85aTUOTzjaiOuRRYAwQYowJN8aEGWPC7Q7lLUamRLFpXzVVdY12R1Eeor6pmVveySUqNIB7Jw+yO47qJG4tskVkoohsEpE8EbntEI8HishM6/ElIpJm7c8WkVXWLVdEznVnTrut2llBU4shW4tsdQhRoQH8/dwhbNhTxVPz8uyOozzDLmCtOcIpMDpwzk4Vka9EZLWIfCMiSW0eu1xEtli3y13wNThWVloUxsDKnRV2R1Ee4vEvt7BpXzUPnj+EyJAAu+OoTuK2IltEfIGngdOBgcA0ERnY7rCrgHJjTB/g38CD1v61QJYxJhOYCDwvIl47SHNZfjkiMMIaUKNUe6cMTOC84Yk8PS+PtYXaNqIOaxvwjYjc3tEp/Dp4zn4YeNUaVHkv8A/rudHA3cBoIBu4W0S89oQ2LDkSH0Hny1YdsnJnOc99u5UpI5M4sX+C3XFUJzpskW3Nt7rxF/zd2UCeMWabMaYBeBuY3O6YycAr1va7wEkiIsaY/caYJmt/EODVE5Iuyy+jX0IYEcH+dkdRDnb32YOICQ3glndyqW9qtjuOcrbtwFdAAB2fwq8j5+yBwNfW9rw2j58GfGGMKTPGlANf0HqBxCt1C/Sjf/dw7ctWh1XX2Mwts3LpHh7EnWe3/51VebvDFtnGmGZgk4ikHOHfnUjrR5YHFVj7DnmMVVRX0jqvKyIyWkTWAWuAa9sU3T8QkWtEJEdEcoqLi48wnjM0NbewYme5Tt2nDisixJ9/nj+ETfuqefIrbRtRP80Y8zdjzN+AR4BH2tz/OR05Z+cC51nb5wJhIhLTwecC3nHehtaWkZU7y3VVVvWzHp67iW3FtTx4wVDCg/RCWlfT0XaRKGCd1Ys3++DNncGMMUuMMYOAUcDtIvKjJZGMMdONMVnGmKy4uDh3xnGb9Xuq2N/QrIMeVYec2D+BKSOTePbbreTuqrA7jnIoERksIiuBdbSeu5eLiCtGW90KTLD+7glAIXBEH6t4w3kbWufLrm1oZtO+arujKIdaur2MGQu2c8noFMZneO6/dfXLdbTP+c5f8HcXAslt7idZ+w51TIHVcx0BlLY9wBizQURqgMFAzi/I4WhLt1uL0GiRrTror2cN5Pu8Em6dlcvHfxin0z6qQ5kO3GyMmQcgIscD/wGO/ZnnHPacbYzZjXUlW0S6AecbYypEpBA4vt1zvzmaL8DpRqT836I0g3rq+gbqf9XWN3HrrFySooL5f2cMsDuOskmHrmQbY7491O0wT1sGZIhIuogEAFOB9le/ZwMHR6FfAHxtjDHWc/ygdTQ70B/I7+DX5FFy8stJjg6me8SPLtQrdUgRwf788/yhbCmq4bEvt9gdRzlT6MECG8AY8w1wuOUJD3vOFpFYETn4c+N24EVrey5wqohEWQMeT7X2ea2kqGASwgN18KM6pH9+tpFd5fv51wXDCA302nkb1GH8bJF9cAGDQ9wOu7CB1UN9Pa0n2g3AO8aYdSJyr4hMsg6bAcSISB5wM3BwyqhxQK6IrAI+AK4zxpT84q/SoYwxLMsvY1SqXsVWR2ZC3zimZSczff5WVuzUH/LqR7aJyJ0ikmbd/krrjCM/qYPn7ONpHaOzGUgAHrCeWwbcR2uhvgy419rntUSEkalRWmSrH1mQV8Jri3dw5bHpHNMrxu44ykY/++uVMeZwo9F/ljFmDjCn3b672mzXAVMO8bzXgNeO5rU9wfaSWkprGxilgx7VL/D/zhjA/M0l3PpOLp/eMJ7gAG0bUT/4NfA34D3r/nfAlYd7UgfO2e/SOhPUoZ77Iv93ZbtLGJESxZw1e9lXVUdCuH4aqaC6rpE/v7uaXrGh/HliP7vjKJvpio82WpbfeqFnVJrXTier3CgsyJ+HLhjKtpJa/jV3k91xlLP0prW/2ofWafxOAubbmsgLHRywrlez1UH3f7KBPZUHePjCYTpeRmmRbael28uJDg2gd1w3u6MoDzW2TyyXjUnlpYXbWbyt9PBPUF3FG7ReVT4POMu6nW1rIi80sEc4gX4+WmQrAOZtLGJmzi6uOa73DwNjVdemRbaNcnaUkZUahYjYHUV5sNtO709KdAh/ejeX2vofTSevuqZiY8zHxpjtxpgdB292h/I2AX4+DEuKJEeL7C6vcn8jt72/mr4J3bjplAy74yiH0CLbJvuq6thRup9ROnWfOkohAX48PGUYBeUH+PucDXbHUc5wt4i8ICLTROS8gze7Q3mjkWlRrCuspK5RV2Htyu75eB2lNQ08MiWTQD9tE1GttMi2yQ/zY+ugR+UCo9KiuXpcOm8s2cn8zZ67ip5ymSuBTFqXNj/bup1lZyBvNTIliqYWo4tDdWGfr93LBysL+f0JfRiSpHOmq/+jRbZNluWXERLgy6Ce4XZHUV7illP70TsulL+8t5rKA412x1H2GmWtqni5MeZK6/Zru0N5oxGp1qI0OpVml1RSU88dH6xhUM9wrj+xj91xlMNokW2TpdvLGJEShZ+vvgXKNYL8fXnkwkyKquu575P1dsdR9looIgPtDtEVRIcG0CsulBXal93lGGP46wdrqa5r4tELM/HXn+eqHf0XYYPK/Y1s2letrSLK5TKTI/ndhN68u7yAL9bvszuOss8xwCoR2SQiq0VkjYistjuUtxqZ0roojTHG7iiqE320ajefr9vLzaf2pV/3o1pWRHkpLbJtkLOjDGPQQY/KLW44KYP+3cO4/f01lNc22B1H2WMikEHr8uYH+7F1Cj83yUqLonx/I9tKau2OojrJ3so67vpoLSNSIvnN+F52x1EOpUW2DZbml+HvKwxPibQ7ivJCAX4+PHphJpUHGrjzo7V2x1E2aDttn07h534jD/Zl52vLSFdgjOEv762msdnwyIWZ+ProNLzq0LTItsGy7WUMTYrU1aCU2wzsGc4NJ2bwyeo9fLJ6t91xlPJqvWK7ERHsr4vSdBFvLd3Ft5uLuf2M/qTHhtodRzmYFtmd7EBDM6sLKrVVRLnd747vzbCkCO78cC3F1fV2x1HKa/n4CCNTo3SGkS5gZ+l+7v90PWP7xHDp6FS74yiH0yK7k63cVU5TiyE7XZdcVe7l5+vDIxcOo7ahmf/3wRodlKWUG41MjSKvqIaK/ToOwlu1tBhufTcXHxEeumAYPtomog5Di+xOtmx7OSIwMlWvZCv36xMfxq2n9uWL9fv4YGWh3XGU8lojUlovnKzQq9le66WF+SzdXsZdZw8kMTLY7jjKA2iR3cmW5pfSv3s4EcH+dkdRXcRV43qRlRrF3bPXsafygN1xlPJKmcmR+PqI9mV7qbyiGh76fCMnD4hnysgku+MoD6FFdidqbG5hxY4KstO0VUR1Hl8f4eEpw2hqNvzlPW0bUcodgq0VfHN0hhGv09Tcwi2zcgkO8OXv5w1BRNtEVMdokd2J1u2u4kBjM6N0ERrVydJiQ7n9jP7M31zMm0t32h1HKa80MjWK3IIKGptb7I6iXOi5b7eSu6uC+88ZTHxYkN1xlAfRIrsTLd1eCkC2ziyibHDp6FTG9onhgU83kK+LZijlciNTo6hrbGH97iq7oygXWb+7ise/2sJZQ3tw1tCedsdRHkaL7E60dHs5aTEhxIfrb8Kq8/n4CP+6YBh+PsKNM1fRpFfblHKpHxal0b5sr1DX2MxNM1cRGRLAfZMH2x1HeSAtsjtJS4shZ0eZzo+tbNUzMpj7zx3Cql0VPD1vq91xlPIqPSKCSYwM1vmyvcS/5m5i075q/nXBUKJCA+yOozyQFtmdZEtRDRX7G8nWfmxls0nDenJOZk+e+HoLq3ZV2B1HKa8yIjWK5fnlOsDYwy3IK2HG99u5bEwqx/eLtzuO8lBaZHeSpfllAFpkK0f42+TBJIQFctPMVexvaLI7jlJeY2RKJHur6thdWWd3FPULVexv4JZ3cukdF8rtpw+wO47yYFpkd5LFW0vpHh5ESnSI3VGUIiLYn0cuzCS/tJb7P91gdxylvEaW1RKofdmeyRjDHR+upaSmnsenDic4wNfuSMqDaZHdCYwxLN5WypjeMTq/pnKMMb1juGZ8L95cspOvNuyzO45SXqF/9zCC/X1Zbn16qTzLh6sK+XT1Hm46pS+DEyPsjqM8nFuLbBGZKCKbRCRPRG47xOOBIjLTenyJiKRZ+08RkeUissb680R35nS3zftqKK1tYEzvGLujKPU/bj61LwN6hPOX91ZTUlNvdxylPJ6frw+ZyZE6+NED7Srbz10frmNUWhTXTuhtdxzlBdxWZIuIL/A0cDowEJgmIgPbHXYVUG6M6QP8G3jQ2l8CnG2MGQJcDrzmrpydYdHWEgDG9NIiWzlLoJ8vj12USVVdE7e9t1oHaynlAllpUWzYU01tvY538BTNLYZb3snFAI9emImvj37qrI6eO69kZwN5xphtxpgG4G1gcrtjJgOvWNvvAieJiBhjVhpjdlv71wHBIhLoxqxutWhbKUlRwSRrP7ZyoH7dw/jLxP58uaGIt5ftsjuOUh5vRGoUzS2GXJ29x2NMn7+Npfll/G3SIP1ZrVzGnUV2ItD2J3aBte+QxxhjmoBKoP3l3vOBFcaYH32WLSLXiEiOiOQUFxe7LLgrtbQYlmwv06vYytGuPDaNsX1iuPfj9WzX1SCVOiojknVRGk+ytrCSR7/YxJlDenDeiPZlilK/nKMHPorIIFpbSH57qMeNMdONMVnGmKy4uLjODddBG/ZWUbG/UfuxlaP5+AgPTxlGgJ8PN85cRaOuBqnULxYR4k/fhG7kaJHtePsbmrhx5iqiQwN44NzBOjmBcil3FtmFQHKb+0nWvkMeIyJ+QARQat1PAj4ALjPGeOzSdIu2lgJoka0cr0dEMP84bwi5uyp45L+b7Y6jlEcbmRrFip3ltLToOAcnu/fj9WwtruHRCzOJDNFVHZVrubPIXgZkiEi6iAQAU4HZ7Y6ZTevARoALgK+NMUZEIoFPgduMMQvcmNHtFm8rJS0mhB4RwXZHUeqwzhjSg2nZKTz37Va+2+LMFizlPh2YESpFROaJyEoRWS0iZ1j7/UXkFWtGqA0icnvnp3eWkanRVNc1kVdcY3cU9RM+Wb2bt5ft4ncTejO2T6zdcZQXcluRbfVYXw/MBTYA7xhj1onIvSIyyTpsBhAjInnAzcDBk/r1QB/gLhFZZd08bl3T5oP92HoVW3mQu84aSEZ8N25+J1en9etCOjgj1F9pPZcPp/XCyTPW/ilAoDUj1EjgtwenZO2qRqa29mXn5GvLiBPtKtvP7e+vYXhKJDed0tfuOMpLubUn2xgzxxjT1xjT2xjzgLXvLmPMbGu7zhgzxRjTxxiTbYzZZu2/3xgTaozJbHMrcmdWd1i3u5LquiaO0UGPyoMEB/jy5MXDqTrQyC3v5OrH3V1HR2aEMkC4tR0B7G6zP9Rq+wsGGoAq90d2rrSYEGJCA3TwowM1Nrdww9srwcATU4fj7+vo4WnKg+m/LDf6oR9bi2zlYfp3D+evZw3k283FzPh+u91xVOfoyIxQ9wCXikgBMAf4g7X/XaAW2APsBB42xnTpJQ9FhBGpUeTs6NLfBkf69xebWbmzgr+fN0Sn61NupUW2Gy3cWkrvuFDiw4PsjqLUEbt0dAqnDUrgobkbWV1QYXcc5QzTgJeNMUnAGcBrIuJD61XwZqAnkA7cIiK9DvUXeMLUq66SnRbNjtL97K2sszuKsizIK+HZb7dyUVYyZw/raXcc5eW0yHaT+qZmlmwvZXyGM6cWVOpwRIQHzx9KbLdAbnhrJTW6ep2368iMUFcB7wAYYxYBQUAscDHwuTGm0WrtWwBkHepFPGHqVVc5OB5n8bZSm5MogNKaem6auYpesaHcPan9cAOlXE+LbDdZnl9OXWML4zN0xLLyXJEhATw+dTg7y/Zz54dr7Y6j3KsjM0LtBE4CEJEBtBbZxdb+E639ocAxwMZOyu1YA3qEEx7k90ProLJPS4vh1lm5VBxo5MlpIwgJ8LM7kuoCtMh2k/lbSvD3FR30qDxedno0N5yUwQcrC3knR5dd91YdnBHqFuA3IpILvAVcYYwxtM5K0k1E1tFarL9kjFnd+V+Fs/j6CNnpMSzerkW23WZ8v515m4q544wBDOwZfvgnKOUC+qucm3y3pZgRKVGEBuq3WHm+P5yYwdLtZdz54VqGJEYwoIf+kPJGxpg5tA5obLvvrjbb64Gxh3heDa3T+Kl2xvSO4csN+9hdcYCekbpegh1y8sv45+cbOW1QApeNSbU7jupC9Eq2G5TU1LNudxXH9fXufkPVdfj6CI9PHU5EsD/XvbGC6rpGuyMp5REOzi6lLSP2KKmp5/o3V5IUFcy/pgzTZdNVp9Ii2w0W5JUAaD+28ipxYYE8Oa21P/sv762mtUtAKfVz+ncPIzLEXwc/2qC5xXDj26so29/AM5eMIDzI3+5IqovRItsN5m8uISrEn0E9I+yOopRLje4Vw59P68ecNXt5eWG+3XGUcjwfH2F0ejSLtMjudE98tYXv80q4d9Ig/XmsbKFFtosZY/huSzFj+8Ti66MfSynvc81xvTh5QAIPfLqBFTt1NTulDmdMrxgKyg+wq2y/3VG6jPmbi3ni6y2cPyKJi0YlH/4JSrmBFtkutqWohqLqeo7T+bGVlxIRHpkyjB6RQVz/xgrKahvsjqSUo43p3do6qC0jnWNP5QFunLmKvvFh3H/OYO3DVrbRItvF5m9uXcFsnPZjKy8WEeLPMxePpKSmgRtnrqKlRfuzlfopGfHdiA4N0JaRTtDQ1ML1b66kvrGZZy4dQXCAr92RVBemRbaLzdtUREZ8N52qSXm9IUkR3D1pIPM3F/PYV1vsjqOUY/n4CMf0imbJtjIdMOxm932ynuU7ynnwgqH0jutmdxzVxWmR7ULVdY0s2VbGSQMS7I6iVKe4ODuFKSOTeOKrLcxdt9fuOEo51pheMRRWHGBHqfZlu8vMZTt5bfEOfntcL84a2tPuOEppke1K320poanFcNKAeLujKNUpRIT7zhnMsORIbp65ii37qu2OpJQjjbPG6XxnTfGqXGvFznLu/HAd4zNi+fPE/nbHUQrQItulvtpQRGSIP8OTI+2OolSnCfL35blLRxAc4MdvXs2h8oAuVKNUe2kxISRFBf8wbke5TlF1Hb97fTkJEa1z+evMXsoptMh2keYWwzebipjQNw4/X/22qq6lR0Qwz106gsKKA/zx7ZU060BIpf6HiDA+I45FW0tpbG6xO47XaGhq4brXV1B1oInpv8oiMiTA7khK/UCrQRfJLaigtLaBE/trq4jqmrLSorln0iC+2VTMI//dZHccpRxnQt9YauqbWLWrwu4oXuNvH68jZ0c5/5oylAE9wu2Oo9T/0CLbRb7eUISvjzChr86PrbquS0anMi07hWe+2conq3fbHUcpRxnTu3WRMm0ZcY3XF+/gjSU7+e0EHeionEmLbBf5amMRI1Oj9KMq1eXdM2kgI1OjuOWdXL1ip1QbEcH+ZCZHMn+LDn48WvM3F3P37HWc2D+eP5+mAx2VM2mR7QI7S/ezYU8VJ+usIkoR6OfL9F+NJD48kKtfyaGgXKcsU+qg8RmxrC6ooFxXSv3Ftuyr5vdvrCAjvhtP6EBH5WBaZLvAZ2v3AHD64B42J1HKGWK6BfLi5aOob2rmqpdzqK7TGUeUAjiubxzGwIKtejX7lyitqefXrywj0N+XGVeMolugn92RlPpJWmS7wGdr9zIkMYLk6BC7oyjlGBkJYTx7yUjyimv4w1sradIZFZRiaGIE4UF+2pf9C9Q1NnPNa8spqqrnhcuzSNSVlZXDaZF9lAorDrBqVwWnD+ludxSlHGdcRiz3TR7MN5uKue+T9XbHUcp2fr4+jMuIZf7mEl1i/QgYY7j9/TUs31HOoxdmkqnrUSgP4NYiW0QmisgmEckTkdsO8XigiMy0Hl8iImnW/hgRmSciNSLylDszHq3P17YuJa2tIkod2sWjU/jN+HReWbSDF77bZnccpWx3fN949lbVsX5Pld1RPMbD/93EBysLufXUvpw5VH/eKs/gtiJbRHyBp4HTgYHANBEZ2O6wq4ByY0wf4N/Ag9b+OuBO4FZ35XOVz9bsoX/3MNJjQ+2OopRj3Xb6AE4blMD9n27go1WFdsdRylYn9I9HpHWVYHV4ry7K5+l5W5mWnczvT+hjdxylOsydV7KzgTxjzDZjTAPwNjC53TGTgVes7XeBk0REjDG1xpjvaS22HWtfVR05O8o5Y4j+Vq3Uz/H1ER6fOpzs9GhunZXLd1u0H1V1XXFhgWQmR/Llhn12R3G8z9bs4e7Z6zh5QAL3TR6MiM4kojyHO4vsRGBXm/sF1r5DHmOMaQIqgZiOvoCIXCMiOSKSU1zc+T+056w5OKuI9mMrdThB/r7857Isesd149rXlrOmoNLuSErZ5uQBCawuqGRflaOvJdlqybZS/jhzFcOTI3ly2nD8fHUYmfIsHv0v1hgz3RiTZYzJiovr/JUWP1hZyKCe4WQkhHX6ayvliSKC/Xnl19lEhgRwxUtL2V5Sa3ckpWxxysAEQFtGfsqmvdVc/WoOyVHBzLh8FMEBvnZHUuqIubPILgSS29xPsvYd8hgR8QMigFI3ZnKZvKJqVhdUcu7w9hfnlVI/JyE8iFevyqbFGC57cQlFeiVPdUEZ8d1Ijg7WlpFD2F5Sy6UzlhAS4Msrv84mKlRXUlaeyZ1F9jIgQ0TSRSQAmArMbnfMbOBya/sC4GvjIXMavb+iEF8fYVJmT7ujKOVxesd146UrsymtaeCSF5ZQWlNvdySlOpWIcPKABL7PK2F/Q5PdcRyjoHw/l/xnMc0thtevGk1SlK4/oTyX24psq8f6emAusAF4xxizTkTuFZFJ1mEzgBgRyQNuBn6Y5k9E8oFHgStEpOAQM5PYpqXF8OHKQsZnxBIfFmR3HKU8UmZyJDMuH8Wu8v1cOmMpFft1mWnVtZwyIIGGpha+26KrP0LrZAKXvLCEmvomXrsqW1sxlcdza0+2MWaOMaavMaa3MeYBa99dxpjZ1nadMWaKMaaPMSbbGLOtzXPTjDHRxphuxpgkY4xjVrJYvL2U3ZV1nDciye4oSnm0Mb1j+M9lWWwtruGyF5dSpcuvqy5kVHo0kSH+fGYNou/KSmvqueSFJZRU1/PKr7MZ1DPC7khKHTWPHvhol7eW7iI8yI9TBiTYHUUpjzc+I45nLxnBhj1VXPHiUmrq9aNz1TX4+/owcVB3vli/j7rGZrvj2Ka8toFfzVhKQfl+ZlwxiuEpUXZHUsoltMg+QkXVdXy+dg9TspJ1tLNSLnLSgASenDac3IJKrnxpKdV6RVt1EWcO7UFtQzPfbu6ac8eX1NQz7T+LySuu4flfZXFMrw7P4quU42mRfYTeWbaLxmbDJaNT7I6ilFeZOLgHj0/NZOXOCu3RVl3GmF4xRIX48+nqrtcyUlRVx9Tpi8kvreXFy0cxoW/nT8WrlDtpkX0EmppbeHPJTsb1iaVXXDe74yjldc4a2pNnLx3Jht1VTJ2+mBKddUR5OT9fHyYO7sGXG7pWy8ieygNcNH0xuysO8PKV2YzLiLU7klIup0X2EfhqYxG7K+u49JhUu6Mo5bVOGZjAjCuyyC+t5cLnF7G3UufR7iwiMlFENolInojcdojHU0RknoisFJHVInJGm8eGisgiEVknImtERKde6qCzhvZgf0Mz8zZ2jYVpdpTWctHziymurufVX2dri4jyWlpkd5Axhue/3UpiZDAnD4i3O45SXm18Rhyv/no0RVX1THl+ITtKdWVIdxMRX+Bp4HRgIDDtEFOn/pXW6ViH07r2wTPWc/2A14FrjTGDgOMBbazvoNHp0cSHBfL+yvbrtXmftYWVnP/sQqrqGnntqmyy0qLtjqSU22iR3UFLtpexYmcFv53QCz9f/bYp5W7Z6dG8fvVoquuaOO+ZhazaVWF3JG+XDeQZY7YZYxqAt4HJ7Y4xQLi1HQHstrZPBVYbY3IBjDGlxpiu0/twlPx8fTh3RCLzNhZ5dYvUwrwSpk5fTICvD+9eO0ZnEVFeT6vFDnrmm63Edgvgwqzkwx+slHKJzORI3vvdsYQE+jJ1+iK+XK9LULtRIrCrzf0Ca19b9wCXikgBMAf4g7W/L2BEZK6IrBCRP//Ui4jINSKSIyI5xcVdc0aNQ7lgRBJN1kJn3uiT1bu5/KWlJEYG8/51Y+kTrwvNKO+nRXYHLN9RzvzNxVw5Np0gf522T6nO1DuuG+//bix9E8K45rUcXl+8w+5IXdk04GVjTBJwBvCaiPgAfsA44BLrz3NF5KRD/QXGmOnGmCxjTFZcnM4mcVBGQhjDkiN5d3kBxhi747iMMYZnvsnjD2+tZHhyFO/8dgzdI7RdX3UNWmQfhjGGf8zZQFxYIFeOTbM7jlJdUlxYIG9fcwzH94vnrx+u5d6P19PU3GJ3LG9TCLT9qC7J2tfWVcA7AMaYRUAQEEvrVe/5xpgSY8x+Wq9yj3B7Yi8zZWQSG/dWs253ld1RXKKusZmbZq7ioc83cfbQnrx6VTYRIf52x1Kq02iRfRj/Xb+PnB3l3HRyX0IC/OyOo1SXFRLgx/RfjeTKsWm8uGA7l724lLJanUvbhZYBGSKSLiIBtA5snN3umJ3ASQAiMoDWIrsYmAsMEZEQaxDkBGB9pyX3EmcP7Umgnw9vLPH8T2uKquq4aPpiPly1mz+d1o/Hp2bqJ8Gqy9Ei+2fsb2jivk/W0zsulAuzkuyOo1SX5+frw91nD+LhKcPI2VHOpKe+Z72XXPWzmzGmCbie1oJ5A62ziKwTkXtFZJJ12C3Ab0QkF3gLuMK0KgcepbVQXwWsMMZ82ulfhIeLCPHn3OGJvL+ikHIP/gVyxc5yJj21gC37qnnu0pH8/oQ+iIjdsZTqdFpk/4yH526moPwA/zhvqM4oopSDXDAyiVm/HUNTs+G8Zxfw3vICuyN5BWPMHGNMX2NMb2PMA9a+u4wxs63t9caYscaYYcaYTGPMf9s893VjzCBjzGBjzE8OfFQ/74qxadQ3tfD2sl2HP9hhjDG88N02LnxuEf5+wrvXHsvEwd3tjqWUbbRy/AnL8st4aeF2Lj0mhex0ncdTKacZlhzJ7D+MZVhSJLfMyuWmmauoqW+yO5ZSR6V/93CO7R3Da4vyPWrcQeX+Rq55bTn3f7qBkwbE88kfxjOwZ/jhn6iUF9Mi+xCKquv4/RsrSI0O4S8T+9sdRyn1E+LDgnjzN8dw08l9+WhVIWc98R1rCirtjqXUUblybDq7K+v4fN1eu6N0yMKtJZzxxHfM21jEnWcN5LlLRxIRrAMcldIiu53KA41c+dIyquoaefbSkYQF6YlCKSfz9RH+eHIGb18zhvqmFs59ZgGPfrGZhibPuQqoVFsn9o+nV2woT32dR0uLc6fzO9DQzD2z13Hxf5YQ4OfDrGvHcNW4dO2/VsqiRXYbeyvruPSFJWy2BmsM6KEfdSnlKbLTo/nsj+OZNKwnT3y1hbOf/J7VBRV2x1LqiPn6CDeclMHGvdXMdejV7Jz8Ms584jteXpjPFcemMeeG8bqCo1LtaJFN61yeby3dyRlPfMe24hqe/9VIju8Xb3cspdQRigwJ4NGLMplxeRYVBxo45+kF3P/JeqrqGu2OptQROXtYT3rHhfLYl1scdTW7tKaeP83K5YLnFlHf1MIbV4/mnkmDCA7Q6fmUaq9LT/w84/vtzM7dTd6+amobmslKjeIf5w0hI0GXe1XKk500IIH/pkXzz882MGPBdj5cVcifT+vPBSOT8PHRj7KV8/n6CDee3Jc/vLWSWct3cdGoFFvzNLcY3l62k4c+30RtfRPXTujNDSf10fUjlPoZXfp/R0uLITzIj3NHJDJxUA/G9onRXjKlvEREsD//OG8o07JTuGf2Ov783mpeX7KDv0zsz7G99f+6cr6zhvbg1UX5PPj5JiYO6mHLaonGGL5Yv49/zd3ElqIaRqdHc/85g/VilFIdIMY452Ooo5GVlWVycnLsjqGUcqCWFsOHqwp56PNN7K2qIzs9mptP6csxvWLsjvYDEVlujMmyO0dn0vP24a3bXcnZT37PtOwUHjh3SKe9rjGGBXmlPPblZnJ2lNMrNpRbT+vH6YO76y+oStGxc3aXvpKtlOoafHyE80YkccaQHry9dCdPf7OVqdMXc0yvaK4a14sT+8fjq20kyoEG9YzgyrHpzPh+Oyf0i+fkgQlufb3mFsPna/fy7Ld5rC2sIiE8kL+fO4QpWUn466JsSh0RvZKtlOpy6hqbeWPJTl74bht7KutIiQ7hsjGpnDs8kZhugbZk0ivZ6qfUNzVz7tML2VN5gDl/HE+PiGCXv8beyjreydnFzGW7KKw4QK/YUK45rhfnjkgk0E8HNSrVXkfO2VpkK6W6rKbmFuau28dLC7aTs6McPx9hQt84zh2RyAn94gkN7LwP+7TIVj8nr6iGc55eQM/IIGb99liX9GdX7G/gi/X7mLNmD99uLqbFwLg+sVx6TAqnDOyun+4o9TNsbxcRkYnA44Av8IIx5p/tHg8EXgVGAqXARcaYfOux24GrgGbgBmPMXHdmVUp1PX6+Ppw5tAdnDu3Bxr1VfLCykI9W7uarjUUE+Powulc0J/SLZ2yfWPrEd9OiQ9mmT3w3pv9qJFe8tIyLpi9ixhWjSIw8sivaTc0trN1dxaKtpSzIK2HxtlKaWgyJkcFcO6E3F41KJjUm1E1fgVJdj9uuZIuIL7AZOAUoAJYB04wx69sccx0w1BhzrYhMBc41xlwkIgOBt4BsoCfwJdDXGNP8U6+nV0SUUq7Q3GJYur2MrzfuY96mYvKKagDoFujHsOQIBidG0Cs2lPTYbqTGhBAVEkCA39H3quqVbNUR320p5rrXVyACN53Sl/NHJhHebmXi+qZm9lXWs6fyAPmltWzYU836PVVs2F1FdX0TABnx3ThpQAJnDOnOkMQIHcyo1BGytV1ERMYA9xhjTrPu3w5gjPlHm2PmWscsEhE/YC8QB9zW9ti2x/3U6+nJWinlDrvK9rMsv4yVOytYuaucTXuraWz+3/NmaIAvkSEBBPr7cNaQHtx8ar8jfh0tslVHbSuu4Y4P1rJoWyl+PkL3iCBEoK6xhbqG5h8K6YNCAnzp3z2MgT3DGZ0ewzG9YogLs2fsgVLewu52kURgV5v7BcDonzrGGNMkIpVAjLV/cbvnJrZ/ARG5BrgGICXF3on6lVLeKTk6hOToEM4bkQS0fuS+u6KO7aW17CzbT0VtAxUHGqnY30h9UzMJEUE2J1berldcN978zWhyCyr5Yv1eCssPICIE+fsS5O9DVEgAPSKC6BERTHJ0MMlRIboIk1I28Ogp/Iwx04Hp0HpFxOY4SqkuwM/Xh5SYEFJiQuyOorowESEzOZLM5Ei7oyilfoI7J70sBJLb3E+y9h3yGKtdJILWAZAdea5SSimllFKO5M4iexmQISLpIhIATAVmtztmNnC5tX0B8LVpbRKfDUwVkUARSQcygKVuzKqUUkoppZTLuK1dxOqxvh6YS+sUfi8aY9aJyL1AjjFmNjADeE1E8oAyWgtxrOPeAdYDTcDvf25mEaWUUkoppZzErT3Zxpg5wJx2++5qs10HTPmJ5z4APODOfEoppZRSSrmDO9tFlFJKKaWU6pK0yFZKKaWUUsrFtMhWSimllFLKxbTIVkoppZRSysXctqx6ZxORYmDHL3hqLFDi4jiu5vSMTs8Hzs/o9Hzg/IxOzwc/nzHVGBPXmWHs9gvP257+PjuB0/OB8zM6PR84P6PT88FRnrO9psj+pUQk53Brz9vN6Rmdng+cn9Hp+cD5GZ2eDzwjo9N5wvfQ6Rmdng+cn9Hp+cD5GZ2eD44+o7aLKKWUUkop5WJaZCullFJKKeViWmTDdLsDdIDTMzo9Hzg/o9PzgfMzOj0feEZGp/OE76HTMzo9Hzg/o9PzgfMzOj0fHGXGLt+TrZRSSimllKvplWyllFJKKaVcTItspZRSSimlXKxLF9kiMlFENolInojcZlOGF0WkSETWttkXLSJfiMgW688oa7+IyBNW3tUiMqIT8iWLyDwRWS8i60Tkjw7MGCQiS0Uk18r4N2t/uogssbLMFJEAa3+gdT/PejzN3Rmt1/UVkZUi8olD8+WLyBoRWSUiOdY+x7zP1utGisi7IrJRRDaIyBinZBSRftb37uCtSkRudEo+b+CEc7aVQ8/bR5fPI87Z1ms79ryt5+yjzub+c7YxpkveAF9gK9ALCABygYE25DgOGAGsbbPvIeA2a/s24EFr+wzgM0CAY4AlnZCvBzDC2g4DNgMDHZZRgG7Wtj+wxHrtd4Cp1v7ngN9Z29cBz1nbU4GZnfRe3wy8CXxi3Xdavnwgtt0+x7zP1uu+AlxtbQcAkU7LaL22L7AXSHViPk+8OeWcbWXR8/bR5fOIc7b1eo49b+s526U53XLO7pTwTrwBY4C5be7fDtxuU5a0difrTUAPa7sHsMnafh6YdqjjOjHrR8ApTs0IhAArgNG0rtLk1/79BuYCY6xtP+s4cXOuJOAr4ETgE+s/qWPyWa91qBO2Y95nIALY3v574aSMbV7rVGCBU/N54s1J52zr9fW87ZpsjjxnW6/l6PO2nrNdmtUt5+yu3C6SCOxqc7/A2ucECcaYPdb2XiDB2rY1s/Xx13Barzo4KqP1kd4qoAj4gtYrXhXGmKZD5Pgho/V4JRDj5oiPAX8GWqz7MQ7LB2CA/4rIchG5xtrnpPc5HSgGXrI+vn1BREIdlvGgqcBb1rYT83kip3+/HPk+O/W87QHnbHD+eVvP2a7jlnN2Vy6yPYJp/XXJ2J1DRLoB7wE3GmOq2j7mhIzGmGZjTCatVx6ygf525mlLRM4Ciowxy+3OchjjjDEjgNOB34vIcW0fdMD77EfrR/TPGmOGA7W0fpT3AwdkxOrRnATMav+YE/Ip93PK++zk87aTz9ngMedtPWe7gDvP2V25yC4EktvcT7L2OcE+EekBYP1ZZO23JbOI+NN6on7DGPO+EzMeZIypAObR+jFepIj4HSLHDxmtxyOAUjfGGgtMEpF84G1aP3p83EH5ADDGFFp/FgEf0PqDz0nvcwFQYIxZYt1/l9YTuJMyQusPvBXGmH3Wfafl81RO/3456n32lPO2Q8/Z4AHnbT1nu4zbztlducheBmRYI4UDaP2oYLbNmQ6aDVxubV9Oaz/dwf2XWSNcjwEq23yk4RYiIsAMYIMx5lGHZowTkUhrO5jW3sMNtJ64L/iJjAezXwB8bf226hbGmNuNMUnGmDRa/519bYy5xCn5AEQkVETCDm7T2p+2Fge9z8aYvcAuEeln7ToJWO+kjJZp/N/HjgdzOCmfp3LyORsc9D47/bzt9HM2OP+8redsl3LfOdvdzeROvtE6UnQzrb1gd9iU4S1gD9BI6299V9Hax/UVsAX4Eoi2jhXgaSvvGiCrE/KNo/WjktXAKut2hsMyDgVWWhnXAndZ+3sBS4E8Wj8GCrT2B1n386zHe3Xi+308/zdK3TH5rCy51m3dwf8PTnqfrdfNBHKs9/pDIMpJGYFQWq9eRbTZ55h8nn7DAedsK4eet48un8ecs63XPx6HnbfRc7ar8rn1nK3LqiullFJKKeViXbldRCmllFJKKbfQIlsppZRSSikX0yJbKaWUUkopF9MiWymllFJKKRfTIlsppZRSSikX0yJbKaWUUkopF9MiWymllFJKKRf7/8WWkgjoBhNWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_sched()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc9bb55-f30c-424c-a4de-cc5836b89510",
   "metadata": {},
   "source": [
    "Here we can see the learning rate increases linearly until the maximum value, before being annealed with a cosine function. The second plot refers to a hyperparameter called _momentum_, which takes values between $[0,1]$ and is often denoted by $\\beta$. This hyperparameter belongs to a technique called _momentum optimization_, which extends SGD to speed up training. Recall that in SGD, we update our weights and biases according to the following update rule:\n",
    "\n",
    "$$ \\theta \\to \\theta' = \\theta - \\eta \\nabla_\\theta L(\\theta) .$$\n",
    "\n",
    "One problem with this rule, is that if the gradients are small in some region, then the subsequent updates will also be small and training can be slow. To deal with this, momentum optimization uses an _exponentially weighted average_ to modify the update rule:\n",
    "\n",
    "\\begin{align}\n",
    "\\mathbf{m}'&= \\beta\\mathbf{m} + (1-\\beta)\\nabla_\\theta L(\\theta) \\\\\n",
    "\\theta' &= \\theta - \\eta\\mathbf{m}'\n",
    "\\end{align}\n",
    "\n",
    "Intuitively, the momentum vector $\\mathbf{m}$ is storing a running average of past gradient values, and this enables the update step to pick directions based on these averages. The result is that SGD with momentum allows us to traverse the loss landscape much faster (especially through plateaus). \n",
    "\n",
    "Let's now have a look at the activation statistics in the final `conv()` layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "86907a10-c8e8-4036-9288-006632d6db2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAADSCAYAAAC8To2hAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABboklEQVR4nO3dd3xb5fX48c+RZMt7O3svkhDIICSMUHYJe7Ws9gelUGgLpXvQFlroohtaoC1QyvhCgQJlBigQAgkjA5JA9iY78d5D4/n9ca/kK1nyiG3Zss/79corukP3PrJ9paPnnuc8YoxBKaWUUkop1ZqrtxuglFJKKaVUX6XBslJKKaWUUnFosKyUUkoppVQcGiwrpZRSSikVhwbLSimllFJKxaHBslJKKaWUUnFosKyUUgOEiPxcRP6vt9uhlFLJRINlpZTqBzQQVqrjROROEakQkfdFZIRj/RUi8pfebJvqezRYVkoppdSAISJzgKOAIcAS4Ef2+lzg+8BPe691LUTE04PHdvfUsfsjDZYHABHZISLfF5GPRaRORP4pIoNF5BURqRGRN0Qk3973GBF5T0QqRWS1iJzkOM7VIrLefs42Ebnese0kEdktIt8VkYMisk9Erk78q1Wq/xORH4rIHvta3CgiZwM/Bi4VkVoRWW3vN1ZE3rb3ex0o6tWGK9U3jAWWGGOagDeBcfb6XwG/N8ZUt/VkEXlIRO4RkZfta2upiIx3bJ8sIq+LSLl9fV7i2Ha2iKwUkWoR2SUiP3dsGyMiRkSuEZGdwMIY537RvsZD/4Ii8qUOnPchEfmbiCwQkTrgZBGZIiKL7M/7tSJy3iH8LAcEDZYHjouB04FJwLnAK1gfrsVYfwc3ichw4GXgl0AB8D3gGREpto9xEDgHyAGuBv4sIrMc5xgC5ALDgWuAe0JBuFKqe4jIYcCNwNHGmGzgDGAD8GvgSWNMljFmur3748CHWEHyL4CreqHJSvU1a4ETRCQdOBVYKyKzgcOMMY938BiXAbcB+cAWrEAbEckEXse69gbZ+90rIlPt59UBVwJ5wNnA10TkgqhjnwhMwbq2IxhjzrWv8Szg88B+4M0OnBfgCrud2cBS4EXgf/b+3wAes99fVBQNlgeOvxpjDhhj9gCLgaXGmJXGmEbgv8BM4IvAAmPMAmNM0BjzOrACOAvAGPOyMWarsbyNdZGd4DiHD7jdGOMzxiwAagG98JTqXgHAC0wVkRRjzA5jzNbonURkFHA0cIsxpskY8w7Wh6NSA5oxZg3wDPABMAr4HfAXrE6jm0TkHRF5TETy2jjMf40xy4wxfuAxYIa9/hxghzHmX8YYvzFmpX2uz9vnXmSM+cT+jP0Y+DdWcOz0c2NMnTGmId7JRWQS8DBwiTFmV3vntT1vjHnXGBO025sF3GGMaTbGLAReAi5v4zUPWBosDxwHHI8bYixnAaOBz9u3ZCpFpBKYBwwFEJEzReQD+xZPJVYQ7bytW2a/cYTU28dVSnUTY8wW4FvAz4GDIvKEiAyLseswoMIYU+dY92nPt1Cpvs8Y82djzHRjzKXAJcA7WDHRdVi9zeuxc5nj2O947PysGw3Mjfoc/QLWnVdEZK6IvCUiJSJSBXyV1ulRu9pqu51b/TzwU2PMko6cN8ZxhwG77MA55FOsO8MqigbLymkX8KgxJs/xL9MYc4eIeLG+pf4BGGyMyQMWANKL7VVqQDLGPG6MmYf1AWmA39r/O+0D8u3bsyGjEtREpZKCiAzGCpBvB6YBHxtjfMBy4MhDOOQu4O2oz9EsY8zX7O2PAy8AI40xucDfaf05Gn0tO9vrso/xljHmvk6cN/q4e4GR9vFCRgF7OvNiBwoNlpXT/wHnisgZIuIWkTR74N4IIBXr1m8J4BeRM4HP9mZjlRqIROQwETnF/gLbiHVnKIh1t2hM6MPPGPMpVhrVbSKSKiLzsMYrKKVa/Akr7aEe2A4cLSJZwEnAtkM43kvAJBH5fyKSYv87WkSm2NuzgXJjTKNYVTmu6OTxfwVkAt/s5HmjLcXqEf+Bve9JWO8PT3SyPQOCBssqzM57Oh9r4F8J1jfV7wMuY0wNcBPwFFCBdYG/0EtNVWog8wJ3AKVYt4IHATcD/7G3l4nIR/bjK4C5QDnwM+CRxDZVqb5LRE4B8owx/wUwxizDGuS+CzgZ6zrrFPuz8rNYA+z2Yl2jv8W6bgG+DtwuIjXArVifqZ1xOXAMUOGoiPGFDpw3up3NWMHxmVjvJfcCVxpjNnSyPQOCGBO3t18ppZRSSqkBTXuWlVJKKaWUikODZaWUUkoppeLQYFkppZRSSqk4NFhWSimllFIqDg2WlVJKKaWUisPT2w1oS1FRkRkzZkxvN0OpPuPDDz8sNcYU93Y7YtHrValIer0qlTzaul77dLA8ZswYVqxY0dvNUKrPEJE+O12xXq9KRdLrVank0db1qmkYSimllFJKxaHBslJKKaWUUnFosKyUUkr1USLyoIgcFJE1cbaLiPxFRLaIyMciMivRbVSqv9NgWSmllOq7HgLmt7H9TGCi/e864G8JaJNSA0qfHuDXHWqb/Pzj7a1cf+J4srz9/uUqpdSAtXF/Dat2VXDp0aN6uyndxhjzjoiMaWOX84FHjDEG+EBE8kRkqDFmX1fPvWF/Nf9asoOS2iYafQFG5KdjDIiA2+Xic0eN4KjR+V09jVJ9Xr+PHp9avou/LtxCsz/IzWdN6e3mKNXrRGQk8AgwGDDAfcaYu6L2EeAu4CygHviSMeajRLdVqc4456+L8QUMF88agcc9YG6cDgd2OZZ32+taBcsich1W7zOjRrX/heKJZbv4z4e78LhdBIKGbSVeRKxt+6oaeW9rKYu+dxISWqlUP9UtwbKIzMf6YHUDDxhj7oja7sX6cD4KKAMuNcbs6I5zt2dHWR0Ar6zZzw/nT8bl0otaDXh+4LvGmI9EJBv4UEReN8asc+zjvLU7F+vW7tzEN1WpjvMFDAA/ePpj/nTpjN5tTB9kjLkPuA9g9uzZpr3991Y2MGFQFi/fdAL+gCE91R3e9o+3t/KbVzZQ3eAnNyOl5xqtVB/Q5a/eIuIG7sH6cJ0KXC4iU6N2uwaoMMZMAP4M/Lar5+2oneX14f8XbylN1GmV6rOMMftCvcTGmBpgPVZPlFP41q4x5gMgT0SGJripSh2SZ1fu6e0mJNIeYKRjeYS9rkuMMazdW83owkxS3K6IQBkgJ90KkBt8ga6eSqk+rzvuU80BthhjthljmoEnsD5onc4HHrYfPw2cKl28bxMMGnaW1be73+YDtcw/fAhFWan8fdFWAsF2v0wrNWDYuZAzgaVRm+Ld2lWqy6obfdzy3Brm3/lOt70n1zf7I5ZLapq65bhJ4AXgSrsqxjFAVXfkK1c1+NhT2cCcMQUxt6elWOGDBsuJFQgaqhp8GGNo9AUIdvL6WbK5lH8u2c5TK3axalclTf6W3195XTOltU00NOvvNFp3pGHE+lCNvl0b3scY4xeRKqAQOOSu3o/3VHHBPe8ytiiT8cVZjC3KYHRhJuOKMxmZn0FGqps1e6vZU9nAtSeMZd7EIn763Bou+cf7/PHz0xlTlHmop1aqXxCRLOAZ4FvGmOpDPEanciCVArjyn8tYtasSgO//Z3W3pExU1Psilj/aWcEZhw/p8nF7m4j8GzgJKBKR3cDPgBQAY8zfgQVYYwu2YI0vuLo7zlvdYH35yM9Mjbk9PcXqaW7UYLnH3LtoC898uJtMr4ftJVZKaV2zH2d87PW4KMryUlbXRKMvyCNfnkNOegqTh2STluLm/a1lvLH+AO9sKmHzwdpW5/B6XGSnpVBZ34zfPrDbJYzIT2d0YSZltU0MyvYyJDeNZr9hV3k9eyob8Ka4GFeURXqqm8OH5XDhzOEMzkkDYF9VA9tL6zhscDaFWV4afQHe3lTC+n3V1DT6yc9IYWd5PVceO4Zpw3MJBg17qxp4ftVehuSk0eAL4HEJKW4XowszGFmQQV2Tn9GFmbhdQqMvQHldM1lpHrJSPeH0WmMMlfU+stI8pLhdGGMoq2smPcXNmj1VzB1XeMi/iz43wK+jH77D89K55ZypLN1Wxo6yOhZvLqHJH2y1X3G2l/OmD6Mwy0uW18Otz6/hzLsW85Ozp/CFuaN0YIIakEQkBStQfswY82yMXTp0a7ezOZBKAeFAGayUie4IlqN7w/ZVNnT5mH2BMebydrYb4IbuPu8Lq63LPSctdpjgtYPlZO1Zfn7VHp5YtosHrppNZhuVsoJBQ22zn5y01nnZS7eV8Wl5PR9sK6PJH2T6iFw+d9RIFm44yM6yOr547Ghy0lJIcbtYtauC/VVNrN1bxRVzR+FxuViypRSXWL2920rrWLWrkmvmjeWLx4ymusHH717dSJbXw7hiD/MmFlGU5SUvI4UFn+xjV3kD588Yxs7yenZXNHDK5EEs+GQ/Vz64DICCzFQmDMpi2fZyALLTPIwrzuRzR43gktkjqWn0s3pXJSt3VvDyJ/vxBw0TB2Vx/oxhVNb72FZaxyd7qvB6XJTXNbNmbzUpLsHlEo4ckUujL8Duinr2VTXy4uq93PHKBjwuoTjby76qxvDPqDjbSzBoBa3Rnlqxu1O/s1S3C6/H1eoLQ6qdJuSSli/NqW4XzYGWmDAtxcXSH59Gbvqh5dd3R7DckQ/V0D67RcQD5GIN9Gulox++xdlerpk3lmvmjQWsP+j91Y1sK6ljT2U9jb4g6SluTphURGGWF4ALZg5n7rgCfvD0x/z0uTVU1DXzjVMnHtKLVipZ2SlQ/wTWG2P+FGe3F4AbReQJrDtF3XJrVymwOjv22MHslKE53XLMfVXW8e6/cjY3PP4Rex0f2Krz3txwELB6GWMJ9ywn4S17YwzffGIVAPcv3sbFs0YwsiAjYp81e6q4/tEPw3+nnztqBL5AkBS3ixMnFbN2bzV/f3trxHNe/ngfv16wIbz8l4VbAOtn5fxSce+iyOc5/XPJdp5ftYdZo/JJS3Hx3s2ntArUv3P6JJr8QdJSIvPIS2ubuPetraS4hYfe2xEOlP/z1WM5OiqdpijLy9iiTC6YOZzbzp8Wtz0dsWx7OQs+2YfX42J3ZQPTR+QycXA2Ww/W8t7WMmqb/Nx0xFDOnT6MDDv3fe3eKl5be4BPdldR1eDjzGlDGFOUSWFWKkNz0zlQ3cjKnZUs31HOqIIMctNTqGuyguQGn5/c9FSy0zz4AkEafUEamv00+YOMLMigtLaJZr8VA+6tasDrcfOFuaPifvHriO4IlpcDE0VkLFZQfBlwRdQ+LwBXAe8DnwMW2t+Gu43LJQzLS2dYXnqb+w3NTefhq+fwrSdXceebm7noqBEMb+c5SvUzxwP/D/hERFbZ634MjIKevbWrFMDEwVmU1zVzzLgCdld0Tw/w//un1aOWmeqmOMvLyp0V3XLcgermM6dwxf0fMG14bsztoUCt0Z9cwXIgaPj2k6vCy3e+sZk739jMU9cfy5yxBSzccICnP9zN+1vLIlJ7nv5wd6vHXo8rfEd7/e3zeXL5Tn7+4jqOGJ7LmKJMXly9F2jpfR9TmME188ZysKaJv9qB9FXHjmbC4GzOOWIoM3/xOhfPGsELq/fwv3UHuGjW8Jg92iLSKlAGKwC+9VyrvsIXjxnNCb97i5MPK24VKHe3OWMLmDO29TlOPmwQ154wLuZzjhpdwFGj47drbFEmx4wr5GuM77Z2dkWXg2U7B/lG4DWs0nEPGmPWisjtwApjzAtYvViPisgWoBwroO41Lpfwg/mH8dLHe3ly+S6+c/qk3myOUglljFkCtJl/1FO3dpUCqGvyM2NkHoVZXjbur+nWYxsgPdXN8h0VNDQHWlVxUB0zZ2wBW359VtztXo81wK/R1zr9sS+745X1vLB6L7NH5/PPq47mj69v5JH3P+Wy+97n4lkj+I8jKP7p2VM4d/owirK8jP/xgvB6t0u467IZnDVtKCt3VTChOJv0VDdfOn4sR4zIY0JxFrkZKfz+c0dy0u8X8bWTxnPVcWMi2nHy5EEIMHNUy6QuW399Fm6XcM70odz/zjau/8yhB4ojCzLY+Mv5uDXVtFt0S86yMWYBVk+Uc92tjseNwOe741zdZUR+BrNHF/CXNzfzzIe7OX3qYCYPyeayOTpISSmlelJdU4BheSlkeT3UNvnbf0InFGd7OWvaEP6ycAtldU2MSM1o/0mq01LcVhCWqApT5XXN1Df7GZHf+vcZDBr+9d4OPtpZwZ8umY7XY+dTNwcQaekF9wWCPL50J2cfOZS7L5+JiHD7+dOYP20IX35oeThQzknz8LWTJkT0it56zlSe+Wg3makeHrz66PCMwNG9o84ZDdNS3Hzw41Njvp5Zo1rPfBhKeTn5sEGcfNigDv9s4gn9HFTX9bkBfon0o7Mmc8nf32dPZQMPvbcDgNljCpgwKKt3G6aUUv3MjNv/xxfnjuZ7ZxxGVYOPyUOzw8GyMYYL732PKUNz+M1FRxzS8UcXZjA4J41Jg7OZOsxKHahq8DFCZ2PuEW6X1bOcqGD52N+8SZM/yI47zm617e1NJfziJWtOpVmj8sNjmc76y2LcLuHWc6aycMNBe3BYgHOPHBoxuP+48UV8dMvpfP2xj1i0sYSbTp3YKn3gy/PG8mX7uGrgGTDzgcYya1Q+626fz/rb5/PQ1UcDcNqf3ubtTSWdrl2olFIqtromP5X1Pu5+awvGGEprmyjK8pLp9RA0UNPkZ9WuSv69bOchn6O8tpmp9mDBPHtGucqocnKq+4Ru7/sT9FkZq9pVyNaSlpJor66xxiEv3VbG9tI6thys5coHl/HQezv4xzvbyM9I4YSJxa2OkZHqYfIQ6++nu+92qOQ3oINlgFSPVXLk+AlFDLFrBF714DKeWL6rnWcqpZTqiNLalglC6psDNPmDFGSmkmWPTj/y5/8Lb29uIyiKp8kfoKbJT1GWVRN4RL41aPv9rTGLLqlu4A6nYfReznKzP8iG/dXstyufpKdYueqPLf2US+/7ILzfUaPzw2kT154wLm6puPNnDAPg1MmDe7jlKtkM+GA5JMXt4oMfn8rLN81jaG4af/zfRt7fWsbBai0/pJRSXVFa21Jjtcx+XJiZSpa3dU7lbS+u7fTxy+0argWZVpnQEfkZpHpc3P3Wlph3CRuaAzz6wad6B7ELPK5QsNx7bbh74Wbm37mYhRsPMq4ok/dvPoW0FBc/+e8aAP551Wy2/fosnvnacdx56YyIcrOxTBmaw447zuaIEbErgKiBS4PlKIcPy+XeL8yirK6Zy+//gK88sqK3m6SUUknNORV1aZ3Vy1yYlUpmausevkO5BR4KwAscs82Feqj3xJic5M9vbOKW59bwv3UHOn0uZXG7erdnudkfDE9qsa2kjoAx5GWk8rNzDwesHPZTpwwOz+522tTB3HLO1Jgl15Rqz4Ae4BfPzFH5TB+Zx+pdlazeXcW7W0o5fkJRbzdLKaWSkrO8WHm4Z9lLXXNkYDymMOOQBoyFZgcLpWEAfOWEsdy/eHurcwBU1lv7f/X/PgTgmnlj2VfVwJ8vnaEVBDoo1LOcqJzlaN9+chX7HXd+59i1hC+eNYIdZXVcNHNEr7RL9U/asxzH/VcexRPXHQPAFx5Y2u21QJVSqr9ZuOEA97y1pdX6RsfsZRV2oJqXkRIx9awIDM5J4431B/jNK+uJNW9Vsz/Irxes56OoCUfK7JxoZ8/yPHsQV11UT/W+qoZW0+z+c8l2Fnyyn7sXbuH8e96lIsbUvCqSy5XY0nEhr687wKSfvsLLn7RMKHrdZ8bxqwutKiqpHhc3nzmFw4ZkJ7Rdqn/TYDmOQdlpHDOukD9+fjoAj36wo3cbpJRSfdyXH1rB71/byHeeXMVb9nTJEBkshypUpKe6GVOYGV7/p0umU9Pop9EX5B9vb4uYPS1k+Y5y7ntnGxfd+17E+lDOcmGWN7wuNKCrtqnl3MYYjv3Nwrjt/+vCLazeVcnMX7xOaW0TvkCQ7z61ms0HWjpLnlq+S2cHpPd6lr/yyIqIQaD5GSn8+KwppHo0nFE9R9Mw2nHxUSNYuOEg/1t7gF+cPy2iNqNSSil4b0spz63aE15+duUenl25J1wTt9ER3IRSJjJSPWR6PXxh7iiG5KRx4cwRPL60pXTcrvL6iJ5iIOIO35aDtUwYlEWzP8gvX14PWJNJhISC5dIaq9fZGMPYmyPmzmrTOX9ZwsGaRoIGnvloN+tuP4N739rK3XbP+b1fmMVZRwzt8PH6G3cv9Sw73Xbe4cybqCmSqudpsNwBn5lUxMuf7GPTgVq9taOUUg47y+q54oGlbe7T5OhZDqVMpNsDrUK3zwHuv3I2M25/HYD91Y1MjzqOswTdrop6JgzKYuGGlkF6zs6MkQXpFGSm8vMX1rK3soEtjlq8AI98eQ5Hjshl3d5qirK9XPXgMvZVteTA7o+qhDT11tciln/36oYBHSx7EjwpSbQjhue2mkJaqZ6i9y06IJT7tnhzScT6irpm/L1ZN0cppXrZJ3uqYq4fU9gyLXFDc0uwXF7XjNfjCvdMOuVlpHLnpTOAlgoXTs51odq6vkDsYC0j1cN504dR0+Tnj69v4vlVeyO2nzCxiLyMVI6bUMSkwdm8cOO88LasOHV4nZ79+vHt7tOfhX59vTXAr7pRJ5xRiaPBcgcMz0tnfHEmizeXAlbPSHldMzN/8Xq4nqNSSvUF/3p3O399c/MhP393RX2H991ysJbvP706Yt2v7Z7iHWX11NgBTaO/JVgurWuOOykEEO6tve3FtTT7gxGl5EK9yWBNZQ1tz+wWmtEv5MaTJwDW4MLolLribC8LbjqBt79/Eit+eho3nDw+7nGBVikiA42I4HZJwkvHhWZnnD9tSELPqwY2DZY76ISJxSzdXsbzq/Zw1C/f4Ny/LgHgyRW7DmnGKaWU6gm3vbiOP76+6ZDuei3eXMK8377Fq2v2d2j/Hzy9mnpHrzHA0WPyw2W8Qr2/ztJxpTVNZKTGL8+Wavc6N/mDnPvXJUz7mZX+EAwaPtldxZyxBXhcQrUdLIcqV3z9pNbB7QUzh0csjyxIZ8FNJ/C/b38m5rmnDsthdGEmaSlurj5+LKkeF09dfyxPXX8sYH0ROH/GMN790SnxfygDiNslCe9Z9vmDzBlTwA/OmJzQ86qBTYPlDjphYhGNviDffGIVEFno/v1tOqWqUqpv2V1hvUd9WlbHMb9+M+bkHNG2ldQB8MTyne3saYmVATG2KJPvfHYSAAftwXXOahgHqhvJSUtp/USHe78wC4CNjioUO8rqqGnyM2NEHrnpKeHb8GV1zaS4he+fcVir46R6XOy442x+ON8KrNwuF1OH5TAoO63d11aU5WXTL89kztgC5owtYMcdZ3PF3FHcddlMhuelt/v8gcDjkoTPgljXHGBIblrMNB6leooGyx10zLjCuNt2lNaFHy/bXs43/r1Sc5mVUglT0+hj2s9e462NLeXalu8op67JzxPLd7G/upHnVu5p4wiWUC/x3g4E1s3+IKt3VQJw9fFj+MLcUXz7tEl43C6Ks60SbiXhYLnl/dAfNGSntZ0TPNFOtQhp9AXYUWa9z04YnEVuRgpbDloD9srrmijITG2zUtFXThjLby8+ggtmDGv3damOS1TPcnTN7cwY06Qr1ZM0WO6gTK+Hn549hfyMlPAtxlCdyZ+9sJb3tpRS2+Tnkn+8z4ur9/LRzspebK1Sqr/zB4L83wef0uQP8P7WMmqb/Pzj7a3h7d9/+mPOvGsxbul4ia/NB62eXFcHSmQu214OwLdOm8jPzj2cX114BN88bSIAg+xg+WCNnYbhD5DtyFNeaj83nlEFGRHLP3j6Y/bYPeUj8tI5Zlwhq3dZAwvL63wUZHpbHcPJ43Zx6dGj8Lj1I687eVyCP84Ay+4U/bebnqKFvFRi6TtHJ1x7wjg+uuX08MCOX1/UUvLogSXbec2R5xeaTlUppbpDaW1TRDrDc6v28tPn1nDf29vCKRbDotIDdpbXd2qmta12b63zPPGEAuHzZwxvtS3L6yEtxdXSs9wcYFBOS0CbltL2R090UPvm+gO8Yw+wLsryMiQnjQZfAF8gaPcst53WoXqG2yUEY8y02N2ie6+1Z1klmgbLnSQiMd/ojTERty6bNQ1DKdWNZv/yDb7yyIrwcqhe8d6qBmoarYoRodrFTqHUzlfX7G83b7naPo4zbSKeUCAcSrlwEhEGZae15Cz7A+RltFSPePwrx7R7/FMmDwLA63ExeWgOr6+z6im7XBKefKSm0U95XXO7PcuqZ4gIiUhZbtWz3MYAUaV6ggbLh+DIEXmAdavxyeusN/2tJXX88fVN4X20QoZSqruExkCEyleCNdAJwOtxh8urpcRIMwiVVtt4oIYz73ynzfOEgm5nqbd4yux6yZlxApfibG84oK5tCpCR6g4H80Ny2h9gd/cVM1l5y+mcMLGYTfZAv1Auc0661ZO8s7yeHWX1FGUN7DJuvcUlrfOJe0IgOmc5VdMwVGLpX9wh+NJxY5g2PJc5Y63c5cvnjOTfy3ZF7KPBslKqu4SCWCefHUA3+YPhgNgfo+atc0KQ6hjHgZaAJ1wXuQNpGOV1zRS2MbBuULaXzXZax4GqRiYNKmLpT05l0caSVukisWSkeshItaawDr3+n593OAC5drC8xJ4oau7Y+AOwVc9xiSRkBr9AVF704By9k6ASS4PlQ+BySThQBsiOUQZJ0zCUUt0lVCbNWS3LZwfIzf5geHus9ImOBL43/nsleysbaPIHEbGOY4xps8JERV0z+W1MzJGTlkJNo49A0HCwppEhuWnkpKVw3vTOVaQI9SIDjMzPiFgXmj3wsCHZnTqm6h6uBKVhROcsD83V0n0qsTRY7gbZMWaj0p5lpVR3iZWTHOpZbg4E2W6Xr4w1m130pCG+QJAUt4sPtpUxZUgOTf4AL3+8L7y9ICOVsrpmmvxB0mLkQIeU1ze3OYtdptdDXVOA2iY/QdPSG9xZR47IDT8emmelb4TqNL+21spj1jSM3uFyJSYNI3oQ4aH+LSl1qDRnuRs4O19mjsoD2p6CVSml4jHGtOoNDt2pcpZ0a7ZvTTf5ApTbs9g1xehFrmuKTL2oqG+moTnAZfd9wHWPrggPwgsJ9do2xeilfvrD3ayxe3Mr6prJz4gfpGZ53dQ1+8Pnb2uK67acfJg10O+yo0eGc7Jz0iOPFevunup5LpFW+cQ9IbpnOdWjoYtKLP2L6wZeT0vvy+8/dyQAixyTAyilVEf9/rWNTL7l1Yhc41BeqLOHLTTor8ERLDfG+JJeZU8LHfLo+5/yv3VWmcsN+2vCgwNDQpUmogf5GWP43n9Wc85flwDYVSjiB8vpqR6MaZmF71CD5fzMVN6/+RR+ecG0lnWOIP2YcQWxnqYSwJ2oahgBDZZV79K/uG5wyeyR4cehno/lOyrYUVrHql2VER96SinVloff2wFAkyNYDU384AxMfDEqZDTGeK8J5TOH/HXhFr75xCrAer+qjRr0F+pZju7drnEE1b5AkOpGf5s9y3sq6wG4+l/LAeJWzeiIobnpEbWX01LcPGFXIgpN0a0ST6R1ikRPiO691mBZJZr+xXWDtNSWH6Nzvvpl28u54J53+eEzH/dGs5RSSShUEs5ZZSAUkDgDE1+MmdPqmq2A9q7LZpBqB5fVDbErYIBVw7imKTKYDuUDRw8WPFjdkq5x0u8XAbQ5Gcg188ZFLB9qz3I8s0fnIwI3nTqxW4/bF4nIfBHZKCJbRORHMbaPFpE3ReRjEVkkIiMS0S6XCMFEVMOIqvKSqjMxqgTTv7hu4LxwnXVOd5RZPR4rd1UkvE1KqeTmDJZDOZvOsCRWxZ1QfvDkITnccPIEAPZXN3L2EUN59uvHtdo/xS3UNUX2IIfygaN7lp3pGqHJTfLa6FkeW5TJredMDS9ndXOw7HG72P6bs/niMaO79bh9jYi4gXuAM4GpwOUiMjVqtz8AjxhjjgRuB36TiLb11gx+GiyrRNO/uG7gLK/kcfQsby2xaowm4L1EqQ4TkQdF5KCIrImz/SQRqRKRVfa/WxPdRgU+R4AQ6lkzMXKWQ1LcQq0d+Gakupk8tKWcWlqKO5yLHPkcV6vByC09y5HBcn1z6x7qWHWdnWaNzg8/zsvQQXiHaA6wxRizzRjTDDwBnB+1z1Rgof34rRjbe0SiZvCLvsvhcsUvaahUT9BguZt53C4euvpooKWsUejz7b8rd7N+X3VvNU2pkIeA+e3ss9gYM8P+d3sC2qSiOAc1xc5ZbllIdbvwuFzhnuX0VDdnHD6Eo+xgNT3VxZjCzFbnaA4EW5W5DOcsR62PVa953oTiNl/DuOKWc7aV36zaNBxwznq1217ntBq4yH58IZAtIq1mahGR60RkhYisKCkp6XLDEjWDX6wvakolkgbL3SzFLZxklzpyCgYN335yNWfetbgXWqVUC2PMO0B5b7dDtc3n6LWNVQ3DmYbh9bhwiVUZA1rqMRdnecPLnhi3rj8tqw8/JyTeAD9nveaiLC/rb59PcXbbM6nlOEq6ZXRhgJ9q1/eAE0VkJXAisAdo9e3GGHOfMWa2MWZ2cXHbX3Q6IhEz+FXWN4cHiSrVW3RSkm7mjnF7KGgMpbVNMfZWqs86VkRWA3uB7xlj1sbaSUSuA64DGDVqVAKb1/85g5BQNQAT0bMcjNjudqSDhYLl0OQN6W1MLlLd4MPjknBeaLh0nC9AdaOPyjofowozIqr6DM7xkt7B4HfxD05m4/6aNmcDVG3aA4x0LI+w14UZY/Zi9yyLSBZwsTGmsqcb5nL1fBrG0x/uDqcKDc7xcqBaP0tV4nWpZ1lECkTkdRHZbP+fH2e/gCP/8YWunLOvS3FZP9IFN50QXucLBCnRYFklj4+A0caY6cBfgefi7djdPVUD3ce7K8OPI4LhGBGJc3uzPxjO4/R6XOHHaSnW+1GaHdiOLrSmiz5ufCFnHzkUsCYpcZbiCqVLNPmCXPaPD/jM79/CGBPRA52Z2vF+lpEFGZw2dXCH91etLAcmishYEUkFLgMiPkdFpEhEQr/Em4EHE9EwVwJKxznTjf5z/XFs/tWZPXo+pWLpahrGj4A3jTETgTft5VgaHPmP53XxnH1a6ENqwqCs8LpGX5A9FQ3h5U12kX6l+iJjTLUxptZ+vABIEZGiXm7WgOCsmRxRDSNGmTjnOn/QELqp5Ux3CPXmhqoHvHzTCaz46Wk8/pVjOHGS9eXm+VV7I3qsQxONNPoDrLPHWOwqb4joWXZpAl/CGGP8wI3Aa8B64CljzFoRuV1EQp+nJwEbRWQTMBj4VSLa5pKer4ZRXtfS0ZSd5omoOKVUonT1r+584GH78cPABV08Xr/h7Klp9AW47tEPw8uf/fM7fLK7qjeapVS7RGSI2FGWiMzBep8o691WDQzOTAV/RDWM1gFJcyAYMYNeaCrsDEevbyiv2Wu/H2V5PRQ58phDnL3G+aFg2bFuzd6qiJzl6HJzqmcZYxYYYyYZY8YbY35lr7vVGPOC/fhpY8xEe59rjTEJuZXpFqGdgihdVl5n1QG/ZPaI8N+mUonW1WB5sDFmn/14P9Y32ljS7BG4H4jIBV08Z9KJrhEJ2ruseo+I/Bt4HzhMRHaLyDUi8lUR+aq9y+eANXbO8l+Ay0wihrwrhJZoObrnOJovEKTQGSxHpV4A4d7m9BhpE/HymPPtEm+ltc3hdWv2VEUEzz09qEslh0TM4FdR38y04Tn87nPTe/Q8SrWl3cQzEXkDGBJj00+cC8YYIyLxrprRxpg9IjIOWCginxhjtsY534AYMHTr82vYV9XAjaf0/9mnVN9ijLm8ne13A3cnqDkDgi8Q7PTtY2cN4+jpfgF8fsOYwjQ2H7TqubekYbS8rX/n9MNwiXCOnZ/sFK86RXqKG49L2FfVGF636UAtQ3PTwss63bCCxFTDKK9r1rKDqte1+45njDnNGDMtxr/ngQMiMhTA/v9gnGPssf/fBiwCZrZxvqQcMPTGdz7D374wq8P71zUH+MP/NvVgi5RSfcGKHeVM/MkrvL+1c5kszp7lQIzZ+vzBIMPz0sPLoTQMZ49xQWYqt58/jbQYvciHDcmOeH5o8J+IkJ7q5kB1S7Bc0+iLSNWYOSqvU69F9U9ul8T8ItedKuqbI9KNlOoNXe0eeAG4yn58FfB89A4iki8iXvtxEXA8sK6L5+1zJgzK5swjWvfeKKUGtlCQvGRL+5NAOHOWY0137dTsD4Zn5rv+xHEtwXIHS7oVZnn537c/E15+7uvH89q3rOXiLC/bSuoAa3Bggy9AQ3OA8cWZPHndMdx85pQOnUP1b4lIw9CeZdUXdDVYvgM4XUQ2A6fZy4jIbBF5wN5nCrDCzn98C7jDGNPvgmWllIollEvc2bvVVz+0nDV7rIHAgRiD/XwBQ6rHxcc/P4Obz5wSrlDRVk3laM5UjPzMVA4bYk2RPTgnLVwbvjjbS31zgLpmPxmpHuaOK9Q0DAWEqmH03PF9gSA1jX7tWVa9rkuTkhhjyoBTY6xfAVxrP34POKIr51FKqWQV6i3uSA9c9LQdv3llPY9de0xEz/Luinr+9PomGnwBPI5JkFqqYXQ8WBYRbj//cI4YnhuxfogjP7k420tJTRNltc0UZmnQolq4xJqdtqdU1FuDTLUKhuptOoOfUkr1oFAQW9KBmceiJ7nzBwxn3rWY9Xa9Y4DbXlzHwg3W8BDnoEF3J9MwQq48dkyrdXsrW+rCjy7M4NOyOoLGhHuelQIrZ7kn0zDufcuqAxCq0KJUb9F7aX3Alx9azqPv7+jtZiilekCo8/fZlXva3jGGoDERgTIQDpQhsipFKNDuTBpGPM6e5sE5adQ3Byira44oVaeU9HAaxkPv7QAip3lXqjdosNzLmvwBFm44yC3Pr6W8rrn9Jyilkoq0Sq7ouFgD+5y8EcFy59Mw4vnhmZPDj9NT3DT5gzT7g2R59WakauES6MkS7CPyrWotZxweq3qtUomjwXIv23qwLvx41i9e59qHl/dia5RS3S06taIz2qth6wyWfXZ5ubRuCJZT3C7uumwGz99wfETwnanBsnJwu3q2znKTP8jlc0bqgFLV6/QvMIG+ftL4Vut2ltdHLL+xPmapaqVUknK7Oh4t+wKRgcfHu6va3N/raQlkfX4rWM7ohjQMgPNnDGf6yLyIYFl7lpWTlYbRvcGyMYYHl2xnzI9epqSmiTwtG6f6AA2We9B/vnost513eHj56DEFrfbZXVHfap1Sqv/oTCzR2V46r2Nqa5/93IwYU1t3hXOqbO1ZVk4ukW7PJ35/Wxm3v9RSXTYnTQf3qd6nwXIPOnpMAVcdNya8nBdjRO/G/TUJbJFSKtGc01b7YszE59TomCWvI3oqDcMpMg2je4+tkptbYk/F3hVV9b6I5aw0/YKmep8GywnwxWNGAcScheg/H+5OdHOUUgnkTK2ob24Jhp9cvpNHP/g0Yt+6Jn+n8jOd+4amx+6uNIyQdE3DUHG4eiANY29VY8Ty2TozruoD9J0vAW4/bxo/PXtqRE3UthhjwiPblVLJzR8RLPvJTU+hutHHD5/5BIAvzBkVnuWvpsnaXlLTfk1miMpZtnuWO1tnuT3O2+CahqGcRIRg2zdLOm1PhVXje1C2l7e+d5L+zak+QXuWE8DlEtJS3LhdwlPXHxt3vyJ7dqwmfze/+yileo0z9aKuyepZXrGjPLyuwZF6UdfkJy+94zmazjSMUJm57g6Wc9NbghXtWVZOLunYzJRteW3tftbutQayXvvwch58dzuTh2Sz7CenaaCs+gwNlhNsztiCcO3IOVED/o4dXwRAQ3Pn8haVUn2Xz9H1Vt/sByIH/UUGywGyO5Gjme3o9Q2EB/h1c89yuvYsq9i6Ywa/6x/9kLP/soTlO8rD1aBuOHlCdzRPqW6jwXIvCE1/GwqaAW49ZyrHjS8EoNEfP1h+9qPdrNxZ0bMNVEodst0V9WwrqQ0vO9MwQj3Lzjxm55fj2iZ/pwLS3Bi90N0xg59TZBqGDvBTLbo6g59zQOvn//5++PFnDx/clWYp1e00WO4Ft5wzlYtnjWDuuJae5S/PGxv+kLvluTXUNvkjnmOMYeGGA3znqdVceO97CW2vUqrj5v32LU7549vhZWcaRqhn2bmuMSoNozOpDjGD5W7uWU5zBN/OHGmlujqD38Hq1rn5131mnP6dqT5H76n1gtOnDub0qYN5be3+iPVpds3UN9YfZPn2ck6ePIhPy+rYsL+G9BQ3X35oRW80VynVBc5e5LrmUM9yS7Bc2dBSKqujwfL0kXms3lUZfs9w6u6eZYAlPzy524+pkl9XZ/Cr9/lbrRuam9aVJinVIzRY7kXRA3mcPTihElMn/WERxsAVc0cltG1Kqa4JBg3r91fz72U7w+vqm/w0+4O8sHpveN2Nj3/E0h+fBljVMDK9HsYVZVJS00SN4w6T1+OiyR9k8pBsHrt2LgeqG2NWzenuSUkARuRndPsxVfJzdTkNo/Vg9tGF+rem+h4NlnvRiILIN4XIYDlyINDjS3eilEoeNY1+/rMiso56ZYOP215cy6KNJeF1B+xb0caYcM/ywu+dxKMffMotz60JB8lpKW7+fd0xDMlJI8vrIas4K+Z5OzO9tlJdIV2shtEUYxKeUQWZXWmSUj1Cg+VeNCw3jclDsrlo1nAg8vZpQydn8lJK9S2N/kCrCUbueGVDq/0unGld/yW1TQQNFNolJEPflEMDgoNBw6xR+XHP99fLZ7Lgk33d0XSlOsQtQrALXcuNMcqkjixIj7GnUr1LB/j1IhHh1W99hus+Mx6IHJgTKswei0d7jpTqEw5UN7KjtC68HLojBNbAPW87s/ENz0sPV8PYsK8GgMlDcgCYP20oYwozWlKw2rnsz50+jL998ajOvgSlDpnL1X3VMPIzUnjs2rk6uE/1SRos9yFpjjeJf7yzjbqm1oMfAAoyW0+brZRKvLm/fpOT/rAIgCWbS5l662vhbY2+IKntzNqZk55CvS/ArvJ6XvrYymMOlZQszvay6PsnM2+iVX+9JwbuKdUVXU7DcPQsjy3K5PgJRd3RLKW6naZh9CFpqZEfrKW1sae81QEQSvU9yx2z8oHdsxyjWoVTbrqHbSW1nPC7t8LrBuV4I/bJsINkzUVWfY1LhK7MSeLsWU7TL4OqD9Oe5T4k+s2iot7Xap9Zo/L0NpVSCeQLBPn721sjPthjia55vKOsjqc/bBngN3NUXqvnZHk9lNREfimOvr7H2QP5mmPkdyrVm9wiBLoQLTsn5InO71eqL9G/zj4k+jbrTf9eGbE8aXBWl+taKqU65/lVe7njlQ3c9ebmNveL/rD/5hOr2HSgZSa/uy6d2eo5uempEbeix8S4a1SUlcr1nxnH/VfN7mzTlepRri6mYTgn32ovZUmp3qR/nX1IitvFA1fOZtH3TgJgV0V9xPYnrzvWCpa7ct9LKdUpmfbA282OwDfaA4u38exHu+Nu//Ol0xlVmEF+RmTvc/TybedPa/VcEeHms6a0WQlDqd4gdhrGoc7iV93YcvdUe5ZVX6Z/nX3MaVMHM6Yoky8eM4qCjMiBfPmZqdqzrLpMRB4UkYMisibOdhGRv4jIFhH5WERmJbqNfUmmPaNeVUNz3H1++fJ6PtpZGXf7Z6cOASD6ys13DNZ95MtzOHFS8SG3U/VfIjJfRDba1+SPYmwfJSJvichK+5o9KxHtCuXRH+pHUm1jS8+yMyVDqb5Gg+U+Kj3FTVld6w9nt8ulwbLqqoeA+W1sPxOYaP+7DvhbAtrUZ4Wut67c0PG4raDigStnM3t0Sw/xjJF54cfas6ZiERE3cA/WdTkVuFxEpkbt9lPgKWPMTOAy4N5EtC005vRQUzFeX3eA0CSUzmtBqb5G3537qL2VjTHXuwUNllWXGGPeAcrb2OV84BFj+QDIE5GhiWld3+PvhustxWW91c4eU8DTXzsuvH7ioJZZ+DRYVnHMAbYYY7YZY5qBJ7CuUScD5NiPc4G9JEBouvVDCZaXbivjYE0TxsB7PzqFr500vrubp1S30XfnPqqsLnbZuI70LJfXNeML6Mh5dciGA7scy7vtdQNSIGhdS10JmV1xyr5lpbVU79QBTiqOjlyPPwe+KCK7gQXAN2IdSESuE5EVIrKipKQk1i6dEkrD6Eis3OgLsGx7y3f0J5ZbLyk9xc2wvHQ8+vev+jCts9xHSZzputyutnuWjTHM+sXrnD51MPdfqaPnVc8SkeuwUjUYNWpUL7emZ/jDaRjdd0fn7188ioxUd0QFHO1ZVl1wOfCQMeaPInIs8KiITDPGRPSaGGPuA+4DmD17dpf/oEPfATtyt/PuhVu4+60tTB2aw9dPHs9/V+4BYOH3TuxqM5Tqcfru3EfNGVsQsRyata+9ahiNPuu98fV1B3qucaq/2wOMdCyPsNe1Yoy5zxgz2xgzu7i4fw5OC+csR63vSvA8f9oQPjOpOHwbG7RnWcXVkevxGuApAGPM+0Aa0OPT4bk6kYYRqie+bl81/162M7x+aG56zzROqW6k78591E2nTgw/Pnf6MJ6x8xyj0zBW7qzguZV7mPSTV3h3SymVbYzYV6qDXgCutKtiHANUGWP29XajesMDi7fxzSdWAdat5jV7qnjlE+tH4ayPPL44M+4xfjh/cpvnOHZcIekp7lYz9yllWw5MFJGxIpKKNYDvhah9dgKnAojIFKxguet5Fu1oyVluez9jDMXZLX/f20rqAPjj56f3WNuU6k6ahtFHOae2veXsKQzKSQOsb/DbS+to9AVIS3Fz4b3vhff7vw8+5ZunTWx1LKWcROTfwElAkZ3j+DMgBcAY83esnMezgC1APXB177S09/3y5fXhxwY4569LANhxx9kRwXJTG7PrffXEcW2e419XH01Dc4CMVH07Vq0ZY/wiciPwGuAGHjTGrBWR24EVxpgXgO8C94vIt7H+VL9kujNvKA53qBpGG9Hy8h3lfOWRFRw2ODu8bl9VI8XZXi4+akRPN1GpbtGlnmUR+byIrBWRoIjETZBtr0akalthVss38tBtrxdXtx7sPDQ3nSWbSxPWLpWcjDGXG2OGGmNSjDEjjDH/NMb83Q6Usatg3GCMGW+MOcIYs6K329wXOGOPhuYATf6WurCNvmBEL9lt5x0efuxMtYglLcUdUW9ZqWjGmAXGmEn2Nfkre92tdqCMMWadMeZ4Y8x0Y8wMY8z/EtEul6v9NIx/vL2NynofS7eXMyQnLdzDHD2PgFJ9WVfTMNYAFwHvxNuhgzUiVQynTRkERPYy33TKBABqGv34oype+IPBiJ4wpVTnVTX4OOPP77DpQE3E+gPVLeUcD9Y00uRz9Cz7Alx81AhOnWxds4PtO0FK9WftpWEEgoal28vCy43+AENzrWujKFuDZZU8uhQsG2PWG2M2trNbR2pEqhj+8f9ms/GXkXNHDM+3BkM0B4Kc+qe3I7btrWxIWNuU6q+WbC5l44Ea7npzc8T6A9Ut5RwbfcGI1IspQ60St3l2b1ldkx+l+ju3hErHxY6W91Y2UNPo59unTWJQtpcfzp9MdYM1xbXOVqmSSSKS5GLViJwbb+eBUIqqo9wuwe1yR6xLsUfM1zT6+LSsHrBu+T783g5KaiMH931aVsfowvgDj5RSh+aMO1tupl07byxfOn4MAD89ewoAJ08exEvfmKfpFapfC5eOixMs766wOnBmj8kPj6e5+dlPADjpsEE930Clukm7Pcsi8oaIrInxr0d6hwdCKaqu8LgEEdhRWh9eV5TlBYHVuyoj9j3x94sS2zil+pMODo86YVIxI/IzAMjPTOWPl0ynIDOVacNzGZ6nZbFU/+VqJw2jptHqRc5NT2m1bVRBRo+1S6nu1m7PsjHmtC6eo8M1W1X7RIRUt4ttpVbpnSE5acwekx8uxQNwwsQiFtsD/Wqb/GR5dZS9Up318icdq5aX5XW3v5NS/VB4gF+caLnBZw2CzUhtuUae+dpxvLellLQUvW5U8khEneWO1IhUnZDqcbGtpBaAZ79+XKvBRF+eNzb8uLxW6y4r1RntFK9oJVO/jKoBKpSGEa8aRl2TFSw7r5GjRufzjVO1xKlKLl0tHXehXaf1WOBlEXnNXj9MRBaAVSMSCNWIXA88ZYxZ27VmD2xej4smfxCPS2KOuh+Z33J7q6659UAjXyDI/9bu79bpe5VKdh/trGDMj15m3d7quPtkp7UOjDO1PrIaoNpLw6i3P3/SU7UXWSW3rlbD+K9dp9VrjBlsjDnDXr/XGHOWY79WNSLVoQtNizssLz2irFxIQWYqD37JKntdHyNY/s2CDVz36Id8+GlFzzZUqSTy2tr9ALy9Kf7EZxfPaj2JgqY5qYFK2ulZrm+20zA05UIlOZ3uOgmleKxfW7zBQ7npKeEBFbVNgVbb39tq5TM3+uLPOqbUgGN/3sf4/hkWKzDWNAw1ULnbyVmua/bj9bjwuDXUUMlN/4KTUKhneUR+7GDZ7ZLw1Ln1jnqvr63dz7ibXw5/2w+NVFZKtfSOtZWcFJ2G8e6PTiHVo2+jamBqNw2jKRAxuE+pZKXv8kkoNOHBiPz4pXfy7ckRSmpbJlL4+9tbCRrYWW6Vnatq0GBZqZDQB34g3ic/kBUVLKdqj5kawNob4FffHAh33CiVzPSdPgntrbKm3T1mXEF43dlHDI3YZ3COl/yMFG59fi1f+tcyAMZETVBSq7OMKRVmOhAsF2RETjLiTdG3UDVwtfQsxwuW/WRqaUXVD+g7fRIb5KiEcc8XZgEt3/RFhDPtAHrRRmvAUnQvmC+g1TCUCgl94Mf74Adazcjn1RQMNYCFg+U4w1/qmgOka8+y6gf0nT6JpUX1ai398ams+Onp4eUrjx0dsb2yIbLmsi8QZOm2sriDM5QaiNrqWY6eSEHTMNRA5rL//OPXWfaTqTnLqh/Qd/ok5vVEvgkNzkmjwNHzNXlIDsPz0pk9Op//rNhFRV1kjvJjSz/l0vs+YMGafZzyx0X85L+fJKTdSvVFoQ/8tqrEOHuSr503FunsDCZK9SNtpWE0+gKs2VPF2KLMVtuUSjZ6fySJRfcsxzK6MIP3tpaxIkZN5QPV1uC/2kY/20rq2FZSx68uPKLb26lUMgh94O+pbIi7jzNY/snZU3q8TUr1ZW0Fy7sr6mnyB5kztqDVNqWSjfYsJ7HonuVYUjpwmzjRA/0afQEuuvddFm08mNDzKtWWjmQjOa8n7VVWA11bpePe3mTV82+rapNSyUKD5SQWa/a+aNHB8vSRea322V5a111N6pDNB2r5aGclX/rX8oSeV6mQqgYfH++ujFjXkdnf3S7hx2dNZkaM60ipgSZcOi5GtLx2bxUZqW5m6rWi+gENlvu5VE9kQD0o29tqn20liQ2WPy23zqeTOajecuWDyzjv7ncxERFy+9Gy1+Pius+M57kbju+5ximVJFyu+D3L5XXNTBiUFd5HqWSm0UoSeuzauXzz1Ikd2je6ZznPngbbaVtpbczn/r9/LuXuhZs738B2hCZD8Qd0um3VO1bvqgTA7/iUj1f+KuTvXzyKwqzWXzaVGqjaylkuq22OGHCuVDLTYDkJHT+hiG+fPqlD+0YHy9lprYPl0EA/iCybtXhzKX/436ZDbGV8NY1WjnTQxL59p1Si+Bxf2Nqqr+x2CfOnDUlEk5RKGm3N4Le/ujHmnUylkpEGy/1cdLB8+LCcNvevb+75wX41jS0l7Mrrm9vYU6me5ZyYx9/GF7c7LtIqMUpFi5eG0egLUFLTxEgd3Kf6CQ2W+7kUd2S+2PhBWWz79VmML45d+7KhOdDjbQr1LAMcqG7s8fMpFc+SzaXhuxvxguX/fPVYPj97ZCKbpVRSaJnBL/LaeW+rVQljajudM0olCw2W+7lRBZHf7HPTU3C5JG5JufoEB8t7KzVYVr3nhsc/4u63tvDi6r1xc+g9OkBJqZjipWHsrrBqlceqvqRUMtJJSfq58YOyIpZz0qxfeShYnju2gKXby8PbQ8Gy6UgdrUNU0+hncI6XA9VN7K6o77HzKBVLoy/yC+GfXrfy8oviDN7rSK1ypQaieHWWq+qtVLvcGAPKlUpG+inQz6U6PuhPmFhEXoY1Otljp2fkpKew9rYzuPPSGQA0+Kxe30APDryrafSFe7wr633t7K1U94oOlkNqm2L/LXaknrlSA1G8ahhLtlhpGPpFU/UX+pfczznfrB69Zm74gz/FZa33elxkej0My0sHWnqWnQOffN1U4u2XL63jvS2l1DT6yU1PIS3FRUOcwEWp7lTb5A+nWTT5Y/89C7GDYv3AVyo2+2MkIme5vtkfcbdSqf5APwX6uegBfiGhnuW0FGvK7IxU6/9wsOwoOlvb2LpCxpLNpazcWdHhdtQ1+XlgyXaueGApNU0+stNSMIZOHUOpQ2GMYdrPXuP7T38MQJMvdrDs/OJ285mTw49D14ZSKlKsNIz1+6oB+PZpHStvqlQy0GC5n4s3S57H3dKzDJBuBwShW9Q+R+9bTYxg+Yv/XMqF977X4XZ8WtaSm1zT6CfL66HJH2T5jooezY9WsYnIfBHZKCJbRORHMbZ/SURKRGSV/e/a3mhnd6hpsv5+/7tyDwCN/vbvZpw7fVj4sQbLqrd14Hr9s+Na3SQilYloV6w0jGXbrQ6QC2YOi/kcpZKRDvDr51Lj3EJOtXuWvZ44PcuONIzqxq7nFVc66ilX1vvITmv50/MFTKtpuVXPERE3cA9wOrAbWC4iLxhj1kXt+qQx5saEN7CbHaiyKq6kelyU1jZxy3Nr2n2O80tm6O6LUr2hI9erMebbjv2/AcxMRNtiVcP4+9tbyU1PaVWJSalkpj3L/Vy8fEtPKGc5xfo/I8UKXluC5Zae5baC5dfXHWBPZUO77dhXFVkiznnM7sqJVh02B9hijNlmjGkGngDO7+U29QhjDJX29OrN/iA/e35th/IpncGyN87dGaUSpLPX6+XAvxPRsOie5V3l9VQ1+Dhv+jBEtANE9R/6KdDPxUvDCA30S7N7lkNpGA32DH7NgbbTMEK+8sgKvvPkqjbb8O6WUr77n9UR6w5WN3HhzOHWueIMuFI9Zjiwy7G8214X7WIR+VhEnhaRmLNyiMh1IrJCRFaUlJT0RFsP2atr9jH25gVsL6kLr3v5k30deq7zjox+6Kte1tHrFREZDYwFFiagXeHPkdAQlxN+9xYQvwyjUslKg+V+Ll7Pcqg0XKhnOdXjwuOSmD3LoWB5y8EaPt5d2epYWx3BSCyhAR8AL980j6NG5/P9Mw5j9pj8VudSfcaLwBhjzJHA68DDsXYyxtxnjJltjJldXFyc0Aa25843NgOwdm9Vp5/r9bj0NrJKRpcBTxtjYibmd/eX29D3yIAxHHTMxnrRrJixvFJJS4Plfi5eznLAvm3mvMWcnurmmY9288LqvfidOcv2bezT/vQO5939Lvcu2hJxrPrmlp7nYNDw6pp9EaWEsrwt+cmjCjJ45mvHMXFwdjiQj1fKS/WYPYCzp3iEvS7MGFNmjGmyFx8AjkpQ27pNrT2w72BNUzt7Wm46ZUL4sYjw0k3zWPyDk3ukbUp1QrvXq8NltJGC0d1fbkNpGMYY3t5kBd8LbjqBkfpFU/UzGiz3c6Ge42jhnmVPy+CljFQ3B6qbuOnfK/lgW1l4fXQaxu9e3Rix7LxJ/fSHu/nq/33EuB8vCAfMzudnpLYEzqFAXXuWE245MFFExopIKtYH7AvOHURkqGPxPGB9AtvXLULpPdvaufMB1t/idz57WMS6nLQU/dBXfUG71yuAiEwG8oH3E9UwZ+m4LSW1pLpdTBmanajTK5UwGiz3c/FG8oeC5TRHMO0MZFfvbrl1XdNONYxQ3lpVg49lO1oGT20rrYt4/spbTo+YDS3Us9yswXJCGWP8wI3Aa1hB8FPGmLUicruInGfvdpOIrBWR1cBNwJd6p7WHxhgT7lneeKCm/f17ukFKHaIOXq9gBdFPmATW4gxPSmIMNY1+stM8muOv+iUtHTdARI/oD5rWPcvD89LZbge4aY792ysdF6rZfN0jKyIqDWwvrWPCoCyq7TfR/MzUiOeFguX5dy7mzktnsOVgLd87I7J3T/UMY8wCYEHUulsdj28Gbk50u7pLdaOf+uYAhZmplNU1t/8EjZZVH9be9Wov/zyRbQJHz3LQUGu/zyvVH2nP8gDw0S2ns+zHp0WsC337d87wN2t0fvhxRX1LgFzdEL8aBrQE3h9+Gjkb3wOLt3HJ399n4/4actJSWj3Pee5vPbmKu9/a0mofpQ7FAXuw0Q/mx/7yNX1kXgJbo1T/5EzDqGm0ZmZVqj/q0tdAEfk88HNgCjDHGLMizn47gBogAPiNMbO7cl7VOQVRPboAOXYPQJ1jcF5+RssbXWgSkbQUFzVNbfcsh2b9mzAoiw37W255O3uZJw9pncfmnPgkpNkfjFvuTqmO2m/X9R5blNVq2w/nT+ayo0cy8xevh9cZu2v54S/PiRicqpSKz+2os1zb5I8YzK1Uf9LVqGQNcBHwTgf2PdkYM0MD5b5hdKE1cMmZLuxxVM6osIPlwkxveIDe8Lz0mMdq9AU59jdvhm93xyobFOv23MmHFXP9ieMi1pV34Jb5bS+u5ZkPd7e7nxq49ts9y0Ny0hiWmxax7QvHjGr1hSyU5XnipGJOnjwoIW1UKtmJfRkFgoa6pgCZGiyrfqpLf9nGmPWgRfuT0TdOmUhOWgrnzxgWXjfTcWu60k7DKMpKparBx+NLd7KnsoF5E4pYsqW01fFCM/SdNmUQf/jcdJ79KLKyUazbcx63i2+eOpF/vL0tvK60tokhUcGNU32zn3+9uwOAi48a0f4LVQNOVb2PHzz9MQCDcrw8+/Xj+fDTCvIzU9hf1UhOWgpN/phlaJVSndBSOs56b8706tTwqn9K1NdAA/xPRAzwD2PMfQk6r4ojLcXN9SeOj1g3bXguW399Fsfd8SYldm3agsxUdlU08OP/fgLA0DYCWbDynl2u1l+e4g38cFbgACtYjmfd3mrejRGoK+X07MqWuw5pKW6G5Lo5+8ihEfukuKJ6lhPSMqX6F2caRl1zoNX7uVL9Rbt/2SLyBjAkxqafGGOe7+B55hlj9ojIIOB1EdlgjImZuiEi1wHXAYwaNaqDh1fdxe0SDlS3BKzZaSnUNbXkNac4bl/ffcVMXl2zn5c+bplCeM6YgpjHbWuU9EvfmIcvEOTCe9+jrDZ+GsZZf1ncodegBq431h3g1TX7AfjRmZPj7hf6QjeyIJ1d5Q3hUopKqY4L3VQOGqhv8pORqj3Lqn9qN1g2xpzW3j4dOMYe+/+DIvJfYA5x8pztXuf7AGbPnq2fYL2sIDM1Yoa9Zn+QN797IjvL6jl58iCy01LCwfK188Zy5Ig8AKYNz2Hj/hqOGVfI4s2lMathhEwbnhuuxRyvZ7ksar2IVa4oVi+2GpgCQcO1j1hjjI8ZV8BXo+6cRPvg5lPJTvNw+M9eixjcqpTqmFAaRiAYpN4XIFODZdVP9fg9ExHJBFzGmBr78WeB23v6vKp75GdEVtIIBA3ji7MYX5xlb7eCjG+dNpFvnTYpvN+zXzueoDHc8NhHQOycZacsrwevxxW3Ju5NT6yMWDYGapr85KZrkKOsSUiWOybEKczytvucUG78Y9fObTe9SCnVWqivor45gDGQoQP8VD/VpWoYInKhiOwGjgVeFpHX7PXDRCRUQH0wsMSeCWwZ8LIx5tWunFf1rO+e3hL0Rt9W80T15B45Io/nbjieb546MWJ9qscVMXvg2KK2pw0WEYqyvJTWxO5ZLq+zep6/eMwofv+5I+11HZhsQvV7gaDhzjc2c9l9H4TXFXcgWA45fkIR44pbl5hTSrUtNCNraLZMTcNQ/VVXq2H8F/hvjPV7gbPsx9uA6V05j0qsK+aO4o+vbwIip8OGyPJyITPamODhFxdMY9iirZw4qf1yXEVZqZTGCICNMazfV02218MvLziCNXusqbg/3l3J6l2VuFzCedOHtXqeGhh+++oG7ntnW8S6eGUOlVLdJ1QJq7YxFCxrz7Lqn/QvW7XivIXtTYnsKRiS07nb1cPy0vnFBdM6fN7QzGtOoV6LomyrXWOLMgFYtLGE/660StRpsDxwvbJmX6t144oze6ElSg08LrFS4gDNWVb9lk6VpmKaPTqfoiwvXkf1izsuOoKvndT2oKmuKMpK5aAjDePDTyvwBYLhms9fswdsZaS6cbskot5zR1Iy3ttSyv6qxvCMg9E2H6jhivs/CAfnKjk0OwagAvz07CmcfJhOLKJUIrhdEu5ZTtdgWfVT2rOsYnrq+mMRgWX2lNU3nDyey+b0bCm/qUNzeGrFbnaW1dMcCHLx395j7tgCbj5rCgC59mBCESE3PSVcCxrgkfd3RAwwjLa9tI4rHlgaXt7wi/kROdUHqxs5/c9WgZYPP63gxEnF3fraVM9xR02KdOHM4VolRakEEZFwB4PO4Kf6K/3LVjGFgo254wpZ9pNTOzVg6lBNGpINwJ7KBuqbrTffpdvLueCedwEiys/lpHkiepPTU9ru0fh4d2XE8om/f4ulP26piviXhZvDj4NGKxYmg+dX7WH9vhr2VjVyyewR+IOGZz/a06qCi1Kq57hEB/ip/k+DZdWuQdmJKatVkGkFOR/trOD3r21std05kDBUMu6Eidb023XtpE5sPVgbsXyguomSmiaK7TxoZyC+cmdlj9/Gb/YHaQ4EydKemEP2zSdWhR/PHl3AxUeN4Pbzp2mvslIJ5BahRgf4qX5Oc5ZVn1Fg9wjGCpRf/dYJEflwOXawnJeRSkFG7CoaIVX1Ph5burPVoK+3NhwMP65vbslj/te72w/tBbRj04Ea7npjM8YYrnl4OdN+9hr7qxrxB4LtP1mxq7yey+57n/e3lrF6V2XEtvNmDMPtEv3yoVSCuUSobbLGlegAP9Vf6SeL6jPiTSSRkepm8pCciHWhYDk7zUNhVirlbUyTvXhLCWV1zXz1xPFcOmckNzz2EYs3l/LT59bQFAhyyuRBlNU1M6rAqgW9s7yeQNCEa4i2xRcI8qfXN3HFnFH4AkHGFGaGezbfWHeAax9Zwd+/OIsZI/P5rJ0TnZeRwuLN1uDEY37zJmBN+T1teG675xsI3ttaSpbXE54NMuSJ5Tv5YFs5aSlbWbSxBICzjxzKd06fFJF/rpRKHBFo9Flf+HVSEtVf6V+26jPcLuHrJ43n3kVbARhdmMGnZfV867SJrfYNpWFkpropzPRSVhd7MhMgPBDw4qNGkJOWwqPXzOVL/1rGoo0l3PLcGm6x95s8JJuRBRnsLK9n5c4KZo8piHvMbSW17CirY19VI39btJW/2W0WgW+eOpGXPt7HFjv146v/91HEc3/2wtpWxzvnr0vYccfZcc83UASDhivutwZiOn8exhheWL0XIBwoA5w5bUh4NkmlVOI5057aGzuiVLLSYFn1KT+YP5mRBRn8e9lOHr1mbtxb68eMK+TxpTupbQpQkJXK+r3VcY95sKaJFLeQ55ga+8aTJ0QEXQDF2V6+dtJ4Xl93gD2VDcxuo52n/PFtoGW61xBj4M43Nsd4RvveWHeA06YOPqTn9hf7HXW2g0GDyyXUN/vZVlLHrvIGpg3PYc2eaq48djTnTR/GUaPze7G1SqlQNRqvx9Whu3FKJSMNllWfc/mcUVzeTpm6c48cSkVdM6dOGcT972yjtNbqPW7yB3j2oz1cPGsEqR4Xq3dVhnt9nT0gs8cUsPa2M/jboq0s3lJKYWYqf750Rjj43VvZenKUWIKdKJxx56Uz+NaTqwB4/CtzmTIkh9+9tpER+ems3VvFkSM1DcMZLJfUNtHkC3LBve+GK5/84fPTKchMTdigU6VU20Kz+BUloGKSUr1Fg2WVlESEq44bA1i5ztWNfpr9QV5cvY+bn/2Em5/9hE2/PJMPtpUB1u36aJleD9874zC+d8ZhEetH5Kfz+LJP+eqJ48IfBG156vpjueQf7wNWDnVoZHjI2UcO5ZfnTyM/M5Vzpw+jrK4pHOz95qIjOv3a+7OqBl/48e6Kev62aFs4UJ4xMo9Jg7K12oVSfUjochyUo8Gy6r80WFZJrzDLqqJRUd/MnoqG8PrFm0v4zSsbALjnilkdPt7eygaCBp5asYtLj47s4d5RWsdJf1gUXn7zuycyvjiLEyYWsXRbOZ/8/AwCQcP20lpy01PJTvNEDD5zu0R7RdtQ7QiW395UypsbDnDtvLHceMoE0lPdGigr1ceE+hNG5Gf0bkOU6kEaLKukV5hp9WiU1jaxv7olWL7hcWtg3QUzhnUqyAqlVry9qSQiWDbGcPtL68LLD395Tnhw2cNXzwlPZuJ2CRMGZR/aixngqu1eeRH4y5ubEYEvHT+GPJ1oRKk+6aA9gHru2PgDopVKdlpnWSW9Irtnuay2mTJHCblQOaP/d+yYTh0v1FOy4JP9zP31GwAcrGnkz29sZuGGg6S4hd997siIKbFdLsHj1supqyrtlItLZ48E4OwjhmqPlVJ9WGjC09D7sFL9kfYsq6QXqs9cVtcUMQV2yJjCzgVbD189hysfXAZYM/3tq2rg2N8sDG/f+IszNR2ghxysaSI/I4Xbzj+cueMKOOWwgV0dRKlkkZuuwbLqv7QrTCW9UM7ypgO1lNU1c/JhxTz79eOYNSqPxT84Oe5kJ/F8ZlIxF80cHl52BsqABso96EB1I4Oy0/B63Fw4cwS5GSntP0kp1evyM/VaVf2XBssq6WXbdZj/tmgr20vrGFWQwaxR+Tz79eMZWXBot/DjzaZ3+LCcmOtV99heWnfIvzOlVO/J055l1Y9psKySXnR5t872JMcyNSoonjUqjy2/OpMXb5zX5WP3BSIyX0Q2isgWEflRjO1eEXnS3r5URMb0dJtqGn1sK61j8hAdHKlUssnTu0CqH9OcZdUvvHjjPM69ewnQPcXxjxlXyDNfO5aL/2bVT37268d3+Zh9hYi4gXuA04HdwHIRecEYs86x2zVAhTFmgohcBvwWuLQr57324eUYY6XN5KankJ2WQnaaB6/HTcAYlm8vJxA0nDy5uP2DKaX6lDSd6lr1Yxosq35h0pCs8ONTpwzqlmMeNbqA759xGBMHZbW/c3KZA2wxxmwDEJEngPMBZ7B8PvBz+/HTwN0iIsaYTsxZGCk91cPmAzWs2VtFdYOfBl8gYrvbJXzpuDEcNVpLUCnlJCLzgbsAN/CAMeaOGPtcgnXNGmC1MeaKRLTtlMmDWLjhYCJOpVSv0WBZ9Qtej5v1t88nxd29JdxuOHlCtx2rDxkO7HIs7wbmxtvHGOMXkSqgECg91JP+9fKZEcu+QJDaRj9N/iAuF+Smp+D1aO+UUk4duRMkIhOBm4HjjTEVItI9PQYdcP+VswkED/k7tFJJQYNl1W+kp2qglWgich1wHcCoUaPa2TtSittFfqYOClKqHR25E/QV4B5jTAWAMSZhXb1ul+DWCkGqn9MBfkoNPHuAkY7lEfa6mPuIiAfIBcqiD2SMuc8YM9sYM7u4WHONleoBse4EDY/aZxIwSUTeFZEP7LQNpVQ30WBZqYFnOTBRRMaKSCpwGfBC1D4vAFfZjz8HLOxKvrJSqkd5gInAScDlwP0ikhe9k4hcJyIrRGRFSUlJYluoVBLTYFmpAcYY4wduBF4D1gNPGWPWisjtInKevds/gUIR2QJ8B2hVXk4plRAduRO0G3jBGOMzxmwHNmEFzxH0TpBSh0ZzlpUagIwxC4AFUetudTxuBD6f6HYppVoJ3wnCCpIvA6IrXTyH1aP8LxEpwkrL2JbIRirVn2nPslJKKdVHdfBO0GtAmYisA94Cvm+MaTXGQCl1aKQvpyGKSAnwaTu7FdGFclZ9kL6evq83X9NoY0yfvH+q12u/0d9ek16vMQzQ6xX632vS19N94l6vfTpY7ggRWWGMmd3b7egu+nr6vv74mhKlv/3s+tvrgf73mvrb60mk/viz62+vSV9PYmgahlJKKaWUUnFosKyUUkoppVQc/SFYvq+3G9DN9PX0ff3xNSVKf/vZ9bfXA/3vNfW315NI/fFn199ek76eBEj6nGWllFJKKaV6Sn/oWVZKKaWUUqpHJG2wLCLzRWSjiGwRkaSYXUxERorIWyKyTkTWisg37fUFIvK6iGy2/8+314uI/MV+jR+LyKzefQWxiYhbRFaKyEv28lgRWWq3+0l7SmVExGsvb7G3j+nVhschInki8rSIbBCR9SJybLL/jnpbMl6voNdsMlyzer32jGS8ZvV67fvXKyTnNZuUwbKIuIF7gDOBqcDlIjK1d1vVIX7gu8aYqcAxwA12u38EvGmMmQi8ScvUwmdiTVk6EbgO+Fvim9wh38Qqlh/yW+DPxpgJQAVwjb3+GqDCXv9ne7++6C7gVWPMZGA61mtL9t9Rr0ni6xX0mk2Ga1av126WxNesXq99/3qFZLxmjTFJ9w84FnjNsXwzcHNvt+sQXsfzwOnARmCovW4osNF+/A/gcsf+4f36yj9gBNYf9inAS4BgFRT3RP+usGaZOtZ+7LH3k95+DVGvJxfYHt2uZP4d9fa//nK92m3Xa7YPXbN6vfbYz7VfXLN6vfat69VuV1Jes0nZswwMB3Y5lnfb65KGfXtkJrAUGGyM2Wdv2g8Mth8nw+u8E/gBELSXC4FKY03RCpFtDr8ee3uVvX9fMhYoAf5l3/Z6QEQySe7fUW/rFz8jvWb75DWr12vPSPqfk16vffJ6hSS9ZpM1WE5qIpIFPAN8yxhT7dxmrK9OSVGiRETOAQ4aYz7s7bZ0Iw8wC/ibMWYmUEfL7SAguX5HqnvoNdtn6fWqWtHrtU9Lyms2WYPlPcBIx/IIe12fJyIpWBfxY8aYZ+3VB0RkqL19KHDQXt/XX+fxwHkisgN4Aus20V1Anoh47H2cbQ6/Hnt7LlCWyAZ3wG5gtzFmqb38NNaFnay/o74gqX9Ges326WtWr9eekbQ/J71e+/T1Ckl6zSZrsLwcmGiPCE0FLgNe6OU2tUtEBPgnsN4Y8yfHpheAq+zHV2HlWYXWX2mPBj0GqHLcpuh1xpibjTEjjDFjsH4HC40xXwDeAj5n7xb9ekKv83P2/n3q26MxZj+wS0QOs1edCqwjSX9HfURSXq+g1yx9/JrV67XHJOU1q9dr375eIYmv2UQnSXfXP+AsYBOwFfhJb7eng22eh3Vr4WNglf3vLKycojeBzcAbQIG9v2CNSN4KfALM7u3X0MZrOwl4yX48DlgGbAH+A3jt9Wn28hZ7+7jebnec1zIDWGH/np4D8vvD76iXf6ZJd73a7dZrto9fs3q99tjPNemuWb1e+/71arcz6a5ZncFPKaWUUkqpOJI1DUMppZRSSqkep8GyUkoppZRScWiwrJRSSimlVBwaLCullFJKKRWHBstKKaWUUkrFocGyUkoppZRScWiwrJRSSimlVBwaLCullFJKKRXH/wdlRChTA2DlrAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7384c0-0c22-45a6-b2e5-e091b56523e1",
   "metadata": {},
   "source": [
    "Okay, we've getting somewhat better, but we still have quite a few activations getting close to zero towards the end of training. Let's look at one last technique that can help us deal with this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf02e2d-6b9d-49dc-9232-4b8f9653c467",
   "metadata": {},
   "source": [
    "### Batch normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe0e4dfe-fecd-4701-975f-0e8cd944a760",
   "metadata": {},
   "source": [
    "A very effective technique to deal with vanishing activations in training is to use _batch normalization_ (or batchnorm for short). This techniuqe tries to maintain a good distribution of activations during training by applying a normalization operation just before or after each hidden layer. The way this works is to zero-center and normalize the activations in each layer by introducing two new parameters $\\gamma$ and $\\beta$ (not the same $\\beta$ from momentum!). The parameters are used to learn the optimal scale of the inputs across each layer, and the batchnorm operation can be summarised in the following equations:\n",
    "\n",
    "\\begin{align}\n",
    "\\mathbf{\\mu}_B &= \\frac{1}{m_B} \\sum_i \\mathbf{x}^{(i)} \\\\\n",
    "\\sigma_B^2 &= \\frac{1}{m_B} \\sum_i \\left(\\mathbf{x}^{(i)} - \\mathbf{\\mu}_B\\right)^2  \\\\\n",
    "\\hat{\\mathbf{x}}^{(i)}  &= \\frac{\\mathbf{x}^{(i)} - \\mathbf{\\mu}_B}{\\sqrt{\\mathbf{\\sigma}_B^2 + \\epsilon}} \\\\\n",
    "\\mathbf{z}^{(i)} &= \\mathbf{\\gamma} \\odot \\hat{\\mathbf{x}}^{(i)} + \\mathbf{\\beta}\n",
    "\\end{align}\n",
    "\n",
    "These statistics make it easier to train models, since we don't have to enforce a global normalization on the data (as we did with the `MinMaxScaler()` in previous lectures).\n",
    "\n",
    "Implementing batchnorm in PyTorch is rather simple: we just add a `nn.BatchNorm2d()` layer after each convolution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b127a070-d0d7-44c8-914d-cebce2bf766a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv(ni, nf, ks=3, act=True):\n",
    "    layers = [nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks // 2)]\n",
    "    layers.append(nn.BatchNorm2d(nf))\n",
    "    if act:\n",
    "        layers.append(nn.ReLU())\n",
    "    return nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5bd23f-8417-48cb-8370-a9dfc30b2bd1",
   "metadata": {},
   "source": [
    "Let's now train the model again with 1-cycle policy training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d4f749c7-c0c1-4a28-b340-9d297cf57678",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lewis/miniconda3/envs/dl4phys/lib/python3.9/site-packages/fastai/callback/core.py:67: UserWarning: You are shadowing an attribute (modules) that exists in the learner. Use `self.learn.modules` to avoid this\n",
      "  warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.216597</td>\n",
       "      <td>0.214471</td>\n",
       "      <td>0.909714</td>\n",
       "      <td>0.970490</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44923836-41e4-4d95-90f3-be584a5870ca",
   "metadata": {},
   "source": [
    "Nice! This is pretty great result after just one epoch of training, and gets is quite close to the results quoted for CNNs in the top taggin review paper. Let's also inspect the activation statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9e31bad8-aa9a-4168-8f76-64f11a0594f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAADSCAYAAAC8To2hAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABuNUlEQVR4nO2dZ3gc5dWG77Or3iVb7jayjY3pxhjTey8BQguQEEgIJCGkN0gIIRACJIS0j5AQILQk1AAGm2Ja6MUG94J7t+Wi3ld7vh8zs5pd7ap3nfu6dGl35t2Zd6Sd3WfOnPMcUVUMwzAMwzAMw2hOoLcnYBiGYRiGYRh9FRPLhmEYhmEYhpEAE8uGYRiGYRiGkQATy4ZhGIZhGIaRABPLhmEYhmEYhpEAE8uGYRiGYRiGkQATy4ZhGIMEEblJRB7t7XkYhmH0J0wsG4ZhDABMCBtG2xCRP4pIiYi8LyJjfMsvFZE/9+bcjL6JiWXDMAzDMAYFIjIDOBgYAbwDXOcuzwV+DNzQe7NrQkSSunHbwe7a9kDFxPIgQETWiciPRWShiFSJyP0iMlxEXhSRChF5VUTy3bGHich7IlIqIgtE5Djfdr4iIsvc16wRka/71h0nIptE5IciUiwiW0XkKz1/tIYx8BGRn4rIZvdcXCEiZwI/A74gIpUissAdN15E/ueOmwMM7dWJG0bvMx54R1XrgNeACe7yW4HfqWp5Sy8WkQdF5G4RmeWeVx+KyETf+ikiMkdEdrvn5kW+dWeKyKciUi4iG0XkJt+6IhFREblSRDYAr8fZ9/Pu+e39hEXkijbs90ERuUdEZotIFXC8iOwtIm+63/VLROTsDvwtBw0mlgcP5wMnA5OBzwEv4ny5FuK8D74jIqOBWcCvgQLgR8DTIlLobqMYOAvIAb4C/EFEpvn2MQLIBUYDVwJ3eyLcMIyuQUT2Aq4FDlHVbOBUYDnwG+BxVc1S1QPd4f8G5uGI5FuAy3thyobRl1gCHC0i6cCJwBIRmQ7spar/buM2LgZ+BeQDq3CENiKSCczBOe+GueP+KiL7uK+rAr4M5AFnAt8UkXNjtn0ssDfOeR2Fqn7OPb+zgAuBbcBrbdgvwKXuPLOBD4HngVfc8d8G/uV+thhxMLE8ePiLqm5X1c3A28CHqvqpqtYCzwAHAV8CZqvqbFUNq+ocYC5wBoCqzlLV1erwP5wT7WjfPhqAm1W1QVVnA5WAnXyG0bU0AqnAPiKSrKrrVHV17CARGQccAvxCVetU9S2cL0jDGLSo6mLgaeADYBzwW+DPOAGj74jIWyLyLxHJa2Ezz6jqR6oaAv4FTHWXnwWsU9V/qmpIVT9193Whu+83VXWR+/26EPgPjjj2c5OqVqlqTaKdi8hk4CHgIlXd2Np+XZ5T1XdVNezONwu4XVXrVfV14AXgkhaOeVBjYnnwsN33uCbO8yxgD+BC97ZMqYiUAkcBIwFE5HQR+cC9zVOKI6L9t3V3uR8eHtXudg3D6CJUdRXwPeAmoFhEHhORUXGGjgJKVLXKt2x998/QMPo2qvoHVT1QVb8AXAS8haOHrsaJNi/DzWVOwDbfY//33B7AoTHfoV/EueuKiBwqIm+IyA4RKQO+QfPUqI0tzd3NrX4OuEFV32nLfuNsdxSw0RXOHutx7gobcTCxbPjZCDyiqnm+n0xVvV1EUnGuVO8EhqtqHjAbkF6cr2EMSlT136p6FM6XpAJ3uL/9bAXy3Vu0HuN6aIqG0ecRkeE4AvlmYD9goao2AB8DB3RgkxuB/8V8h2ap6jfd9f8GZgJjVTUX+BvNv0Njz2P/fAPuNt5Q1Xvbsd/Y7W4Bxrrb8xgHbG7PwQ4mTCwbfh4FPicip4pIUETS3MK9MUAKzq3fHUBIRE4HTunNyRrGYERE9hKRE9wL2FqcO0NhnLtFRd4XoKqux0mj+pWIpIjIUTj1CoZhONyFk/ZQDawFDhGRLOA4YE0HtvcCMFlELhORZPfnEBHZ212fDexW1VpxXDkubef2bwUyge+2c7+xfIgTEf+JO/Y4nM+Gx9o5n0GDiWUjgpv7dA5O4d8OnKvVHwMBVa0AvgM8AZTgnOQze2mqhjGYSQVuB3bi3A4eBlwPPOmu3yUin7iPLwUOBXYDvwQe7tmpGkbfREROAPJU9RkAVf0Ip8B9I3A8zjnWLtzvyVNwCuy24Jyfd+CcswDXADeLSAVwI873aXu4BDgMKPE5YnyxDfuNnWc9jjg+Hedz5K/Al1V1eTvnM2gQ1YQRf8MwDMMwDMMY1Fhk2TAMwzAMwzASYGLZMAYxIjLWrc5e6hrTf9ddfpM4TS/muz9n9PZcDcMwDKM3sDQMwxjEiMhIYKSqfiIi2TgNLM7FsVOqVNU7e3N+hmEYhtHbdFvvccMw+j6quhXHYgxVrRCRZZjXpmEYhmFEsDQMwzAAEJEinE6OH7qLrhWRhSLygLUtNwzDMAYrXZKGISKnAX8CgsB9qtrMckVELsLpOKXAAlVt0V9w6NChWlRU1Om5GcZAYd68eTtVtbA7tu16i/4PuFVV/+ua9e/EOV9vwUnV+Gqc112NY+pPZmbmwVOmTOmO6RlGv6Q7z9nOYt+xhhFNS+drp9MwRCQI3A2cDGwCPhaRmaq61DdmEo4P6JGqWiIiw1rbblFREXPnzu3s9AxjwCAi3dKqWESScboz/ktV/wugqtt96/+BY3rfDLeL1L0A06dPVztnDaOJ7jpnuwL7jjWMaFo6X7siDWMGsEpV17hG14/hNLbwcxVwt6qWAKhqcRfs1zCMTiIiAtwPLFPVu3zLR/qGfR5Y3NNzMwzDMIy+QFcU+I3G6XjjsQmnY5SfyQAi8i5OqsZNqvpSF+zbMIzOcSRwGbBIROa7y34GXCIiU3HSMNYBX++NyRmGYRhGb9NTbhhJwCScfutjgLdEZH9VLfUP8uc/jhs3roemZhiDF1V9B5A4q2b39FwMwzAMoy/SFWkYm4Gxvudj3GV+NgEzVbVBVdcCn+GI5yhU9V5Vna6q0wsL+2RNhGEYhmEYhjGI6Aqx/DEwSUTGi0gKcDEwM2bMszhRZURkKE5axpou2DcPvLOWjburu2JThmEYRhzqQo28vGQbNfWNCdffNeczKutCPTwzo69TVtPA3W+sIhy2BmhG/6XTYllVQ8C1wMvAMuAJVV0iIjeLyNnusJeBXSKyFHgD+LGq7ursvndW1nHzC0u58qGPO7spwzAMIwFPz9vM1x+Zx6MfxC8Wf2nxNv782kquftjcFYxobnlhKb97eQWvL7e6fqP/0iU5y6o6m5gcR1W90fdYgR+4P11GaXU9ABW1Fs0wDMPoLj7bXgFAcUVt3PUBcdLe31/T6RiIMcCorne+n+tC4V6eiWF0nH7dwW9XpSOWU5P69WEYhmH0aTyxvL28Lu760poGALqgx5UxwBD3QqrR3hxGP6Zfq0zvAzoQiFfMbxiGYbSVyroQZdUNcdet3+XUhWwrix9Z3lXZJKLrLYJo+Ah6Yjls7wuj/9KvxXJtg1NsEmq0K1bDMIzOcOTtr3Pgza/EXecV7m0pq4m73i+id1TGjz4bg5OgG8wyrWz0ZwaEWPZyogzDMIyOUVYTP6oMRFwwisvr0Di30zeXNono4vL40WejdUTkNBFZISKrROS6OOuvEJEdIjLf/flazPocEdkkIv/Xc7NumYClYRgDgH4ulp1L1fIaE8uGYRjdQUNjmPrGMLnpydQ3hpt93obDyrz1JUwdmwckzms2WkZEgsDdwOnAPjhdNPeJM/RxVZ3q/twXs+4W4K1unmq7CLoqw6zjjP5MvxbLNW5kub4xHDfaYRiGYXSOajeqvMeQDAB2VNZy43OLmbN0OwAVdSGq6xs5dHyBsz6BY4bRKjOAVaq6RlXrgceAc9r6YhE5GBgOxM+l6SW8NIwtpTWWz270W/q1WPbSMMBsaQzDMLqCRjcC+N7qnfzg8fmRNLdRuekAbC2r5eH313OV66lcUeukb+wxJJOAQHGFRZY7yGhgo+/5JndZLOeLyEIReUpExgKISAD4PfCj7p9m+/DSMP78+ique3phL8/GMDpGvxbLNT6x7BfOhmEYRseocsXx1x+Zx38/3cyKbY5t3IjcNAAWbS6LjJ23viTic5+fkcyQrFSKLQ2jO3keKFLVA4A5wEPu8muA2aq6qaUXi8jVIjJXRObu2LGjm6fqEPS5Vc1evLVH9mkYXU2/Fst1DU3R5OoEbVgNwzAGO2XVDfxhzmdsTeBm4U9j88RuiptsunCTI46H5zhiecHG0sjY5dvKI2I5Oy2Z4TmpCRuXGK2yGRjrez7GXRZBVXepqnc1ch9wsPv4cOBaEVkH3Al8WURuj92Bqt6rqtNVdXphYWFXzz+Krz30MS8s3BKJLIPZChr9l34tlhsaTSwbhjE4aU+dxitLt/Gn11Zy39tr4673f356zhZeM4l1u6oAGJ6TCsDLS7ZHxpbVNETSMLLTkhiWnRaVhlEfCkfSOoxW+RiYJCLjRSQFuBiY6R8gIiN9T88GlgGo6hdVdZyqFuGkYjysqs3cNLqb+RtL+feHGwB4dVkx1/77U9JTgpH19lYw+iv9Wiz7/ZUtDcMwjMHCZ9srmPCz2by3amebxpe70d+Nu6ujlpfVNHDmn9/m5SXbIsu8BiMpQUcsew1JRro5y+AI55SkAGXVDb7IchLDslMjYvnVpduZfMOLfP2ReR05xEGHqoaAa4GXcUTwE6q6RERuFpGz3WHfEZElIrIA+A5wRe/MNj7n3v0uP3tmUZTzRXKwX8sMwwAgqbcn0BkawhZZNgxj8PHG8mJU4YVFWzliz6Gtji+trgdgV1V91PJXl25nyZZy/vnuusgyT/xWuwGI7a5vcl5GMkfuOYR3V+1iXEEGqtUxkeVkhmWnsquyjpr6Rr7mFgC+umw7RttQ1dnA7JhlN/oeXw9c38o2HgQe7IbptZl6313f1izj6kKN7KioY0x+RmRZifs+zc9M6Z4JGkY76deXfP7Ico1Flg3DGCR4+qOqrsnz+NZZS/nyAx/FHV/iiuXVOyqj0tc2uJHmDN+t8vKaBsJhjTQp2e0Kl4yUYMQRY0hmKrnpyZRWN0Si1tlpSRTmpBFWOPPPb0e2l5rUr79mjA7gd6dqrRnJD59YwFF3vEFdqOk7/KBb5nDQLXO6bX6G0V76dWQ55Iss11gXP8MwBgmenVvIF7X7h5uPvKmkOipKB1Ba3RD5vWxrOWPzM6hpaIwU/Pm791XUhaioC+FpHO+uXVpykNz0ZMCJ+OVVJbuR5RApwQBpyUGGZTt5zWt2VkW2l5bcJMSNwYG/kO+eN1e3OPa1ZcWAE/xK7deKxBjI9OtL/oZGJd39ILbIsmEYg4Fz7n6Xh99fD0Q7AnnMWtjcnssTy+C0rj7oljl87i/vUOIu31nZVJRXUdtAeZzW1ynBADmuWM5OSyI33RHLVXUhMlOdz2FPLPupqgtZ06hBhj8NI5by2gZO+P2bLHJdVjyzjJBV/xl9mH4ulsPkpDuXopazbBjGQKeqLsSCjaWRSLC/sDnFTXfISmseniupricvwxG63i3yXVX1kWK+nZVNuczltaEoce2RFBQ8y9yACDmeWK4PkemGBD17OY+iIRmEwmpNowYZizaVJlw3d91u1uyo4s5XVgDgGcuFWhDYhtHb9GuxHGpUctKcL4AaE8uGYQxwYrvjRbkAqbesuegorW5ghCtkK315zos3l0eNK8hMYdbCrVFpGR7JwUAk+pcUEDJTkqhpaKS6rpHMFEcsD82Kjix7z/251U/P28S3/vUJ76/e1eKxGv2Xbzz6ScJ14srj+lCYZz7d5L1taWi0yLLRd+kSsSwip4nIChFZJSIJvR1F5HwRURGZ3hX7bWgMk+1GUUwsG4Yx0Ckuj2744U8/8wqp4tlollTXM8wVy7t9jhixt8u9dQvcyGChL60iORiI2IClpwTJSAlSVReiqj5EhpuGkeIr5jtz/5FcdIjTY8MT6HWhRn745AJmLdrKL2cuBpy22gtbiEQa/YMtpfEb3jTDDSW/v2YX3398QeSucHlt8ws0w+grdFosi0gQuBs4HdgHuERE9okzLhv4LvBhZ/fpEQor6SlBkoNClYllwzAGGHPX7eagm19hvts1L1FkWVUJu2I5NnBQF2qkur6REW5TkZIY+7h47HD34zUiEXHaFl9xRBFfObKIK44oIj0lSF0oTHltKBJZ9nPBwWPIcYMZnlgur2mKMHvC+9J/fMjZ//duq3My+jZH3P56m8ZJguUX/f39rpuMYXQxXRFZngGsUtU1qloPPAacE2fcLcAdQJf1Qg01hkkKBMhKTYq6zWcYRtsQkbEi8oaILHWbHXzXXV4gInNEZKX7O7+35zoYufuNVZRUN7B0i5Mu0VwsO5HhUFgj7hU1DY3856MNfPPReZRU1bOquBIgkoYR67U8ZUR25PFXjxwPNHkrF7ppFMkB56siMzWJX35uXzJTkyICeVdlXaTAD+D960/ggoPHcMSeQyK5zFV1joD3p3dkpiZFFf61VAR44d/e4+bnlyZcb/Qfwgn+z/Hy5A2jr9AVYnk0sNH3fJO7LIKITAPGquqsLthfhIZGJTkoZKclR+XhGYbRZkLAD1V1H+Aw4FvunaHrgNdUdRLwmvvc6CF++MQCbpu9jFJXXFbWedZv0ULXS6PwF9DVNjRy/X8X8eLibRx0yxzO/PM7AM3SMA4YkwvAV44sirz2xL2HAbC1rJaUpADZbk1IcrB5PNBrY7yjoi4qsjwyN507LzyQ1KQgWa5Y9hqXzJy/OTLuo7W7+ftbayLPl22tcPddw67KOn705AKWbClje3ktH68r4YF317J6R2WLfzej7/PVB+f29hQMo910e4GfiASAu4AftmHs1SIyV0Tm7tixo9Vth8JNkWWv65RhGG1HVbeq6ifu4wqcNrujce4OPeQOewg4t1cmOEh5+pNN/P2tNZHuZ/HcL4BIg5E63/JENprDY8TyjWftw7rbz+ToSYWRMZ4n8vbyWvLSkyOpEklxWhZ79SJ1oXAkZzmWoiGZAKx0o9sfrN0NwHkHOfGU219cHhnreT4fftvrHPzrV3lq3iYuf+BjNpU05cKe+Pv/xd2P0bu0Zg24z8icNm0nHFaemrepK6ZkGF1KV4jlzcBY3/Mx7jKPbGA/4E0RWYcTvZoZr8hPVe9V1emqOr2wsDB2dTMaw0owIGSlJUUiL4ZhdAwRKQIOwqkrGK6qnmHvNmB4gte06wLXaB2/IPaCAE1i2RHHXiFdQyh+ZDkeI3PTSAoIW1xR6qVIDMlqains+dZvLaslNz05sp/kOGI5P6PpdfFylsFpXjKuIIMFG0upD4VZtqWco/YcynWnT2k29k+vrWTFtoqoZTsr6/jFs4ujlq31NTwx+gYtOVlcMmMso/LSEq73s7u6nh89uaCrpmUYXUZXiOWPgUkiMl5EUoCLgZneSlUtU9WhqlqkqkXAB8DZqtrpezFhhUBAyE5NsjQMw+gEIpIFPA18T1Wj/MTUCRvF/TZs7wWu0Tp+z3jPIaDMLYyrbWhkVG4aK245jW8eNzEiUvxi2V9E52dIVgoFmSls2OW0uPYEbmpSkNP2HREp2vNYWVxJipt+ES8NoyDTJ5ZbaL1WmJ3Ki4u3cfuLy6moC3H0pKEMy0ljxviCqHELN5Vx6h/favb6pVuj7e2Ov/PNhPsyegd/q+pYQo1KQBKV9TUfaxh9kU6LZVUNAdcCL+Pcwn1CVZeIyM0icnZnt98SjWElKI4Jf6WlYRhGhxCRZByh/C9V/a+7eLuIjHTXjwSKe2t+gw2/8PCKnraV1VB03SyenLeJtOQgIkJyMEB9YxhVjXpNosjriJw0CjJTIl7J/qK8v112MDedvW+kCNDDiywnxRHLw3KabOVaEsvjhzqpGA+867TjHlfgtOL2bt3/84pDOGLikKjXFA2JbtcN8I8vd4njqNENtNR0pjGszd4/08blxR3bYI1JjD5Kl+Qsq+psVZ2sqhNV9VZ32Y2qOjPO2OO6IqoMzkkYCAhZFlk2jA4hIgLcDyxT1bt8q2YCl7uPLwee6+m5DVb8Law9Ybt8a1N6QqqbKuFFfRvDGmUXt9n1uz1/2pjIsikjshER9h/tFPWlJQfIdVtX+0lPCTKx0BG3f7p4aiT9wnPD8DMsu0lYj8xNfJv91+fuF/Xc84P2Cg5zM5K5aPrYqDFv/vh43vjRcZHnN5y5NydOcYoPs+N0KDR6l5bE8oTCzGaR5YwEaTuxYjlsLbCNPkK/7uAXViUoTs6yFfgZRoc4ErgMOEFE5rs/ZwC3AyeLyErgJPe50QPUxrmlXeELBqQlR+cRNzRqJHUjP6NJAF9z/ES+dpRjBTcmPx0gkvpwwJi8uEV7AL8+d3/yM5I5elJhJLLsbzbi5+A9HEfB0XnpCY8nLTlIMNAklk7a20l/v/Xc/bjpc/tw0Ng8ctKbi6fxQzN540fHcf3pU7jyqPEEAsKFB49JmB/t8cA7a3ln5c4WxxhdS12cPPmT9h7Of646jG8et2fU/x+ISvfxE5v7HO9cMIzeoF9fonsFfjlpydSFwtSFGklNin8SGobRHFV9h8R9Ak7sybkYDnVx2lX7SXM/4zyx+9c3V/GvDzcATnvpEjd1IyMlyAFj84AmYe3lGR9SlNg2+/CJQ/j0xlOiXpco5/Teyw7mf5/tiNjQJaIwK5Vt5bX87oIDIo4beRkpXOH6Og/3pX+8/ZPjI4/HD83k68dOjDzPSU9me0UtlXWhiC1dLDe/4Pgx7zc6hxe+fXSL8zK6hlCcCHBOWhKHu+k1wWaR5URiOfq9X13fmDAKbRg9Sb+PLAcCEmnJWlxe18orDMMw+jaxt7S9SLKHF5Xz0jD+8vqqiB3c0KymPOL05CCHTxhCdloS33AF5/F7DeOvX5zG90+a3Ka5pLoR5UT1WUOyUjlv2hiklQKufUc51mFe/nLz9bm8+oNjWPyrUxlb0Dxf2eOYyYWowlcf/JjiCqdxiqqybmcVz366maLrmqz8F28uZ9763S3Oqy8hIqeJyAoRWSUizXzNReQKEdnhuwP0NXf5VBF5320qtFBEvtDTc49XmJfmE8SxkeW2iuXHP94Yd5xh9DT9+pLNKfATRuU6twA3ldS0+EFrGIbR14l1FshNT6a2oSkQEJuG4cdvA5eWHCQvI8iim06NLAsEhDP2H9nmuXhRvcZO5o7eccEBPPL+eg4alziiveew7ITrPA6b4KSRfLR2NzNufY3bztufdbuq+Pv/1sQdf9n9H7H05tM6NukeRESCwN3AyTiNvT4WkZmqGtu28HFVvTZmWTXwZVVdKSKjgHki8rKqlnb7xF3ivT+SfQI5tsBvd4KW67FpGL97eQWn7juCPYdldcEsDaPj9OvIspeGMTTb+YIoqY5/AhqGYfQXamPSMPLSU6Kee2kY8cSyP7KcmiDPuD14jhmdFctDs1L5/smTm0UY20tsmt31/12UUCiDcxv/ygc/jljw9WFmAKtUdY2q1gOP4TQGahVV/UxVV7qPt+A41/Soj2NjnKYk/pz4YydHT2fKiPhNSvzt0D3O+PPbnZydYXSefi2Ww+rk0nm5a2YfZxhGfydeZNmP54aRHEcMe41CUoKBVlMj2kJmF0WWe5PXlhdzwp3t6/z39sod3PPm6shz1SbHkafmbYp0G+xCRgP+nINN7rJYzndTLZ4SkbGxK0VkBpACrG7+0u5rItQYbp5nn+S7MDptv5Es/pVzh2NcQQYn7xO3xxFXPdzcKKu+BacNw+gp+rVYdiLLNIlls48zDKOfE1vgl5sRa/HmCNfkOFHaSC5o53Wysz0vstxKO+Oe5E8XT+X3Fx7Ibz6/f9z1z1xzBL89/wBmFDU1PdlZWUdpdT3rd1Xx4qKtUePf+mwHCzeVAvDZ9greXFHMZfd/xB0vLY90Q3zkg/XsfeNLLN9Wzo+eXBBX1PUAzwNFqnoAMIemdvRAxA/9EeArqhpXYXZXEyEvZ/nWz+/H5YfvATRPvchKTWL2d47mhe8c1ewOw9++dHCXzcUwuoP+nbPsFvhlmlg2DKMfMmfpdmaML4iKHsfaZeXFRJa9CGe8NAyvqCqekO4IXmS5JWu4nuacqU0B1wsOHsMPnphPSlKA/36yGYCDxuVz0Lh8ioZmctHf34+MvfuNVTz6wQZqGhpZe9sZkcj7lx/4CIB1t5/JKX+I7iD49sqd7Dsqh6fmbQLgtD86KQE7Krq8mHwz4I8Uj3GXRVDVXb6n9wG/9Z6ISA4wC/i5qn7Q1ZNrDe/Ow8TCrEihfTCON/c+bqHn9kBt1PLOpucYRnfTryPLYbfALzkYIC05QEXfz0szDMMAYPWOSq56eC43PLs4anlsZDkvJrKc7grYeGkY6W6KRqCLxMd+o3O5/vQp/PWL07pke11NSlKA/7t0Gr+/8EAAzjqgqXjxkKJ8/nnFIZzi3vL/x9trqXEjxU/M3cicpdujml74nTQ8rnp4Lkfc/joLN5VFLd9eXtfVqSkfA5NEZLyIpAAX4zQGiuB11HQ5G6djLu74Z4CHVfWprpxUW3l+oROtTwpI5O/S0gVb7PvzmMlDu29yOHcPFm92/oc7K+t4Yq65bBjto99Hlr0r0uE5aWwz6zjDMPoJG3dXA7C9LDrKFmsd50Wdh+ek8tUjx3PuQU5kNTlOC+pMN7KcyBe5vQQDEuVz3FcRET79xclk+br7iQjHTxnG8VOGNRPCP316UYf2MzovPdIh8fkFWyL/i86iqiERuRZ4GQgCD6jqEhG5GZjrdsP9joicDYSA3cAV7ssvAo4BhoiIt+wKVZ3fJZNred58tHY3//nI8fkOBoQGN385GOf96X+dx6jctFb7I+ysrIsqXp23voSpY/PaHJH23z245l+f8NHa3RwxcQhj8s09y2gb/TayrKqoNn0pjCvIYIP75WMYhtEbVNWFOPBXr/DGiuJWx+6qdNx7UmN8lGML/Ia4IiEojnD1Gnik+NIwrj1+T/71tUMjKWldpJX7FfmZKXFTU9rDg185hBOmDOOqo8czZUS0lV1WahLv/PR4LjzYaSNeVd+1aX+qOltVJ6vqRFW91V12oyuUUdXrVXVfVT1QVY9X1eXu8kdVNVlVp/p+5nfp5BLw3PwtfOHepqyPYEA4YqITJT50fEGil0Us4vYans1717fe++j4O9+MPF6wsZTz73mPP8z5rENzLi53Lk7jeUMbRiL6bWTZu9XjXVkWZqWydmdVb07JMIxBzrKt5ZTVNPDn11Zy/F7D4o6prg+xbGs51a7Yqg+F2VRSTV0ozAPvrCUYEFKCAerdBg1eI4+dMd60fmuuLx++B8Ny0pi3vsRZFydfdDBz6PgCPlzbvEHJ14+ZQNHQTFYXV3LfO2s5dnIhx7n/N1Xlybmb+HDtbm79/H4EA4KI8OvP78eT8zaxs8KsSmMDVMGAcOzkQpbfclqkU2M8st07ANNb6CTpp8LndOV5NC/cXJZoeIt4ueomlY320H/FskaL5dyMZEqrLWfZMIzew2s17Vm4xeP6/y7iuflb+MqRRQB8uHY3R93xBlNGZLN8WwVJASE9JRgRy+PcRkuxFlr+NAwvouq5YWSn9duP9m7hjvMP4Lg73yQ5KJw/bQyPfbyRG87cm68dPQFwhPHPztg7ym5PRLjokLFcdEi0Q1tqUpArjxrP/mPiewUPJmIj+d5FWktCGWBMfgYvfPsoJg9vvRFNLClurn5DBy3lvP9wuA85vBh9n377ierZOnppGPkZKVTWhWhoDHf6VpxhGEZHKHebKuS0IFaXb60AaHYnbPk2Z3korKQmBanAiaaNzHXSLk7aO9qb1p+G4dl0ee4VXqc7w6FoaCbrbj+TUGOYitoQ5bUNfN6Xbywi7Upd+cVZ+3TDLPsfsXnz7XG12G90bof26Ynl+sa2ieW/vLYy8tjvmBXbWtswWqLfiuWmyLLzvCDTieTsrKxjZG7fsTkyDGPwEPKKm1pIg/Civu+t2hW1XAS8YFdqUoB5N5yE4gi55becFtXkAaKjet7jcUMyeOobh3PAmLxOHsnAJCkYID8zhb9+0Xx9u4KUGEeWrrKA22t4Niu2V8Rd5+2hrc1Kfu/LbV5dXBnZQEMoOrI8a+FWJg/PYlIHot3GwKffhmC9nGUvsjyx0Okdv3J7Za/NyTCMwY33BR5PM7yyZBtF181irptXHBsZ898VTk0OMCQrNeIAkJYcjMpRhmjrOL9wnl5U0EzEGEZ3EJsbH3tB11F+evpezZZ5DWJC7nf/os1lkWVt5ZlPN0c6/dY3Rr/2W//+hJNjfLbbS3FFLZ8lEPlG/6bffqKGY8Ty5OGOWF6xzd6ohmH0DnURsdxcNFz9yLw2b2fNjtaLlf23wK2pg9EbxL7Nu8Lfe93tZ3LClOFMcAtbPb7/+HwguvX6b2YvizyuCzVSFdOYLNZZ5sH31lHsNpSpD7WesxwOK2XtqIU64rbXmzW2MQYGXSKWReQ0EVkhIqtE5Lo4638gIkvdnvavicgend1nOKbAb0hWKiNz05i5YEtnN20YhtEhPLHcE9ZtrXnTGkZ3E3LvjngR5cYutGN79QfHRj1/cfE2IDrXeFVx053kz/3lHfb95ctRr6mqSxx5bkvO8h9e/YwDb36Fkqq2OZ+EurZRjdGH6LRYFpEgcDdwOrAPcImIxFY/fApMd3vaP4WvTWdH8XKW/VeyFxw8hsVbyqyTn2EYvYInluMV2ufHdOKDzt22TkvutzcGjQGC55fstbHuyovEeFHqqx6eyxX//DjyPCMlyCPvr2PPn83ms5gUzI27q5l2y5yE2/fnMocSCGevhbq/MHBHRR37//Jl7nt7DeDczS66blaPpl/88IkFfO4v7/TY/voC1fUhiq6bxe9fWUHRdbMiHRl7iq74tJ0BrFLVNapaDzwGnOMfoKpvqKpnyPgBTt/7TuG5YQR9Z+feI3NQJdJhyTAMoyfxbvvGi1pV1TU2E7ix6Rp7taO4KMVcf4xe5m//W+38/tLB/PmSgxhb0LUd8V75/jEUZjd17puzdHvU+rTkIL94bknciO7/PtvR4rYXbCzlN7OXUXTdLF5fHr+JkNce/XuPz2dVcQUlVfUccftrVNSF+PWsZZz313e51U0FmTm/e+5qPzd/M+fc/S53vLQ8suzpTzaxaHMZry3bzq9fWNot++1rbCl1msn85fVVgJN/Ho+6UCPf+c+nUXcduoKu+LQdDfgbrW9ylyXiSuDFzu401g0DmhwxvM5YhmEYPUldgyOSY4v36kNh6hvDjHC770WWx4xLTmp7aE4GY5s+o8+gqpH836FZqZx94KhObe/hr87g1s/vF7Vs8vBsZn37qISvyU5rfrfGa6XtteBORHJQuPctJzqcqJ7AKyCct76EHzyxgMVbyiLRdIBPNpTylivKt/iCdDvcvwvArso67n1rdVSL7/bw3cfms2BjKfe8ubrZuisfmst976xtdRtlNQ38/X+rI7VefYH5G0t5at4m7nlzNa8s2cb9rRzH7EVbo57XNjSyeHMZsxZGL/9gzW5mLtjCr55f0qXz7VHrOBH5EjAdODbB+quBqwHGjRvX4rZiC/wAhnhiuY35RYZhGF2Jl4YRG1n2Co9G5Kaxblc1mSlBquobuWTGOD5au4vVbkFfcjDA9adPYVSe2V8afRv/92ys33JHOGZyYdzlw3LSuO28/bn+v4uarVuxrbzZstqGMCKwZIuz7oYz9+af765jc2kNRUMyWLfLucmdk5YcVyss3lxGWJWctGSq65tynhduKmNzSeK71v/1RToPufVV1t1+Jgs2lvLbl5fz7qpdHDAmj8MmDEn4+ngs2FjabFllXfvbrF/4t/f4bHsl+47K5ahJQ9v9+s6wqaSamvrGZpZ85979brOxXzmiKG76TW1DI3fFtDevbQhzlpuKcuYBZ/LJhhImDM1kp3uh4gVPu4quEMubAX+LozHusihE5CTg58CxqloXux5AVe8F7gWYPn16i5dAse2uAUblpRMMiHMCdfIq1zAMo714aRiehdxbn+3glheW8rfLHF9fzwM+PzOF5789g6IhmQQCwjl3v8uCjaUkBwJ8/diJvTN5w2gH28pqI4+7+y5HrMuFxycbSpst21Zeyzpfw5+vHT0h0qnxsY82cJ0ruhMF1c5qIRf4ujiCPRFvLC/mKw825Vcv21rOYROGUF0fojGscaPisZwTIyhVlasemttsnKom/B80hjWSz72jsjbumO7kqDveAGi1BTpAbaiRjJTmsvT2F5fHHevRGFbO++t77DEkg0tmOIFWfxfV4opahmWnNdtGe+iKNIyPgUkiMl5EUoCLgZn+ASJyEPB34GxVjZ8c1E5i210DZKYmMXl4duSK0jAMoydpiiw7n08/fmoBK4srOfH3/wOI5HRW1oWYUJgViaKkuvlk7UnDMIzeZKsrlp/91pHdvi/Pb7wtHH/nm1Ei1c/FM8bxyS9O5ryDWsoU7RpiC9B+9fxSlmwp4/Q/vc3+N73S4mtVNW7TlfrGMO+v2dVseUMLLiRPf7Ip8vj7jy9obdrdxsX3ftDqmNqG+IWWa3Y2t9Ks83lsF1c478X1u6ojwtrzml++rZwZt77Gvz9sOS2nNTodWVbVkIhcC7wMBIEHVHWJiNwMzFXVmcDvgCzgSffqZ4Oqnt2Z/cZLwwAYk5/O+l2te5QahgEi8gBwFlCsqvu5y24CrgK8Cpmfqers3plh3yUcVj4rrmDKiJzIskjOsvtFVxrj0TppWFbc5d4He2yTh9Z4/OrDyI3jsmEY3c22MiclYVRu5yJ2beGcqaPIzUjmK/+ML4ITcf/l05stK8hMYWc3pWr6u3D+PiZtABx7Oy9t+KdPLeTd1TsZnpNGdloSD35lBuDcnTrqjjei8p49zr37vbj7nXzDi/zsjCmMK8jkG4/OY0ROGgfvkc/dX5wWdzu9wfyNpXzuL+8wPCeV+y4/JO4Yf5OZu99Yxe9eXsG628+M2+Tp1WVNcdfDb3u92XrPaci7qJu9aCuXHtpyem9LdEk5tarOVtXJqjpRVW91l93oCmVU9SRVHa6qU92fTglliB9ZBhidl87mkpoOJ9MbxiDjQeC0OMv/4DtfTSjH4dEP13PaH9/mhmcX8bzr7x5Jw3BzlutiokMjXWHxjZhUi1RXLCe30+Hi0AlDosS6YfQEDY1hfvGcU0A1pB1R344iIhy/1zDGFjTl8ntt41tifExjE49EKdbfPG4iuektX3yOyU9cTzCuFTcQf33d43M3sqmkhnnrS3hzRZNzR0lVQ0KBu2xr4rvmv5m9nHtcd5Jt5bXMcgviSqujLwxUler6EJfd/yFn/OntyDY/3VDCt//zKY1hZVtZLV976OMW/aU/217BVx/8mJeXbOPUP7zFT55aENFdW8tq2OuGF/nmo9GFk4s2l0WJ3Fi+8s+PueCe91ixrYLfvbwCgC/d92FUyk9b8cRyppvW0dlatn7rPRTb7tpjTH46VfWNlNe0PwneMAYbqvoWsLu359EfWenmAT76wQa+/Z9PgaaIcqKGB1lpSay7/UyuO31K1PKUiFi2NIzBTBsafF0hIjtEZL778zXfustFZKX7c3l3ztN/97Ynu0f+84oZkccnTBkWeVyYnRrXszwrgaCOd7v/xrP24aenTeHX5+4X5xVNHDo+ukjveydNijxu78VuPCrrOt4norwm+rX3vb2Gf7wd7TIRCitvLN/B2yt3snRrOd94dB5vrCjm8399j+cXbGFnZR2Pf7yRV5cVc8WDH/P68mi7Po+fP7OI15cX8/VH5rFiewVPzN3EU/OclI973lxNXSgcaSQTy6MfrI+7fMX2CuauL+HUPzZ1QXxn1U6Wd6Az8zPzN1N03Swu+vv7ANTUd04T9luxHPFZjhNZBthYUh37EsMw2s61bsfNB0Qkv7cn0xfxBK6HqkYiyfHyDQHy0uNXaKd0MLJsDBza2OAL4HHfXZ/73NcWAL8EDsXpffDL7jxvP1lf2l2bbpE9h2Vx0+f24ZIZY/F/88/6zlE8+Y3Dm43PTo0fJf6pe7H6w5MnR5ZdPMPxKUgksD388bkbztyb7500OdJw6Odn7N3qaxLx/monF7mituOibm1Mbu+vZy1rNubFxdtY7nMRKa9piEpveeyjjby6zBHICzaW8tUH5xJqDLN4cxkbdjXpqngFhT9+aiF3v7GKD+LkVfu54dnFbTugGPYY0nYf7427o51LWsrrbgv99pM5ns8ywJh854+5qQWLF8MwWuQeYCIwFdgK/D7RQBG5WkTmisjcHTtabgIw0Iit0N9VVR8RyyXV9fxmdvMvqrwE+cVeg5EkiywPZlpt8NUCpwJzVHW3qpYAc4ifXtUl/OTphUDvNMa54sjx3HbeAZw9tcnxalh2WuS730+iLpdTx+ax7vYz+faJk7j2+D2dsW77+OxURyyPzE3jnZ8eDxDlj37QuDxO3mc4KUmBiMvGVcc4vw+fOIQx+enNotxrbzuTZ645osXjuuQfH1BT3xixhrvxrHjXSU34I+utcd60poLG7/zn00hjD4Cq+uiW4H949TMWxRQn/uyZRZz1l3c45ndvRLodJrqh8LuXVzTrpthVdCaYEArHD2C0lR71We5KWkrDAOviZxgdRVUj991E5B/ACy2MbbPd40CjIkYsr99VFclZ3llZH2l44CeRdZIXWbaufIOaeA2+Do0z7nwROQb4DPi+qm5M8Nput3x45lstC8Du5IQpw6OeF2ansu72M3novXX8cuYSnrnmiDZZ2v3o1L340al7RZ57keWwKmPyM1h3+5mAc+eotiFMekqQLx66R9Q2rjluT645zhHd7/z0BFSV8ddHl3ocNK4p0H/AmFwWbmrerjkUDkeaqh25Z8t+yO35rLjroqnsOyqXW+J0+0t0F8zPE3ObHDXumvMZPzltSjPt1RlG5KTx9WMn8KvnW+5GmN6K9VxLhAZrZDmcoMAvLyOZpICws7JvVIAaRn9DREb6nn4e6Ng9swFObOFMeU0o4obh5wvTx7b6xRZxw7DIstEyzwNFqnoATvT4ofa8uKvvBOW0wSu4p/ny4Xvw6g+OjRKn7cErCIstOxAR0lPaJtZEhE9/cXLC9Z4XcCwNjcqdr6wgIDChsKk48dJDx7HoplOYeW2TTV9sGlhrJPKqbi9/fXM1RdfNala83BmSgtImp4q9R2a3OiYRiepI2kq/jywHY65uRISCzBR2W8trw2gVEfkPcBwwVEQ24eQ9HiciUwEF1gFf76359WV2V0UX09Q0NMb9AslIDfK/nxwX12zfw3KWDdrQ4EtV/cmg9wG/9b32uJjXvhm7g664E+R3msppxTmiu3n+2qOapVqICHu6Fo0dwRPEmakdj2KC03jo1R8cS7WvsGzWd44iIMJn252CtcLsVMJhjTg1/HrWUkqrGzhwbF7UZ8EvP7cPqUlB9huVG1nWFjcQgKe/6eRyV8UUuE0alsX/XTotUky3/+jcqPSLqWPzmB+ng6DHvPUlgOtAFudO/iFF+Xy8rqRNc8xNTyY1Kfrv/Z0TJ3Hm/iOjiv32GBLf3aQthDrZ6rvfiuWIz3KcxJkhWansqrLIsmG0hqpeEmfx/T0+kX5IbOX5na+sYHecz50Dx+RFOvclwvuiMLE8qIk0+MIRvxcDl/oHiMhIVd3qPj0b8BLjXwZ+4yvqOwW4vjsm6b8g9PJ7e4v9x+S2PqidDMlM4cen7sUZ+49sfXArxIr2fV2xu3G3Uyh3wOhcDhqXx52vOJ7M//3EuTY6dd8RUa/zPh8CAeG606fwwZpd/OS0KRxSVMDc9bvZoyCTW+PUSAAUuQKzuq4pN3lCYSb/ufowhmalMmVENsu3VXDNcRPJTkvmS/d/CMD3T57M5Q981OoxPnDFIZz6x7dITw5S4/NJPmWfES2K5XOnjiIrLYmhWalccPAYAB65cgaX3e/s86Bxeew1IpsHrpjOVx+cyw1n7k2Z+5k7NCu1WfbAPV+cRqMq5TUhfvbMombj7rrowFaPpSX6rVhO5LMMMDQrhZ0WWTYMoxvxG+gDrNkRXYk+riCD6UX5nL5/9BdfPFLbeUvVGHi0scHXd0TkbCCEY/l4hfva3SJyC47gBrhZVbvcEvKDNbuiuunFC1b1d0SEb7lFf93FEXsOZdq4PH506l68FMdercBt1fzT06Y081z+xrETIz7t5x40mnMPGs2W0pqEYtm7o3XV0RN45IP1HLxHPk9/s3mueVZaEkdNasqT3mt4NlmpSZGCw0RMGpbFMZML+dpR43lx8VaeX7CVyrpQwuJKgO+csCc/OGWvZsuPnlTIEROH8N7qXZEiyROmDI/kjW8urWHuuhL+dMlUbn5+KcNz0rj/nbVMGZHN6b6LG08sz73hJN5btZP73lnLyfu0/jncEv1XLCco8AOnQ8866+JnGEY3UhMjlj28Ll6Hji/gdxe2LZrh+SuHO3mr0OjfuA2AZscsu9H3+HoSRIxV9QHgge6cX1taFhutk5WaxH+vcfKPGxrD/Om1lVHrvbSsbx43sdlr45HpS/G6aPqYqII8T7SOG9JUrOjnvGmj+c3s5c0aqqQnB1n8q1Mpra5n6s1z4u73zP1HEggID3/V8b8+ZnIhFbUhXli4ley0ZA7eI5/ymgbm/OBYiq6bBcAt5+zLZYcXtXpMQnNtNzovnf9cfRgA/3fpNMqqG7j/nbVccUT09mYUFUSyC47YcyhHtFIs2Rb6rVhOVOAHMCQzNVJRahiG0dWoakKxnJWaREVtqF3NGryLftPKRl8ltituZ29rGw4HjMnj75cdzNcfaep21950rAxffvVvLziQTSU1vLd6V1xxHMvVx0zkiiPGNysYTHVFdmYLqTaHTShotmzauHxeWLiVvUZkN4tg52UktyqUvfin0vqHYW5Gctxj/M/Vh3V5F+d+K5a9wsbYAj+AYTmpVLt+hVm9nFNlGMbAoy4UJtFncbYrlttiWxVLuIs/4A2jq4i9kEvkGW60n9jOne3t5OmJ62HZTorMA1ccQnlt2zsBxnPWSG1D0fGXDtuj2bKvHFnEyfsMZ2xMpHreDSeR2gbrNy+i3JmPQidQ0bUpQv1WSUbSMOL8Hz0D8e3ltWQVdrwq1jAMIx6x+cp+vEhMe9I5u9Kz1DC6g9imDrkJulEa7Se2X0ZyB2oYHr/6MMYPdYr50pKDCT3d20q8i/0Xvn0Ury8v5q45nyUcIyLNhDI4xgvtoa+FDfqtWG4pDaPQvboqLq9joollwzC6mEQpGNAUpWmPAPY+xiyybPRVYps6xHapMzpOdcznSWoHXHEOnTCkq6bTjN9feCDjCzPZb3Qu+43OjYjl7qCvxg36rVhO5LMMTbeHymrafhvCMAyjrdTUJxbLnkjOaGMDA2hyFTCtbPRVYn1qp3SiQYQRTW3M50lvuozcddGBfLwu2kjlfNfarSe44cx9+MWzizmkqGNNZbqLfutX5EVg4r2p8lzblbIaK/IzDKNreGXJNkrc5gEtRZa9XL/2NGzwPsUssmz0FZZsKYs0ngCaWZjFNpEwOs4h46ML5XrzY+C8aWO47bwDem3/e43I5olvHN5iE6feoN+K5cMnDuHhr85gZG5as3V57pdUabVFlg3D6DzFFbVc/cg8vvPYp0BTzvLxexVGtaUFGJnnNCDxim3ahBuNNqls9BX+MGclv3i2qdP9SXf9rxdnM7AZPzSTdbefyYyi5u4SRt+gb0n3djAsO41h2c2FMji3P5ODQqmlYRiG0QVsLnHauW5wO2/V1DsVOd88bk9mjC+IeIhOLMzktvP254DRuZw3re23Lr0bZF1td2QYHSUYaEp3NHqIdtimGT1Ll4hlETkN+BNO16H7VPX2mPWpwMPAwcAu4Auquq4r9p1gPuSmp1hk2TCMLuG22csBpxvWx+t2873H5wOOcT/ACVOGsb28lhe+fRQiwlXHTGjX9vdz2+AePrHz5vmG0RUEAxLplGv0DLnuXfG+3vb+rR8f32Iq2kCk02JZRILA3cDJwCbgYxGZqapLfcOuBEpUdU8RuRi4A/hCZ/fdEnkZyX0yZ7mitoGMlCQCEt92xTCMvsdHbsFLRkqQG55ZzM5KJ38zPcX5UnvgikM6tf0Dx+bx6S9OJj/T7LiMvkEwELCOkj3MHecfwEHjNjB9j75V3BbLuCHNreEGOl1x+TIDWKWqa1S1HngMOCdmzDnAQ+7jp4ATpZuVYl56cp+LLIfDyv43vcLEn83mW//+pLenYxhGK5RU1TP5hhcjz9fvqo6yNkrvwiIUE8pGXyIozR0wjO6lIDOFa47b0wJpfZCuEMujgY2+55vcZXHHqGoIKAO6zxQQJ7Lc18Sy38pu9qJtvTgTwzDawvJtFdSHmjoG7KysY/m2isjzISZwjQFKICAJc5avOnp8D8/GMHqXPlXgJyJXA1cDjBs3rlPbyk1PYdnWitYH9iC7q/teWohhGInxt53NTAlSFeOH2tkuWYbRV0kKSFwrw398eTon7zO8F2ZkGL1HV0SWNwNjfc/HuMvijhGRJCAXp9AvClW9V1Wnq+r0wsLCTk3KiSz3LXHa1+bTHqrqQmwurentaRhGjxAOK6pKtU8ce5ZwhjEYCAYkbhqGde4zBiNdIZY/BiaJyHgRSQEuBmbGjJkJXO4+vgB4XbvZIyk/I5mq+saoW6i9ze6q6LSQ/mQT9duXlnPk7a9z/X8XmZ2QMeD5zmOfMv3Xr1JZF4os83u6X3roOJbdfFpvTM0weoSASKTAb93OqsjyoIllYxDSabHs5iBfC7wMLAOeUNUlInKziJztDrsfGCIiq4AfANd1dr+tkRvp4td38pa97l8edX1IyLfGq8uKAfjPRxu4bfayXp6NYXQvLyzcyq6qetbsqIws23tkTuRxckBIb0c7a8NoCyJymoisEJFVIpLwe1JEzhcRFZHp7vNkEXlIRBaJyDIRub6zc/Fbxz05r6ksaWzB4HNCMIwuMfNT1dmqOllVJ6rqre6yG1V1pvu4VlUvVNU9VXWGqq7piv22hNfFry/Zx3k5yz85bS/ASW3oL/i7lD38/vpenIlh9ByLN5dHHu87qkksBwN92wfV6H/4bFhPB/YBLhGRfeKMywa+C3zoW3whkKqq++P0M/i6iBR1Zj7BgNDY6Ijltb7IctEgtA0zjAH7iZ+X0fdaXpdU15OSFGBoltMGt7q+/5h6NzSGSU1y3i6HTrCWnMbgYPm2JrGck57Mkl+dygUHj+HaE/bsxVkZA5S22LAC3ILTq6DWt0yBTLcmKB2oB8rjvLbNBKUpsuy5N119zASzNTMGJQNWLA/JdATp9vK6Xp5JEyVV9RRkpJCV6piQVNX3n8hydX0jh08cwtiC9IjYN4yBSnaac46u21UdWZaTlkRmahJ3XnggBWYZZ3Q9rdqwisg0YKyqzop57VNAFbAV2ADcqaq7OzOZYBzruMsO26MzmzSMfsuAFcuj853K9c2l1a2M7Dl2VzWQl5FMhpvrWFXXfyLLVXUhMlOSyEpNpqK2/4h8w+gINXHu+mSnJffCTAzDQUQCwF3AD+OsngE0AqOA8cAPRaRZz3URuVpE5orI3B07drS4v2Ac67i+3obZMLqLAfvOz01PJictiY27+47dWWl1PQWZTZHl6j4eWV62tZxT/vA/Vu+opKymgZz0ZLLTkqio7TupLYbR1dSHws0ssyYNy2L80MwErzCMLqE1G9ZsYD/gTRFZBxwGzHSL/C4FXlLVBlUtBt4FpsfuoD32rJ51nN+1ye87bhiDiQErlsGp2t1U0nOR5Y27q1u0g9tdXU9+ZgoZbovcvh5ZvmvOZ3y2vZIXFmxlZ2U9I3PTyE5NirLTMvo3IvKAiBSLyGLfsgIRmSMiK93f+b05x+5m9qKtzF60NfI83sXgnB8ca1E1o7tp0YZVVctUdaiqFqlqEfABcLaqzsVJvTgBQEQycYT08s5MJiCCanRtTUqSnQPG4GRAv/PH5KezqaRnIsufbCjh6N++wZPzNiUcU1JVT35GMpmpThpGX48sby1z/nYfrXP6x4zISXMjy3173ka7eBCINQy+DnhNVScBr9EDVo+9yTX/+oRr/vVJ5Hm5vb+NXqCNNqyJuBvIEpElOKL7n6q6sDPz8ZqP+F2b7ILRGKz0qXbXXc2Y/Aze+mwnqtrtFbyfrC+J/L5o+thm6xvDSmlNAwUZvshyH3fD2OxeaLy7yhXLuWlkpVlkeSChqm/FsZg6BzjOffwQ8Cbw056bVe9S3oe82Y3BharOBmbHLLsxwdjjfI8rcezjuoyAK5ZrG5r6AZhYNgYrA/qdv8eQDGoaGtlSVtv64E5S3+h8oCSyg1u2tRxVKMxObYosd0J0vvXZDo773RvdFp1evLmMkhjbveE5aWSlJlPZhyNv5bUNHHXH63yyoaS3p9KfGa6qXl7CNmB4ooHtKRjqi5THSbnwlp20d8LDNowBj9eprzbU2GyZYQw2BrRYPnBMHgCLNpV2+7686vltCYT5efe8B8CovHTSkoKIdC6y/IMnFrBuVzWri6taH9wBPt1YCsCh45s8lScWZpKdlkR9Y5jahr4ZFZ+/oZRNJTXc+fKK3p7KgMBtS58wEb89BUN9kZXbmzr0eTZZXi3BD06e3CtzMoy+QFIkstw3P+sNoycZ0GJ5ZG4aADsqut9r2RPLCzaVUlrdvGugN5dDJwwhEBAykoOdiix7+mVnVfcc24ZdVaQlB5hQmAXA+dPGkBQMRPxn+2oqhhf5iLU8MtrFdhEZCeD+Lu7l+XQbuyqbzp+//W810CQOrJ21MZgJuKmL3nfb906a1JvTMYxeZUDnLOe7jQN2VXV/y+tq9wu2LhTm1WXFXHDwmKj1dQ1hzp06KmIbl56S1KmmJN4X+c5uuhAoqXbyqzfudtxEDnO79nnzr6gN9cnmJJ7QCYdbGWi0xEzgcuB29/dzvTud7sN/Dv7u5RXM31jK6DzHoz09OcjrPzy2z14YGkZ30pSG4XyY7jcqtzenYxi9yoAWy8nBAHkZyT0SWa6tb2RUbhrbK+pYvys6NWJ3VT3bymvZb3TTh016SiCqcKK9JAWcmwLdVblfWt1AbkYKu90LjcnDs4Gmxgx9NW/Zc+oImVpuEyLyH5xivqEisgn4JY5IfkJErgTWAxf13gy7lpr6xqiIcWWMfeOcpdsjj9OTg4xw7wgZxmAjEJOGkWy2ccYgZkCLZYA9C7P4bHtFt++nur6RrLQkRITNpdF2dVvc52MLMiLL0pKCXZIL1l2V+2U19eSlJzOhMJOlW8sZmeeIhkhkua5vOgascP/XnbkQGUyo6iUJVp3YoxPpAd76bAdffuAjnv3WkUwdmwe0fNGXlmLiwBi8xOYsJ1txnzGIGfDfBvuMymHplnLCvo5ctQ2NzXred5bKuhCZqUnkpic3E7Be0d/wnKYoVVpy58Sy54IRr5q/K9hV6XQbvPFz+zDn+8cwLNuZe1qy85bpq0Uf97zp5J12JsXFGJg8/YnjgT7f55RSlSDFIiCQYjZZxiAmKBZZNgyPAf/u3290LlX1jXz38fkAqCpTfvES1/+3U37tzdhVVc+QzBSy05KapUZsK3fE8ogosRygJo7grA+F2yTkvYr98pruEYXby2sZnpNGalKQSW4KBjgiH5wc7AvueY+i62ZR1kd8af3CZ3dl9+epG/0LzyP2pueXMmuh44yXKB85ORjodm92w+jLxPosm8eyMZgZ8O/+4/caBsDzC7YATTmJT8zdRFl114m83VV1FGSmkJ2W3KzD3fbyWgICQ7NSIsucyHLzVIHJN7zIZfd/2OK+wmGNfMl3R2S5si5EVX0jw3OaF/B5Yrk21MhctxHLquLKZuN6A6/j4IShmVTUhQg1WiqG4bBhV3XU++Ge/60CEkeW60L23jEGN83SMIJ28WgMXga8WC7MTuWKI4oAKLpuFlc/Mi+y7u1VXdNEQVUpqWogPzOFnLQkKmIEbHF5HUOyUknyXZm3lIbx3updLe6v2ve67shZ3l7ePG3EIzXJS8NoEhPaR2zaNpc68957VA5gbYsNh9Lqeo753Rs8O39LZNmW0lp2V9VTVR9iYmEmB43L670JGkYfxIsse3dALS3JGMwMinf/2VNHxV2+wbVF6yxV9Y3UN4YpyHDSMLzI8kdrd/PBml1U1oXISYuupUxLDnY4euWPhnWHIPTE8rCWIss+wV7RR6y1vEJKz+JoZ2X3u6AYfZ8tpc0bBe2uqmfaLXNYsa2CrNQkGtyo8w+tEYlhAP6cZUvDMIxOvftFpEBE5ojISvd3fpwxU0XkfRFZIiILReQLndlnR9hvVG6k+h3gj1+YSm56MlvjfIl2hBLXXi0/w0nDqKwL0RhWLvr7+1x87wcs2lxGRkq0WE5PDkTM3j3akqusqqzd6VjTJQclEll+afE2vvWvT6IKGTuKZxc3JDOeWHbeMv485b5iI+c1mDh84hAAVmzrfhcUo++zq4XGPat3VJGZmhRpUz9uSEbCsYYxmAiadZxhROjsu/864DVVnQS85j6PpRr4sqruC5wG/FFE8jq533aRkhTg2W8dyaNXHsojV87g3INGMyovPZLj2llK3dznfLfArzGskSgnOBHs2G5gaclBakPRYjlewV8s/3h7DRff+wHgWNF5OcvfeHQesxZtbZYv3RE8EZ8Rp4NZapKzrMTX6KUr9tkVVNSFSEkKMH5oJgDFcfy1y2oaTEQPMna1UuyZmZrEWfuPBGCi27HSMAY7zcSy5Swbg5jOiuVzgIfcxw8B58YOUNXPVHWl+3gLTuvcwk7ut0McNWkoR09ydj0qNy2S49pZdld7keXkSNOONTujG5PECs94OcttaX+9aHN55PHkYdlU1oWiosnVDZ0Xri21+w0GhOSgsNtXHLmltGsuOjrCmyuKKbpuFht3V1NRGyI7NSniBR0b8V65vYIDf/UKp/7xrYj1njHwaS0dJzs1ie+dNJlFN51ikWXDcPGyLmotZ9kwOi2Wh6vqVvfxNmB4S4NFZAaQAqxOsP5qEZkrInN37Oia4rtEjMxL68LIsiuW3cgyNOVD7znMiVRlxqRhpCU5Hfz8xXFtyf3N9uU+Tx6RjWr066rrO+9/7EW405Obi2VwGqr422x3Ve53W3hv1c6oAson5zreuQs2lVJZGyI7LYlgQEhPDlIZ0zjl1D++FXl87b8/paa+kbc+6973mdH77PRFlodlN08tykxNIhAQstOSyXEvdg8papZRZhiDimAgupjbcpaNwUyr734ReVVEFsf5Occ/Th3VlzBhVkRGAo8AX1HVuJVtqnqvqk5X1emFhd0bfB6Vl05pdUOXRBijc5YdMeu12PZSAoi5g5XmRm39RX6e6G6JBnf8Xy45iLH56UB05Cw2D7ojeII7LYFYTk0ORooA/eO7g3U7q1hV7KRNlNU0cOl9H3LNvz5pNm53VT0zF2yhyp1LVlpSlIduqDGMP5379eXFHPu7N/jyAx8xf2Np1LZCjWF+8eziSG54R9hZWUfRdbN49tPNcdfPXbebd1ft7PD2jbazy3d+nH3gKH57/gFR6zNToy9kl918Gv++6rAemZth9FUiBX5uumCSpWEYg5hWxbKqnqSq+8X5eQ7Y7opgTwwXx9uGiOQAs4Cfq+oHXXkAHWVUriM041XKt5fdVfUEBHLTm9IwPAE7zm1xHev5m5bU3FWipKp1G7jq+kYmFGbyuQNHMdSNkq3ZURW1vrPUNDSSkhSI5KzFkpoUiDRaSUsOUBfqPrF83J1vctJdTkTYiyh/4vo7P/juWmYtcm5s3PjcEqDpIiUrtcmVZPz1s9jz5y8227aX0/zBml0s2VLG/72+EoD5G0t55IP1XPd02xrXrN9VxcX3vs8by5ve/kff8QYA33t8fkS01zY0MnPBFirrQlzwt/f54n0f8sj761i/q4r3W7ELNNrPM59u4sF31/LkvE2RZRmpSVF3ZwCyUqMvCtNTghZFM3oFETlNRFaIyCoRiVcD5I07X0RURKb7lh3gK6ZfJCLNvT/bQSAmDSM5YOeEMXhJan1Ii8wELgdud38/FztARFKAZ4CHVfWpTu6vyxiZ63yObC2riaRKdJTiCsdHORiQiEWcJ9oKMp1GJKHG6KB7kwVbk4gu8UWWG8MaV6xW14ciKR1j8x0hftXDc6PWd5ba+sa4xX0eacmBiCgflp1GfQ81cPDEr+czfdPzS5uNOW/aaMBpMb52ZxU3fq4Wvw30Z78+naSAMOFnsyPL5m8o5Z43V1NW08DxU4axy71TkJKg+ltVaQwrScEAq4orImL+gzW7mTQsi79celBUseZ+v3w54TH9whX5AKtuPT3Ki9voOBt2VfP9xxc0W56ZEmyWi7/3yJyempZhJEREgsDdwMnAJuBjEZmpqktjxmUD3wU+9C1LAh4FLlPVBSIyBOiUCb/fOi4pIBHfZcMYjHT2m/l24GQRWQmc5D5HRKaLyH3umIuAY4ArRGS++zO1k/vtNKPyvMhy5/OWiyvqIrmQsZHlfdwGGUdPGhr1mvQULx+sSVRVRuUexxe91fWNkS/7ojjFSIkanbSH6vrGhPnKEJ2ekZ+R3C3dzraV1bL3L16KWub9fVRp1vgFnGrtOy84EIAZ4wsAuPQfTd0QT9t3BClJgWYf+i8t2Raxwjvzz+/wnpseUZjVPL8V4Dezl7Hnz18kHNaIUPZYWVzJaX98u83H6WfVjr7RCXEgsHZX/BSapGCgmUXjiXu3WGphGD3FDGCVqq5R1XrgMZwi+lhuAe4A/LdFTwEWquoCAFXdpaqd+jLw0i5qGxrtTosx6OnUGeCekCeq6iQ3XWO3u3yuqn7Nffyoqiar6lTfz/wumHunGJGbhkjXpGEUV9T6xLIT9fXE8r6jcvjg+hO53O0i6OGlYfgjkP7HidIpakPhiFhNCga4//LpUeur6xuZv7G0U6kRNQ0ti+VUX8Q1J71rxfInG0rYVVnH8Xe+GfX3CIc1khsuAtvLmzscNDRqRAj/4qx9gOhW3Lecu1/kcV5GcsI5eCkmz87fzDOfbmq2/h9vrwW6thnLiJy0hOLcaD/+zpaThmVx/rQxAAQFJg/P7q1pGUZLjAY2+p5vcpdFEJFpwFhVnRXz2smAisjLIvKJiPwk3g7aU0QfEL9YtqiyMbgZtJeLycEAhVmpXRNZLq+j0BXLGSlBggFhZ4Uj7FKTgq4wj/6widcJz1+cl0gs1zU0RonV4/caFrV+xbYKzr37XX4VJ0WhrdQ2NCYs7oPoyHJ2WlKX5Syv21nFeX99jysfmtvMc7qiLsTH63YDTr55WSttvgvjuB74lz1/7VEJX7t0q2PPF1b4/uML+NGTCyKNWh56b11k3MO+xwDfO2lSi3Nqibd/ejxDTCx3GeW+Ow/jh2ZGmukEAsLYggzW3nZGb03NMDqEiASAu4AfxlmdBBwFfNH9/XkROTF2UHuK6Jt8lsMWWTYGPYP6DBiTn87mTorlcFjZVVXPsGwnB1pEyEpNiog970s6lng5y9VRYjl+1LI+FI4Sy7EpBZvc41m8uay9hxI1j5Zylr11SQEhPTmJuoauiSx7ndZi3SnAiRRuLWu6C+AXQ+/89HjAiTh7xLYXX/KrU6Oejy3I4ImvH87L3zum2b427o5+Tzw1bxO3vLCU0up6fjmzKcf493M+ixr33RObxPIPT57M7O8cze8vPLDZ9j32ciOc+4/OtS+jLqC4vJYjbnuNxz7aQHlN0/mTm54ccULxomUiQjAgHDahoDemahjx2AyM9T0f4y7zyAb2A94UkXXAYcBMt8hvE/CWqu5U1WpgNjCtM5PxxLJX8G0Yg5nOFvj1a8YWZDDPdVboKBVua2v/bf3stKRI5DORkbsnomsTpGEksoCr86VhxMNrxBGQjt82q2lojDT2iIfXujs1KUCq64bx+b++y5QR2dx23gHNxu+oqOOZTzdx1dETmkXY/cTrBHjO1FE8N38L//5oAy8sdJwvNpfWcOusZZExI3PTyc9I5rrTp0SW+fdz+eF7NLMHg6a85oU3ncIBN72ScF4Az3y6mWcS2MD59/nkNw6npKqeU/YdAcDeI7P54ZMLOGxCAY9dfThLtjitzz1LwV2VdXGbvxjt47n5m/nuY/MB+OOrK/n8tKa719P2yI8U3A7NSoksX3Xr6T06R8NohY+BSSIyHkckXwxc6q1U1TIgUvwiIm8CP1LVuSKyGviJiGQA9cCxwB86Mxm/WG4pbc0wBgODWiyPK8jghYVbaWjs+G0mr9DMa2YAXpFfDalJgYTisLU0jN1V9XHTIepC0WkYsXhFcN4H3crtFVTXN3Lg2Lw2H1NNfSNDW0gJ8IRnanKQ/IxkdlbWs7Oynk83lDYTyyVV9Xzpvg9Zsb2C4/Ya1mK+aFVd8wuEsw5wxPI9b0b3sfFykdOSHYu7T288JeF2rzt974TrwPnfvfS9o7ns/o8iouqyw/bgkQ/Wt/i6V39wDBOGZvHs/M2RL5NDiqIjlSLCOz89PuKKsu+o3Kj1lnrReeatL+G7j80nOzWJiroQITe/PSDw1y9O49R9RxAKKxMKMznVvYgBWrxwM4yeRlVDInIt8DIQBB5Q1SUicjMwV1VntvDaEhG5C0dwKzA7Tl5zu0hyreLqQ2Hr3mcMega1WB6bn0FjWNlaWtvhNrfe7V6/d2t2alPkNRGeCH7kg/VMLyqgMDuV3VX1BANCY1i5+pF55GckNxOBdQ1hUpPiRyLzMpIjhU2e7c/Jf3DcGtbdfmabj6myLkRmC9FOb11qUqCZ+FPVKBFy0C1zoubeElUxBXN7j8zh4D1a7qS25FenJVz39k+Od7r5tSFyO2VEDg9/dQZPzt3EDWfuzfrd1S2K5ae/eQR7DnOE/3lu8VgixuRbC+Xu5Px73gPguydNoiAzhR88sYDHPt5IenKQ0/YbCThOKWcdMKo3p2kYraKqs3FSKPzLbkww9riY54/i2Md1CX6BbGkYxmBnUJ8BY92GIRtLOt6u2YssZ6dFp2GAE3lNhJfm8N7qXfz1zVWRefgjryXVDVHtsMHpppQakwf932uO4JlrjiA9OUhJtTOfQMDJp+4IJVX1FGQmjnhmuHNPDgYY7VrwebTkEOGlplTVhaJs8hK9dlh2aiQi63HDmXtHXZgkapwCzv93VMz8WmLvkTnc+Ll9CASEcQUZHDgmt4Wx5qjQV/A80y89dBz7j276n8UWiRqG0XaSk5o+W00sG4OdQX0GjC1whNSG3R0Xyztci7ghvlxIz5/Su6UfD78IrG0Io+pEuCcWZkaNW7CpqVCvMaw0NGrEds5j2rh8DhqXT3pKMNLYJCBCVRsblHhNNpy5NFJV30hBZuIctSHu3LeU1jA8J7pJ1PayxFZ8nlie/utXmeaLOHvERpb/+IWpzcZ87egJzHSdLPz5p11NMCA88Y3Dmy3/6pHjeezqwyJ520bv0tAYpriijmuP35OMlKTIBTDA14+Z0IszM4z+jT810dIwjMHOoD4DRuamkxQQNnZCLHtC2/8lHdutLx7+iGh6cpCq+kZqGhopGhItll902zkDkU55sZFl/3Y80bt2Z1VUwdz976zlf5/F99W87cXlTPzZbMJhpdSNTOdnJhaiJ0xx7OpCYW1m0eZ5FEPzFt+eGK5paKQ+FKYxrPzyucVs2FUdWZ+aFODDn53IwptOSTiH8UMz+b9LD+LJbxyRcI5dQWpSkEevPJT/XHUYJ04Zxt++dDDXnT6FwyYM6db9Gm1nc0kNjWGlyC2Y9Of4X39Gy7nqhmEkxh9NtsiyMdgZ1OGxYEAYnZ/eqcjyyu2VDMtOjXKP+O5Jk3hteXGbt9EYDrPTjULH5k7f985a/v7WGhb88pRISkaiXGh/I5GtZbW8tHhb5PktLyylIDOFT35xcrPX3fvWGsARul7BYWYLkdMx+U5Efsb4AoIBYUJhJgKs3lHFNl9kOdaWrzbUGGWJt3hzGQ+9v57FW8p56Ksz+Ls7j9hodTx6Kv/0KLfz4uETB59Adu2pKoBGIKSq01t+Rc9SU9/IB2t2AdHdLG/9/H7kZ3TfXQfDGAz4o8kt1d8YxmBgUItlcBwxNpZ03Gt50eYyDojJbT1gTF6bXvvvqw7l0n98SHV9Iz97ZhHgdHLz40WKl2wuY+KwLICEBX6xhWxvrUzcoWn2oq386dWVPPOtpujs2yt3RCznEvlDQ5PDgydIXv/hcdQ2NDLlFy+x3RdZfmul0zb6yW8czoV/e5/ahkbW7GjehriqLsRsXwQ9lrd/cjyl1Q1MHJaZcIzRbRyvqjt7exLxOOPPb7N2p/N+2sN3R+aLh+7RW1MyjAFDshX4GUaEQS+Wx+Rn8PKSba0PjENVXYjVOyo564CRHXr9EROHMmFoJtvKa3lvtRMh86c1JAeFBjelY/m2Cka7Ed1EV/mxNnNvrogWy42+gr9v/+dTGsPKyu1N7aB/+vSihNuKJdbhIS05SF5GclQaxuaSGlKCAaaNcxwtaurDPDe/yav4ty8vB5y805Z8nccWZDDWekcYPooraiNCGbo3f90wBiPBgBAQp5Op5Swbg51BL5bHFWSwu6qeyrpQi4ItHku3lqNKVAW+x7PfOpKkFpwaPLLSkiJCGaILBScPz2bJFqf18pIt5ZGUgERCtqWuexDtDjAkM4XiijreXR0/aJjeiliOx4icNLaVNRU1VteHyEh12n+nBAM8/P46drltowHeXeUct/f3B/i/Sw9q936NbkOBV0REgb+r6r2xA0TkauBqgHHjxvXYxLzceoAJQzPNM9kwuoHkYIC6UNgiy8agZ9CLZc8RY+PuavYemdOu125xc3L3GNI8PWBqG5uADM1KjUR8f3HWPpG22QB7jWgSyyu2l0d8ituSsxyPhsYwW0pr+GjtbobnpFFcUcdvX1oRd2xrkeV4jMpLZ+mWMqrqQmSmJvHw+00+xanJgSih7KekuiHiD330noXt3q/RbRylqptFZBgwR0SWq+pb/gGugL4XYPr06R3zKuwAnrPKny85iGMn23vGMLqDlCQTy4YBg9wNA4j4BG8pbX/estdxzu/7214K3Q5u+RnJXHnUeAA+d6BTvDZlRJOX75bSWupCzv4SuWG0JnBV4bcvLed7j89n0eayFsd2pAXz5UcUsaWslhcXb4sq5IP4raw9RIiI5axO/C2NrkVVN7u/i4FngBm9O6MmvPfLHgUZ5KZbK17D6A689IuUYPu/DwxjIDHoxfLIXEcs+3Nt20plnfOFndnO9A0/ewx1cn/9+cR/uOhAlt58alTL6ZLq+oj1W2sFfkkBSegxu6mNxYyxzUDawlF7Omkit81eRmUL4th/EQCOiN9cWkt2alKLTUaMnkNEMkUk23sMnAIs7t1ZNeFFlnNMKBtGt+H1DLDIsjHYGfRnwNCsFERabqaRiMraECKQ0YGUBY8jJjoCs9wnLpOCATJSkjhn6mh+dfa+/PyMvVGFv7zudPpLlFvtzWN6UT4/PW1K3DENjS23nPbwC/W24gndXVX1fLRud9S6c6c60fILDh7DX784LbJ8eI6zn40l1Z2K0BtdznDgHRFZAHwEzFLVl3p5ThG8yLJFlQ2j+0gKuJFlE8vGIGfQnwFJwQDZqUmRSBU4BWfzN5a2+trKukYyU5IIdCIauv/oXI6ZXMifLp7abF0wIFx+RBEj86Lt5PIy4gsEL7IcDkMgIFx9zAS+dFh00dXOyqa84e7M9fR8o28/b3+AiJPHBQePYUJhVmTc7y+cCsAn60ssStiHUNU1qnqg+7Ovqt7a23PyU1bjXFzaBZZhdB+tefsbxmChU980IlIAPA4UAeuAi1S1JMHYHGAp8KyqXtuZ/XY1uRnJUWL53LvfZcPuatbedkakyr62oZFPN5Ry2ISCyLJt5TXNOti1l2BAePirLaeCDsmM3keiaJrXgrnejR7/7Iy9UVUe/WBDZMxuX5Fda+4ZHWFoVio7K+siDUk8YfydEydx8B75ke53Pz9jbzaX1nDQuDzA6QaYk2Zi2Wgb5bUNZKYEo7xgDcPoWhpdsWzWccZgp7NnwHXAa6o6CXjNfZ6IW4C3Wljfa+SmJ1NeG2Lj7mp+/syiSEc/vz3VLS8s5ZJ/fMCJv/9fpPHGup3VUZ3DuotYD9lEIrcg0xGb/uI6EeGbx03kq0c6xYOefdyh4wv4/smTu3yuj3/9MAA27nbEspcykpoU5IQpwyPjrjpmAjedvW/UseQmiJgbhp9HP1jP/e+sJdsurgyjW/FKaSwNwxjsdPYMOAd4yH38EHBuvEEicjBODuQrndxft5CTlkxxRS0X/u19/vVhUxR2e0VTHvMnG0oBWLOzikN/8xrgNEYYkdt6a+bOMszX1e/606ck9JT1OurVNkTnJf/0tCl87sCmxikXTR/D418/nMnDs1l3+5ksv+U0Lj20azxy89yo96ZS54Kjtdvk/mOx0j6jLdzwrFNn6HlzG4bRPXhpGCaWjcFOZ8+A4arq9SnehiOIoxCRAPB74EetbUxErhaRuSIyd8eOxK2au5qMlCCLN5c3c8TYXt7UYGNHRR0n7d10eOGwUlLd0CHXiPbiT7s4bb8RCccNdVNC4nUU9H/YDYkp3ktLDnLz2fsy74aTADjabX7Smbludl032pNTevr+iY/NMGIxsWwMNETkNBFZISKrRCThnVoROV9EVESmxywfJyKVItLq920bZwTQpgZbhjGQaVXJiMirQDwV83P/E1VVt9NXLNcAs1V1U2tdtnqrwUF6StOf4ZjJhbzlWrQVu+K5Mazsrqpj75HZFGanMGfpdspqGmgMa7N84u5iRE4a28prGZufOO1jYmEWL373aPYant1snd9uLl7HwaRggCFZqSy/5bROfTAmBQNkpSZRUt1+W73PHzSmw/s1Bgf1oba5uRhGf0NEgsDdwMnAJuBjEZmpqktjxmUD3wU+jLOZu4AXu2pO3leBWXoag51WlYyqnpRonYhsF5GRqrpVREYCxXGGHQ4cLSLXAFlAiohUqmpL+c09SrqvyccfvzCV2oZGjrj9dYpdR4ddlXWEFQqzUwmFlbKahkgB2/Cc7k/DAHj5+8ews7KuVeeNRF0I/dXME31uFLF0pHNfLLnpyVTWhUhPblsB1t++NC1SnGgYiXji44385OmFkecdsTc0jD7MDGCVqq4BEJHHcFIdl8aMuwW4A/ixf6GInAusBaq6akIBN8BlYtkY7HQ2DWMmcLn7+HLgudgBqvpFVR2nqkU4qRgP9yWhDE0uErnpyRRkpjAqL53koFDl3uZds9P57CkakkluejINjcpn2ysA2KMHCvy8ubUkclvDL5a7wwXDj2dt19ZufKftN5JjrGWx0QpPzN0YeXz8XoU8d+2RvTgbw+hyRgMbfc83ucsiiMg0YKyqzopZngX8FPhVV07I08iBVu4KG8ZAp7Ni+XbgZBFZCZzkPkdEpovIfZ2dXE/hRVO91tfgCOjqesc5Ys0ORyxPKMyM5OQu3OS0ix7XQ2K5s6T6IsY9JZbNA9doK+GwcsdLyzn37ncjNo7V9SGufPBjHvvIKbrd7GtJf+zkwqjz1TAGOm79z13AD+Osvgn4g6pWtrKNdtUFiUWWDQPopM+yqu4CToyzfC7wtTjLHwQe7Mw+uwNPPHrd5AAyU4KRyPLybeWkJQcYlZsecXt48L11pAQD/cYb2N/1r7tTHvLSnaLH7E60ATcGF++s2sk9b64G4Kl5m/jioeN4Y/kOXltezGvLixk/NJOtvi6b+T1QWGsYPcxmYKzv+Rh3mUc2sB/wpitiRwAzReRs4FDgAhH5LZAHhEWkVlX/z7+DjtYFmVg2BjumZoB8NxLqz9fNSG2KLL++vJhDxw8hEJAoZ4r6NraO7gv4P+zSkrvXBig/s31pGMbgpj4U5kdPLqAgM4WctCRueWEpv5m9jP18hahfuPcDUpICXHX0eO5+Y3XUOsMYIHwMTBKR8Tgi+WLgUm+lqpYBEasiEXkT+JEbnDrat/wmoDJWKHcEt9s1QUvDMAY5Zp4IBN1PhHRfekJmSpAqt7nH7qp6Jg938oX9tms3nLl3D86y62jNlaSzFA3JdPZjzslGAhrDykuLt7K9vJbvPzGf4oo6vnjoOA4pKoisX7CxlFP2Gc5F0x2XlGnj8vjhyXsx/8aTO5W/bxh9EVUNAdcCLwPLgCdUdYmI3OxGj3scK/AzDAcL/dFkpXbm/k3+xBkpSVTXNVIXaqS6vpE8t+FH0dCmHOXLjyjq0Xn2FzzBc8AYi/4Z8blp5hIe+WB95PnovHS+dfyefLa9grdW7qAuFKa0uoGvHT2BcQUZbCmt5dsn7EkgIJFz0TAGGqo6G5gds+zGBGOPS7D8pq6ajyeWW3NhMoyBjollYP8xuSz51alRnsCZqUG2ltVGio289Au/X3FbbNH6Eh/9/ERSemDOB47N49lvHRnX79kwXlmyjUc+WM9ew7OZXpTPsZMLOXmf4YgIB4zJ48OfnURFbQMbdlez7yjnguvRrx3ay7M2jMGHdxPS0jCMwY6JZZfY5hmeG8bWUqeoaFh2U/pFYXYqOyrq6G8My+4ZT2iAqWPzemxfRv/iqXmbGJadyvPfPiphG93stOSIUDYMo3doiiz38kQMo5cxsZyAzFTHDWPNTseJZ4IvR/KNHx1HqB8V9xlGX2Hj7mreWFHMZYcVJRTKhmH0DQIWWTYMwMRyQrzI8uriKpICEtV8JMss0QyjQzwxdyNhha8fO6G3p2IYRitYgZ9hOFhoJwGeG8aq4krGFWT0u/xkw+hrlNU08MTcjRw8Lr/H2sQbhtF5rMDPGOyYAkxATnoyqrBwU2lUCoZhGB3j0Q/WU1xRxw1n9U/LRcMYbHg2o0kmlo1BjonlBOS47hdbymqZOCyzl2djGP2fFdsqGJ2XzgFj8np7KoZhtAFPI5tnvjHYMbGcgDxfp76JQy2ybBidZUtpDWPy03t7GoZhtBEvZ1lpc2dswxiQmFhOgL+ttUWWDaPzVNaFyElLbn2gYRh9Ai+yHDatbAxyTCwnwN8lbIJFlg2j01TWhcxJxjD6EV7OclhNLRuDGxPLCfBHlvMzrb2uMfgQkdNEZIWIrBKR6zq7vaq6EFlpJpYNo7/gRZbVxLIxyDGxnIC8DLtdbAxeRCQI3A2cDuwDXCIi+3Rmm5V1oWadMg3D6LsEIpHlXp6IYfQyJpYTkJYcBOD8aWN6eSaG0SvMAFap6hpVrQceA87p6MbqQo00NKqlYRhGP+LsqaMAGJuf0cpIwxjY2DdXCyy/5TRrRmIMVkYDG33PNwGHdnRjyYEAb//keBPLhtGPuOywPbho+thI8MgwBiudUoIiUiAic0Rkpfs7P8G4cSLyiogsE5GlIlLUmf32FGnJQWvzaRgtICJXi8hcEZm7Y8eOhOMCAWFsQYbl/xtGP0JETCgbBp1Pw7gOeE1VJwGvuc/j8TDwO1XdG+f2bnEn92sYRveyGRjrez7GXRaFqt6rqtNVdXphYWGPTc4wDMMweorOiuVzgIfcxw8B58YOcIuCklR1DoCqVqpqdSf3axhG9/IxMElExotICnAxMLOX52QYA5q2OtCIyPkioiIy3X1+sojME5FF7u8Tem7WhjHw6axYHq6qW93H24DhccZMBkpF5L8i8qmI/M6ttG9GW2/pGobRvahqCLgWeBlYBjyhqkt6d1aGMXBpqwONiGQD3wU+9C3eCXxOVfcHLgce6f4ZG8bgodVqGxF5FRgRZ9XP/U9UVUUknsFMEnA0cBCwAXgcuAK4P3agqt4L3Aswffp0M6sxjF5EVWcDs3t7HoYxSIg40ACIiOdAszRm3C3AHcCPvQWq+qlv/RIgXURSVbWue6dsGIODVsWyqp6UaJ2IbBeRkaq6VURGEj8XeRMw3/cB8CxwGHHEsmEYhmEMUlp1oBGRacBYVZ0lIj8mPucDn8QTyiJyNXA1wLhx47pk0oYxGOisj9NMnFs+t7u/n4sz5mMgT0QKVXUHcAIwt7UNz5s3b6eIrG9l2FCc208DiYF2TAPteKD3jmmPXthnmxmk5+xAOx4YeMfUm8fTZeesiASAu3DuzCYasy9O1PmUeOv9d29FZIedrwOCgXZMffJ87axYvh14QkSuBNYDFwG4RQffUNWvqWqjiPwIeE2cRvPzgH+0tmFVbbW0XkTmqur0Th1BH2OgHdNAOx4YmMfUFQzGc3agHQ8MvGPqR8fTmgNNNrAf8KbzVcoIYKaInK2qc0VkDPAM8GVVXd3azux8HRgMtGPqq8fTKbGsqruAE+Msnwt8zfd8DnBAZ/ZlGIZhGAOYiAMNjki+GLjUW6mqZThRNwBE5E3gR65QzgNmAdep6rs9OWnDGAxYezrDMAzD6GUSOdCIyM0icnYrL78W2BO4UUTmuz/DunnKhjFo6O+9Z+/t7Ql0AwPtmAba8cDAPKaeYqD97Qba8cDAO6Z+czzxHGhU9cYEY4/zPf418OtumFK/+du1kYF2PDDwjqlPHo+omkObYRiGYRiGYcTD0jAMwzAMwzAMIwH9Viy3tS1oX0JExorIGyKyVESWiMh33eUFIjJHRFa6v/Pd5SIif3aPcaHrsdnnEJGg253xBff5eBH50J334267ZEQk1X2+yl1f1KsTT4CI5InIUyKyXESWicjh/f1/1NvY+dq3GEjnrJ2v3YOds32HgXS+Qv88Z/ulWJY2tgXtg4SAH6rqPjiNWb7lzvs64DVVnQS85j4H5/gmuT9XA/f0/JTbxHdxClI87gD+oKp7AiXAle7yK4ESd/kf3HF9kT8BL6nqFOBAnGPr7/+jXsPO1z7JQDpn7XztYuyc7XMMpPMV+uM5q6r97gc4HHjZ9/x64PrenlcHjuM54GRgBTDSXTYSWOE+/jtwiW98ZFxf+cHxAn0Np9nMC4DgGIonxf6vcKq8D3cfJ7njpLePIeZ4coG1sfPqz/+j3v6x87X35x5zHAPmnLXztdv+rnbO9pGfgXS+uvPql+dsv4wsE78t6OhemkuHcG+PHAR8CAxX1a3uqm3AcPdxfzjOPwI/AcLu8yFAqTo2SBA958jxuOvL3PF9ifHADuCf7m2v+0Qkk/79P+pt+v3faACdrzCwzlk7X7uHfv93GkDn7B8ZOOcr9NNztr+K5X6NiGQBTwPfU9Vy/zp1Lp36hUWJiJwFFKvqvN6eSxeSBEwD7lHVg4Aqmm4HAf3rf2R0noFyvsKAPGftfDWaMVDO2QF4vkI/PWf7q1hurS1on0VEknFO4n+p6n/dxdtFZKS7fiRQ7C7v68d5JHC2iKwDHsO5TfQnIE9EPA9v/5wjx+OuzwV29eSE28AmYJOqfug+fwrnxO6v/6O+QL/9Gw2w8xUG3jlr52v30G//TgPsnB1o5yv003O2v4rlSFtQtwr0YmBmL8+pVUREgPuBZap6l2/VTOBy9/HlOHlW3vIvu9WghwFlvtsUvY6qXq+qY1S1COd/8LqqfhF4A7jAHRZ7PN5xXuCO71NXj6q6DdgoInu5i04EltJP/0d9BDtf+wgD7Zy187XbsHO2DzDQzlfox+dsTydJd9UPcAbwGbAa+Hlvz6eNcz4K59bCQmC++3MGTk7Ra8BK4FWgwB0vOBXJq4FFwPTePoYWju044AX38QTgI2AV8CSQ6i5Pc5+vctdP6O15JziWqcBc9//0LJA/EP5Hvfw3tfO1j/0MlHPWztdu+7vaOduHfgbK+erOs9+ds9bBzzAMwzAMwzAS0F/TMAzDMAzDMAyj2zGxbBiGYRiGYRgJMLFsGIZhGIZhGAkwsWwYhmEYhmEYCTCxbBiGYRiGYRgJMLFsGIZhGIZhGAkwsWwYhmEYhmEYCTCxbBiGYRiGYRgJ+H9I/iQYxUpuqgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75d85343-2876-4c3b-9167-f11262d15287",
   "metadata": {},
   "source": [
    "This is looking much better: the activations evolve smoothly and we've managed to prevent half of the activations from vanishing. Since batchnorm claims to work with large learning rates, let's ramp this up to 0.1 and see what we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "37ac31bb-445b-406f-90cd-a18d37138346",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lewis/miniconda3/envs/dl4phys/lib/python3.9/site-packages/fastai/callback/core.py:67: UserWarning: You are shadowing an attribute (modules) that exists in the learner. Use `self.learn.modules` to avoid this\n",
      "  warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.241621</td>\n",
       "      <td>0.324524</td>\n",
       "      <td>0.879571</td>\n",
       "      <td>0.951736</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.232091</td>\n",
       "      <td>1.657844</td>\n",
       "      <td>0.655914</td>\n",
       "      <td>0.913255</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.222661</td>\n",
       "      <td>0.819799</td>\n",
       "      <td>0.612086</td>\n",
       "      <td>0.906052</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.220339</td>\n",
       "      <td>0.276002</td>\n",
       "      <td>0.885429</td>\n",
       "      <td>0.957145</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.217905</td>\n",
       "      <td>0.410375</td>\n",
       "      <td>0.813257</td>\n",
       "      <td>0.920187</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.209055</td>\n",
       "      <td>0.433610</td>\n",
       "      <td>0.802971</td>\n",
       "      <td>0.920204</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.204782</td>\n",
       "      <td>0.363477</td>\n",
       "      <td>0.832943</td>\n",
       "      <td>0.945951</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.198509</td>\n",
       "      <td>0.206748</td>\n",
       "      <td>0.913429</td>\n",
       "      <td>0.972911</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.191656</td>\n",
       "      <td>0.202564</td>\n",
       "      <td>0.916000</td>\n",
       "      <td>0.974434</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.182151</td>\n",
       "      <td>0.196292</td>\n",
       "      <td>0.918886</td>\n",
       "      <td>0.975237</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit(10, lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10359325-adec-4f54-989f-d1b9b89940d1",
   "metadata": {},
   "source": [
    "Great, this has given us a small boost and only took a few minutes to train a CNN form scratch!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdbb5628-c819-4469-810e-a55fb4e2c621",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "* Implement the same CNN architecture from the review. Can you get close to their results?\n",
    "* Read the [1-cycle policy training paper](https://arxiv.org/abs/1708.07120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bcf2a4a-72d0-4efe-b569-f2ada3dfd2d5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}