all_learn_process/codes/chapter4/第四章学习.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第四章 神经网络工具箱nn\n",
    "上一章中提到，使用autograd可实现深度学习模型，但其抽象程度较低，如果用其来实现深度学习模型，则需要编写的代码量极大。在这种情况下，torch.nn应运而生，其是专门为深度学习而设计的模块。torch.nn的核心数据结构是`Module`，它是一个抽象概念，既可以表示神经网络中的某个层（layer），也可以表示一个包含很多层的神经网络。在实际使用中，最常见的做法是继承`nn.Module`，撰写自己的网络/层。下面先来看看如何用nn.Module实现自己的全连接层。全连接层，又名仿射层，输出$\\textbf{y}$和输入$\\textbf{x}$满足$\\textbf{y=Wx+b}$，$\\textbf{W}$和$\\textbf{b}$是可学习的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.592464Z",
     "start_time": "2020-12-10T06:06:02.289049Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch as t\n",
    "from torch import nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.599190Z",
     "start_time": "2020-12-10T06:06:02.593207Z"
    }
   },
   "outputs": [],
   "source": [
    "class Linear(nn.Module): # 继承nn.Module\n",
    "    def __init__(self, in_features, out_features):\n",
    "        super(Linear, self).__init__() # 等价于nn.Module.__init__(self)\n",
    "        self.w = nn.Parameter(t.randn(in_features, out_features))\n",
    "        self.b = nn.Parameter(t.randn(out_features))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = x.mm(self.w) # x.@(self.w)\n",
    "        return x + self.b.expand_as(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.615178Z",
     "start_time": "2020-12-10T06:06:02.601185Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1640, -0.5664,  0.8481],\n",
       "        [ 1.4325,  0.4646, -2.2152]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = Linear(4,3)\n",
    "input = t.randn(2,4)\n",
    "output = layer(input)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.623157Z",
     "start_time": "2020-12-10T06:06:02.616146Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w Parameter containing:\n",
      "tensor([[-0.2084, -0.7339,  0.9232],\n",
      "        [ 0.2622, -0.0191, -0.2329],\n",
      "        [-1.4246,  0.7418,  1.4044],\n",
      "        [-0.0562, -0.9471, -0.8934]], requires_grad=True)\n",
      "b Parameter containing:\n",
      "tensor([ 0.3705, -1.0313,  0.4124], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "for name, parameter in layer.named_parameters():\n",
    "    print(name, parameter) # w and b "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，全连接层的实现非常简单，其代码量不超过10行，但需注意以下几点：\n",
    "- 自定义层`Linear`必须继承`nn.Module`，并且在其构造函数中需调用`nn.Module`的构造函数，即`super(Linear, self).__init__()` 或`nn.Module.__init__(self)`，推荐使用第一种用法，尽管第二种写法更直观。\n",
    "- 在构造函数`__init__`中必须自己定义可学习的参数，并封装成`Parameter`，如在本例中我们把`w`和`b`封装成`parameter`。`parameter`是一种特殊的`Tensor`，但其默认需要求导（requires_grad = True），感兴趣的读者可以通过`nn.Parameter??`，查看`Parameter`类的源代码。\n",
    "- `forward`函数实现前向传播过程，其输入可以是一个或多个tensor。\n",
    "- 无需写反向传播函数，nn.Module能够利用autograd自动实现反向传播，这点比Function简单许多。\n",
    "- 使用时，直观上可将layer看成数学概念中的函数，调用layer(input)即可得到input对应的结果。它等价于`layers.__call__(input)`，在`__call__`函数中，主要调用的是 `layer.forward(x)`，另外还对钩子做了一些处理。所以在实际使用中应尽量使用`layer(x)`而不是使用`layer.forward(x)`，关于钩子技术将在下文讲解。\n",
    "- `Module`中的可学习参数可以通过`named_parameters()`或者`parameters()`返回迭代器，前者会给每个parameter都附上名字，使其更具有辨识度。\n",
    "\n",
    "可见利用Module实现的全连接层，比利用`Function`实现的更为简单，因其不再需要写反向传播函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.629139Z",
     "start_time": "2020-12-10T06:06:02.624124Z"
    }
   },
   "outputs": [],
   "source": [
    "class Perceptron(nn.Module):\n",
    "    def __init__(self, in_features, hidden_features, out_features):\n",
    "        nn.Module.__init__(self)\n",
    "        self.layer1 = Linear(in_features, hidden_features) # 此处的Linear是前面自定义的全连接层\n",
    "        self.layer2 = Linear(hidden_features, out_features)\n",
    "    def forward(self,x):\n",
    "        x = self.layer1(x)\n",
    "        x = t.sigmoid(x)\n",
    "        return self.layer2(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.636092Z",
     "start_time": "2020-12-10T06:06:02.631136Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layer1.w torch.Size([3, 4])\n",
      "layer1.b torch.Size([4])\n",
      "layer2.w torch.Size([4, 1])\n",
      "layer2.b torch.Size([1])\n"
     ]
    }
   ],
   "source": [
    "perceptron = Perceptron(3,4,1)\n",
    "for name, param in perceptron.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，即使是稍复杂的多层感知机，其实现依旧很简单。 构造函数`__init__`中，可利用前面自定义的Linear层(module)，作为当前module对象的一个子module，它的可学习参数，也会成为当前module的可学习参数。\n",
    "\n",
    "module中parameter的命名规范：\n",
    "- 对于类似`self.param_name = nn.Parameter(t.randn(3, 4))`，命名为`param_name`\n",
    "- 对于子Module中的parameter，会其名字之前加上当前Module的名字。如对于`self.sub_module = SubModel()`，SubModel中有个parameter的名字叫做param_name，那么二者拼接而成的parameter name 就是`sub_module.param_name`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为方便用户使用，PyTorch实现了神经网络中绝大多数的layer，这些layer都继承于nn.Module，封装了可学习参数`parameter`，并实现了forward函数，且很多都专门针对GPU运算进行了CuDNN优化，其速度和性能都十分优异。本书不准备对nn.Module中的所有层进行详细介绍，具体内容读者可参照官方文档[^1]或在IPython/Jupyter中使用nn.layer?来查看。阅读文档时应主要关注以下几点：\n",
    "\n",
    "- 构造函数的参数，如nn.Linear(in_features, out_features, bias)，需关注这三个参数的作用。\n",
    "- 属性、可学习参数和子module。如nn.Linear中有`weight`和`bias`两个可学习参数，不包含子module。\n",
    "- 输入输出的形状，如nn.linear的输入形状是(N, input_features)，输出为(N，output_features)，N是batch_size。\n",
    "\n",
    "这些自定义layer对输入形状都有假设：输入的不是单个数据，而是一个batch。输入只有一个数据，则必须调用`tensor.unsqueeze(0)` 或 `tensor[None]`将数据伪装成batch_size=1的batch\n",
    "\n",
    "[^1]: http://pytorch.org/docs/nn.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面将从应用层面出发，对一些常用的layer做简单介绍，更详细的用法请查看文档，这里只作概览参考。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 常用神经网络层\n",
    "#### 4.1.1 图像相关层\n",
    "\n",
    "图像相关层主要包括卷积层（Conv）、池化层（Pool）等，这些层在实际使用中可分为一维(1D)、二维(2D)、三维（3D），池化方式又分为平均池化（AvgPool）、最大值池化（MaxPool）、自适应池化（AdaptiveAvgPool）等。而卷积层除了常用的前向卷积之外，还有逆卷积（TransposeConv）。下面举例说明一些基础的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.738847Z",
     "start_time": "2020-12-10T06:06:02.637088Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADICAAAAACIM/FCAABaCUlEQVR4nGX9+dMlWZIdhh13vzci3vJtuWfWXl29THfPTnAGhiFpBAmSv0lmNOnvk2SizARBIo0gODAAJAAS2wCD6Zneqpfaq3Jfvu1tEfe6H/1w42uMTFXWbVkvX+b3XkRcX46fc1z+b4Q6QysI0WzakbHfXr68KF0yMVdQAddsqoYUWYhERyiQAiIAQBpDRQgKPCAQIagC8UhpcgNcBK4UEuIenHzsVkfDkLLGASIK0QqCsFANk8pbby9TwuGLf/vSug4JISpVQRWRCJiGsAT7949fKwAQIPDXfkWQ7T8J3vwHfvMWEIC0/5ebl29ex///2+e/6T/8Dv/a/37zJs7vkr/+OQAAqn/tvQBF8Nd/hIB1TEGQDgkh1BUKIFgLyIpkTgBKAcIgAkcIEAAURABCyvxXkAChUBIQcQqBgEQVCQjIaB9Iov0hcS+dawgsEEpRYbtCDDAQu11vlHxy93UNigsEFIgI2s8jBIYsX5w/TlUgAKABhKi2T+UAQ8iqFEEAEISAImRA5D/cthABIRQhQkCGQoKCAOevhri5T4RDKUFoBcTqIWUzV1VQDAQMQSGUAEP2rwdT1eP3Xj33mlTIUKGAFEAgAkhO/vrTV1UFAapCBAJTIUmG0wkg2oNPBCEASaoACIoCArKCQZFo34sUiJAi858QFSCCYgKIqApGUFQAFRErh1I9ov2WJlWBiChIkkRcXhanDPffG9w9CAgDEYEAKBKgZXn56+dFVKAkBIRIyiZgeIVQRYQmqiqAMDhfWSJIantIBcIQBaEQhqpAhRARaQ+JqoooQCoEJAUhEAUSwKScxogIWDJTtZRERNohCAIyvdk5oet3HygjIoJQiCAEKgKo2fj4kyuKKtGeA4CaVILe3h9gUAJkQNoTQpkvsghBiAg1iQhFlYQIBRRRUUFA2/lUEQjkJnpQlxDQlBAzUR+niHAkE1FARNQAUMEAweuLKYh8+8Mj92ihQBRQAVQt5Tx++fHLHRhKiEBUINrCJ52E5qwQDWE7QSKYQ14EICJAu3ZBRnsMCBAipLPFGWmfvT21FqCogqRIkBBVcUjUqQQBigEqUEBM20cSitRXm0rK4uEjiXAQAm1PqahatsOXn1yZKaAIQCAQAiKMYIQIIICAkhntngmCBCER83GIYDv0ZASChEQ4Jeik0ANksN2MCECiBohgiDAoCYBoTI52oNrfrHQSaHEtgM2bQw3Y6QenQhJRwABpydTMDp9/uhGTUNP56EFUVSFkBCAa1YMgnKQoqHBSBC6miBa0IoI3QSwAUQToU7h7jSBB0quThLTDSYIS7aaSEoQo3NniJiJICKM9BO2GxpurQkp//0NzRwvQZLiHmOw/e3wASsBMoWCLzikJAA9AAK8eZY7IjDnACkXbPQgyKCoCMjxACsRMIiI85ntDtsshIsJ2d1r+NJIUUxFCNcIBEgSjvYkqIqIiIpDDm9EJO3r/bkS4pnbIIyDYf/rLqygeFGPSFn2gkuzmxotSVLUFwzl9U4DQVnmIMCDzmTBSAIRoey0oIolAQKESyrl6kBbABaAoolIUmO+nQCUACFtGa185hJB4c9qbSnfno9cTWUxBCE1N9r/+dGuSYCLepRZEAbU5QLasrxBJYJjOFQp1zoqirUAIbXEA0p5sDSGgFSACQgEDMCGpjJv6AzpnUAABgiFwd4EAIUB7Simcb785ds8GQdLVB29+5SIBhShEdfzq823SENMIbfkN0vJFy2gAI6IlT6FHUKL9RrSf0W6gMGJO9EHOtWB7MghQwxlsDz0EojIXTMKbjMoQiIUzIhiBdhnbAygkW+AWubp0iNjJ+0e1eDtRZunw+S+uwNRlBURrgsCkRcb27N98n6gwgiSEogBBQTCRAmr7oQFqtGexxTSKUERDEN4OA6gAJSiABBiqBIVBmjjpAUSEtOKtXY+5CJprBS0vT/pE7e5/cD6pBKlitv38sw3EoHARuqu12K4SXiNC2leSaDdFACAgDM7npd2fCAZFWr4QkgQEAYEoAZZaSCLoDIItw7ZbT7baj6SIaPt1eJAQIEA6naKtGiFJXr0ZnbSjDx4wangNlf0Xn27UxDpTDUaVRChFTd0DEiK8qaqkPecQtNdCqIAowFCJ+Tiy5e24KcYVrgiJEFKiVSot/yBMGSmEFIlAyx4iDEYIRL0VrAAp0WJjC+/wV2e9gfnWh69GA9T08PXnO9DM6A5G0BMogFqUSgBJNObQB4gEdC55oRBtJWErOKRldrjonLuYEAxRUtOyo1efSq0a1u4a2pkDtH0vkQAixIPt47YLOPcbrqQzHAREtq9WJiqLd77+nBS1w+NPNhBTUyVDKy2SSECVpdYATFopI2I6F82iTkBETUIFc5UFaQ9Jaw6g7QorxXQA0nJtFkljvD7f7ScxFaWIQOe7Q8icVOceLQBo6By6QIG3Kg8gBP76tDNFPvvOm71qmr7+fAOltkjlAVASCKgxanGoyZxVWomIVsGzPdet6lORAEWp7Y2KokYKydT3SOpY9pROKV0Mx+9Mb15fHmqiRAKgflO9EaL1plEUkC4xh/NoPVOQLiSFwOHFKonp8u0PfqHZv/nsShGqBnHCXRRIDDGF1zqGJKAVhwRb9lYgKFQqaCqCCFOQKioEFRYUOgK0vuusloRhKa6SIg7FOhnefnvz+tkFVJ0a0rpnoLLlltBoBTklRLSFrQDAYLSoRSH9/Kg3RT799vl5PPvsGgKFIAoRSBKBRLEsXuo01pzmvksUgJNEKPwmMRtBaItNUGG7cKSBUMqyL2S4ZB9FtDA30IKWz87e+frLrdFAgQTnKspl7lhaVGknRFpean2PiN9Em93Tky4Rw8Nv/9XjzzZIKqoGMKp1EYAmiKqUMu33kZKqANpCKV0NrAoSkRHhEipsBYU42i8D0lmKw9hZ1XAMQj8kFXdxTYwkYKTVd+79+kkZWgVjcZNySReZQzd4UxDEb5CP1hA1wOHy2bpT6vr9X3x6TiEttYZXTYKhksSSllIOm11aa2sx5roIsLDWH0soENoKKhjbAQId+c64LIf9gXWXEDJEq3yM2MugVU0pYPD20dkno0Os5e5WEUc4lQFI6x6DJhQp0lrOIERDQBWhv7ydRSUdl1eEaUML3GFamQBLahq1TturwwnQomvrv4Q+KZOEUiWgDCVAoc7PBsG82O9wMTpFQhkl+5gH8yyTiiZFb3Sq0AT9d+786rFrnsEezFk05hg4X3cXUVUndG5HfY53Yvunq5yFL35akwDJoIhCM3cHsqaUpdQ6Xl/GcasSMN9ddw1tBwJUaVUtAzegEhly7+p8ml8zRFBdorikQwxijN5ImDrEGLj1O8efbDwlZ6sFWpcSAcrcykDmGjkkRMG5cYNRlDx/eST+5h/+UimqSUI5hVp4CDJ3ydRLGXdXm67lutbOesNvFNB2C1wJhlChEtCGAuX8pFYjVCqsqovDqqiOOaUsIuFqodEAJ/H+e7d//rgmzD11zGEDpioqTlICgqSt62kdAzHHR8SLO931P/4nbgIoQlGpWWrV1PFqlzRqrfuL87JAq1KoJIMRhBJJJVoxKCLqgKDMiI0k+7V0ZAggakYVTcW1ZIUFFeHRVYE2wA4i6e2TH/9qZ5rUWsMqRCuQBYLwIFwoYhGtqWrtHKAUEavn+//tH22SQJSu4d6LBJMM3eU2JUSth93VJImikIZIYUYTG9ITJKgUhgjbhaIicFJHVS+SkHNI0cTkNUlWTiqTmqUM6bShWySEXP9h/vHEnLKATp1bAxFRghI1AiGdzhVqq9YgDXFQ++bnf3athJmGBCfN7i6pW2633W8l1lr2FxtR2M3jL2x1okCkIZk3OKaBou1YiqS7nwtLaAcg1L0jiQFOqhahqQFi5FzYExypyz9+eP7rJ1kMJF09TGBJRaUS8LEqzazV1d6O+VwKqPov/+01SCZT8RLIwRDtht1FfPdvpVLKYXsR2dHqTiVdVCgighDzdjfa4yMEQwFhGIbXW1ZIMndkKZ1UmNINNLdgTknVqlIVLgwPTcvc5bv43n//mIPQYd5QhohMwCB1PKiryU0jJPNFCxNTef7xtRCiFowa2isrJC3lOt76vQday2FzeQgxzEjlHNRDSSgrKIh2cxCgmKopQKR44w5NiaGGqvQaUVyqdzXUCFVDtEEDxrFgfet0mViLvvdHsj1UB73hDsG5bTCdrs8vdqOLMRytSQPmfHD+k6cANWUlCKYUE0OXq5CHv/8wJ6+H66tqN1D9PO2IANsxTU4ooQ15allASBPdbkM1m0Qyp6ubWTvWXtW0dxc3qFX4NOX1ojdHFSol/+CzPz/AIC5gbbARQYGYHPY765KqhgBUNuRDIOnwyecjVUUNEh6WfILYshvH299/tytpPFyf75MREvxrBwSWqjAULpTkDCPlBginKqRuqopKSEJI0LtkMCBCICYWEgyYie/Rna671guLllA9+s+efxGdKcRbdarqrYmT8iYNq2RZ2Q5IQF1ATfHNzzcgNJu41NJlFtpC+7w9+uC9vlDHzfkGppjjA+aUpzOewoAqQcAYIXMzAZhvA1ANJq/qNVSFKh6wjKQSwRhZ63R14WePbvdRSuuQzadDvfd37h7GGnTMnz9atSjJyvWbq32hWMOctRXLKuc/v1Cz1CdSImzQUpG7IfHog3fXHPe6u77yJC3I0oPzICPIEBFr5VzMwJ9AVBUIxG6CKUJzDU6TdkkLcqhZUjWrEyqN4S+uj98+TWWsLR+5V1Mgf/tv9dN4KO7z1WnVqplm7C+utvtKaWgIRFTN0u7n36xOk2ZziFf0UpnMQq1/+GilnDRtLg+SGRIxt8tz+qAIxEQj5sbWbW7IVGByKEgCEXUGXbUDu05pELEpaBopOa+ms0c9a9EOHgGlJaPQLf/h13/mKdRgrcckRBpGVMftfhg0m5M3oJrhya8OVcLEwVqi62s1YaS8uPVoLTEtjtLlVaRWj8y1oAYJFWu1tYFo9S4BKjwkVK3QIAgz0hkpKxXKAhO1MuXIZtTry/7BnY7wJO7uGpqDJQRe0/I/nX61DctSeYNnCcBwmG8O42Rq7dAKREzOf3VdoTlrIKaSu3oQkc4837l/mmuxdUqXxQRQUuEz4kFECIGawxquEqpQUCVEVKlp6yatUXUzQpJCkItWiyhGgzO9vrz7aKGFfVeICs2sJRxJFR7prf/Tz//xkwJLNwNPp5hABbx6YZ2aSoN0RMzGT76sQusUoEcedB+dQdSO7h2rhBxnSRNMWks+V9Jt3tQqkdB5oCekCRkNvuvKFNppoJVEkC45UoLRJCbWRGG88g/uhTOzaERSLZMkms4le9TV31j/6S8lp4gZPxWNCDGbJh2GLiVtMUBU48nHW9HUAB1Il8fIAEJuPzpWOtYdkMRUqCGqDdSVGaOFAhTS22w4rE1RFBFHunHXZBKCQEC6HJY7cTE6QgRap3H9/plZwQFKkyroUkUOiFRVhnjJv7X/5nzVBaKh+WRtZVnsz4ehK52AiATRy4/fqIqZOFN4Z9Okpg49fe9eioplD4S2zkYhqq0NaXCjBJUyA6UMb9h4IKLKO9w65kEOgmbhZknrFGIm4bXudt1772QKUaVGHErkzjyj3kR3hirrD/8Ypbj/ZpIkMFOqlovXF9vD1E6P6PTp11RVNTcLF8OExHCs334wsNS+F0BSan0n6WCbezagVMGEgCkDOo9eVIKIo9NPvGsoglKSmTpEUzmIkVE5ufDew9sBnwitWqDZQNFI+17aNIpAEu3+6LMvpuJkGwgEW7GnPp73ncQiIQzg818cBKpGMAKdVsnZD3723tvLqL4eLKiSFBQoSa9t3oM2wEEQdLsZwkrSVsY73sZWQHUNE0sEkTTpoYgi1CfW3fL92wMWY5lyKpMtxFyYVAuyKw8iSQkEVd7/r/+vu9pOSOvhIKYuGptngltvn8rT3fHuzc9ew6ApVULDxGuyKCXdf/fYJ18uDaRLEsQNeDqXUa3mMhHCJMRAhpqydYp+/N43UyJFmVhURc3E3DUHIWXHuj97535H38NQNtINqZXD7eGsIEyV1QGk9Nvf+zde0XgX7We2Ybpf3f+d3z+J6QjLZz/+IsTC1KG5qriLYfThwbfOpNR+mRokmRrHQR2MeZbXarhgNbdiEgFoeKiSDi8PF2+qQCHZoSDV1FIx2yPpbusib7+j2h32sRzKPvUWruJG9ySVktFXV6qGUop0P/yL4pznmARCQbCGfuv//IeymYaTTt67szMJmHryERCRnKZJTj56kErkVd/SDZOEQNFg8/hrAIcXBuIGdhQq6BB4fv9QrJpYuIhRqUQQ4y51st/Xw+m7R4tUD3vYYZ+OELUT1xDSpjazLMaaQqQKJu2+8+6zUhviJKCER3jU/g//2+9ebxani6Ta//6fXoNJnSJO9TBx6vDWo4W7LjudK+fkgoZNa8OnlUG2uY57sjbJFBCgGpxHt6/b7C2UoLpl4TQVP+p8ci933rmV0mHX91pK1qqKCBd1xKS50wiNmmLKACNhsrv/yd+fAmINpCei1MqTv/Pf3Hl6uHukUsX6j979iTKq6hhMqDC4929/cMTCRQdHgwpTYiMDtAQyl8BAhJg4tY2foPMcQaa7y1ebEko3G2GeBqtThBzrIcZxeuvdLnNbGT6lEz+klVUUKVkOklPipFogRUELS2Cx/Puffh4VFoogGeN+4jv/h/9UPud7C5aKSLZ+96eURp2QKGoqwPH796xE1zeqDyhM0igMiQ2CafgD22A3KW0eSavAlEK9E8WLAWL7cRVdzzq6LBbjPte9Pnprmco+8tK9Q2VOQhtFbMO81lo1xiG5ppkzAdGDr/7mqwp6GyWUcZrsB//t71w/Pnq4iOoSPtR02s6tOKVGEoma3no0lEi9zXNMNC6JWEA6qFBIhVDVREzF5jEPADWAwoln7qIGjWtZWjZ3cem7QO87PHwk3FfXNPWDjHXofVQYp6lbaMAmGotqoCsOZIEwCt79k09qhJJwH7ebW3/0X90/P79zx2qU0aRLAFw1aymukCRSRrn73nG4DBkzKtLQMMwUEd7MXWdOj6haSyk2D/sFGI7j+mAh7nJkIGNKXRiYpjfy/lq6cw6r7XbZo2SyCqrGdFgsMEmGuauUzjav0pGKEK5w5N+9+7GnAMLrYfroP/kenvrDlVTUKjATqaOoUkRBt6zcH87evyeT9501yI3i7Yu083FD5hPY/Ep7tR2fgEKEXK8O1y5TktwFXYexS9JxX20j75zZdueQq7zgoTOVXenJuuNKwzBSDbAwTud+xxrQVEPI9E58AgnGNC7/5IfH1/t0Zwh3OFVNheVCESGmJSwJYrJH7w6M1AJWowwxklK0Eb9MwaC06ksMAMJEQFQlAsrwadG/PGguYlZCpPeU1GIKbNJbZxGlLPTVyRKSIkJyrYZrnnZl6ozOEOTE8Vm6o25wIQln1HTv1QYRkT54+3R8XRfrVKOCVEkGcP9czGpVCYeRRW9/cCJVujw3AA0KTTPECkk21/8250VADCHaJhguAbif8tL6Qw8cxPMQUweMhwRf3urC92l9qad2GJJXj64f9lcx5MqujNGxLrJU22yOejBmWF9kUsnd2TXp63fuxHnVkwElglCFqYjg8lxMLG4g1rT86FHnrjZ3HnNySDIPVAmBWCPjhYkEAEObJrSiUYIVd0JNoGCuWaLkLkYZ5LC8p9y74UKOTIJenB3c95vTfmIANe+TFj3sMxdWgJQkxUGhGkWnWFlZ3r7THUb261RrCKDKlgPk1SaJiLlXzYoun721cpfOGotCxQMgUptCzzQUbZBxS/WNeKQgIkRBF5Z827WzNNE0CZCNU+6x7W6Z+rbXi3ysZYHzIWUiXVY500kRBRodBx6u+myyS4PR3GsxK8Kaue/uHd5ebA/W5SwlAoAJGO20vtgJG/6eCMurR7etRtcwNAJs2E6KOaVry9cqNyEMjCmbwxAEQmB0Pz12tWwGqdFLMmEaYpPvGWvJ5TovXZcHWE1d2mzHfs2oKVGqdFxy8rUQuqRI1Ni3wZXUmkTumO6Y88AKEmECIKZOSKlfHjJcNQIilvo7D1cMy/obeqEIqZJSCVBCwAjMI90ZEmpko5msBaiDy36TrFumulehhPcmHOVOln0sy3axiFSvgHWne62HRZbCzsckElO/Zc6ufuiTVmJyL51FAfZ1fZTr9dClPscYM/UJ5DVyCOPqkxCA7jRNNW49OJWKZDpX6aCCKkiVsPaxYx6r8oYMLjd4vIRQ1QF0cA8dwnwEY7JFlX2+ZyjM0/nxmqrbabnQvvirxa10uOotulKT0veWwWoWVVWLi1RU1VonnpzgUochmdZSDW10DLk4zzlT9atvjJGlJ0VR0umdvkbOAvwGw4XSIoXS25lubRrlJnmomoAiEg3cp5BZAysH605YrRc77E7vUaun8eJ4qFmnqdei1baLtVTqOFhV33LB0Fo8q/STSxQ3W4y+66fD8m461L7rknhl6MxThexe77v1lZzgLy+yqsRYwZL74wdrhHbWxg3RAAZhICFm4rfMRNfGT2yEaGnFGkGRkCDhlehLz0Gv9ktZlbq6naZDWpxfnh0ZK2WdfCsyHh3j4GY6ap5C/ZB1K10Oj6EbC8LroDj4mO4c+yYvOwv3IqKN6SIi9fVW+1ebR4urH8XCqqJ6YqxOHt7Vwk4A+Pxwzfkj3cBgOp93AmyMpJm26LhJKxGRnWDXHfLywHQ4HsvqLkbCLq6PeoiPqZN9v9znpUwUq1llPy7IzK0iCbSQg4wi4Yd+OsjpmV/r0Cfx4hKSRG/o25dv5GT88gz6688x9YtpimS1W95+Z8WQZDcMirmhFSJRHcJQwmaOdWPVykxGCYE3TqWHx5JVZCQhnpA59fesjildHBbHqJysMtY20nq/ln4hI2gc+3HTZRhQs3NyoSuqH2J1Zte66PrkpTpERVVAhaiML+T48GR1AvvZ3iJKhOaQfrh9K03I1rpJaSM2CjmX723CB8xdCQTU9pw1OrCSiQQcK5ecd0ncFukQ0/HdVFxxUY/WzFNJpS6GupETO4iyBMcuqdVLlwVcfeoKWCbv7JCmms+OxjEveoudNyhTtUlDVPF6ezy9Psvjg+3PLSurmoCpO77be5gBM+O8cR/Ixg6ygCi0jWnYJlMixojWBQIkKkQZstIe3eALSO+Y1mcTXLo3dTGIBmQTR71s9YghY7fYcz9aFjVZoyLCbDRalDSKH/T4dt1zuUIdi2unKiKmggDU9Or5MO2Wu/3Dsz97IpogsbAaabh3gkA2kNHYor95sphASU7VpG2OK2yc7CDdE93QKksiIsJSCluMk+bVYdTbedpnPZflSUy5xoHdglc8yTyMo6TefG9XT/MfQqLCgomjliTObbaH3b4cdckONZDSzGhHG/BKvEBcb97s335o/37Xtal7qqYnd3ONlERmztBN20ECqZFMoCqtVqSQqoxGDjLlzIRQCZAiJlqlUy66bhl7lXRRl2tz1Ho9LGSssub+6oVinOxscXho9s2z5ZB0EynW8Dcv0gcssbjHy/5W1pgqEqAmM+IMiBrON/7yq4v+wQf3n/wkgpWhMYatHhzBJamgEYjmNqMdl9Q+vzDsZqTdIDpFWDsviZh5VIhq7DSNldC0ttGsvzislzrC625Q9Fe2Tnz1/Pj2Ybc/fDW+s+4Gvn7RxfFxjmdWhxf4+NUfxoP1dVkfDSwUA2jSmqJGyTPZPL349Bn7s299aP/0RW/S6G7UW3esQpM2acpNxGqgLlIrP4ywpK1gDLlhFgYCotoYW6KVKEzjMmoZh5RLWKTLw2qZPLCrnapfy+CyG9+x6/PYbMrmXrUiq7G+/vTYMO2uNlhd/PyHH9aXdnvFSjdtmMLNjFUgWl598cWTTdcdvf/B6sf/C1ICPYjJTx4eo0o2URLqrQhpJDUBUxPCRDKBzlS2G9j0hi4LtGdSs280NPebcRvDRUGn17ujVZpgW/aaZK9HOm0vjvvXL63GyUbs6RDb5y8rHvCzfD35pOf5+/f3V7dPevFG1poJdGAkiHH34rPHl7va5fW3vnV2/aebZIqQ3mKS07vJo8szkUhJBEWlsSAlhRprowQFVdjKFxUEVQAa6xyYNYRPiwky6y4O7Fh1v1hmB67rUuGQBWp9tegvr05kczd8fPPqrjzZXS3vb68l191k9uhPvrOp751pdW+P9w03HjDqxee/erEV025x561vnbz5J38pKQEG2iGWd9cM69LMUwu5IV8qGCJJldAqmVCyNVttkjDrY0KFUBcJknhVcgkzjuuFXsthHFYLSOxLhtKnhdZ6sVzuL/puf/j8auubo323snuc9ofoQzA++tvf+ubknaHc3OiWckkzqI6//PdvJkROeXn/nXsnv/h/f1EtM4hkgNy6lyOG1C51OyVzxwEImFL7+6KNpUIxD9xFlJwFVC1eByxdXJ0JUsXxkcTyep+HgcJtdKZR6zJxetUttI7lab3e7RTDWe/bssT6dd4sD7v+e//V0ce3P8iH0TqGOE2aOksYMj198rM3apGWt28d3bp1/OLvfp6ytUNgav1bJwjtcytgb6DdebINiaQgqRKzcgWgtxJZSQUkGmmZKiph22/u7TzyopauO2xTvxTWfR0ImabVMHGc7vLy1YvSd1GqDB8syhfPh/Xejp4cDgW/9d/Y14/etX1RcyJcZ02GiMbTXz+9czQWOzq5dTT0Q//4//llJ5pIhJgxHz8cwC7JTICcw5ZIkzgEkwQkRzCkac1k/rYKIGpuLza9BdTik9+V4PF2FxHbWB0nxnZcKKNOKRfRXvni8Ru7e3gyMeTe2e6bJ/4t7m599upWf+uD/2y6+tbbXqoYy41yh6JifPnLn+pKT1ZYHPVJIh+d/99/oWI5EAaIT3bvRDw6I+SG33oTe9txSNq4qdKY+CJo4kgC0eA5GueGXsXsyWYhsjiVq/7JdjVk8fPDsVHrPvJ5XZ3Qn3yxWa9fvOGw0+WD5xdf5+8++Gr5yVZXh7/xuxfjh29xcioqqVQqRETs+if/qt53390/VVEidWe7/9fPRbUpc0y55/rBMiLn1k3NQ9s2ppVWzafQxiq9uT43z54CUCFnIq4YVWDd7vH3OaGu08Xkyyw+lpWhwHicuDm/HK6f4VRejHa0G9ePps8meevo+v6vx2/99O4Pbz2X753oWN0k6HB2AtAkvvoH5wt5eImJhInY8mj33/1FG5FBEDCa3j6Rit6g0bBpypxHMLMc0gwBULz6DJq2HivCm8AlZlmuKtDv//1HPa6nI+5e3slFpst1Rq1Wl8m68eKiinXddZke+TM5Sk8v0mrx+N7T8//48fD9O48X37vrhcpGxFSGqjDq5//sk5O7l1cne276IWtaLn/5px9HsiSicIggsLjT1dp12vQqwMyRm8VBJKGmqqYRTSiBOYc0FggpgInUaDQo0dw9/onlRa6rtFprHrddNkClz1fPX4/77bRb2uuL6+XbF55OpnNmfP16fHJ89+qH+OXp79wmPSBUrSXMAIaXMumj04vl65p4WZ151f+r/8vPIaoBM1YQIt3xMfc1pyY28SY2UplnBwDYivkQgUclIigioDb+SRP4SKMdqwjEMv/sada7y3p1r1S4DibZtOoEi3xn4Yvl5etL3LO9dV68w26DVO7s/vC9L9bfWdOjVnfCK5Oq+XSIvHj4XuYadr1IUm19dvgHf++VStd11sR/YqqL07Uf2NusLpxx0Jkg38CeBGGADW8JKABqC2wMtqIXSklNJ0X6+V/dNaY3dXWlKEvDJFOXNkybJ2PG8Xo7ZZf+uizvvhasjliFZyfl/uU7H+US5uEhrHWWdFnKORYP3rzaPrx+vTgu0eHn/8snNUmkJFAiwtQ8dWe5Rp/niKoz2tNocmw6igShhEgwAoJI87dtoulZoTKP2UXMLdfPD0dX6G9dW9lnkXEni+Spf/P4WsclYyuah+v+VtrtVFhsHC66zXa5eNvqqOZTNXUVoHruVUQlre6/vLq4fnD4/AeZuyc//2Q0agbEBO6UFJFPTlhTbwRbWzE/KMKYvwZFZ0Z0E1X5zN8Qoc76NtWmcZcmtlDTacwLXXBKQM+xDGcr2b2aPGTavNjvok9peSkf2rkwyv78+ird90lWVA+ph+KgaSCmtBxMVYK2PFnmiy+G/XOkshsFQeRMMWFFUMKGWwt6Tg2zhvA3lgO/GXsoUtPJNqkWbwSAgIIiYjOhEI0MPLNRIH15/jSLHqUNj1bwst/sXvb5aoxssVo9wXh4b/eUSkeRW+9cLe6/fbYS7eRQPDFhXzs5WonOagJZLUxeLLuL+7VL21CqUBCqFapk2NGZ1bTIrgLe4KRsLXn7BRuooE186IQg5jcz1NgwuoYJN+CLaoYKmT7fnAR0qqsh4oDxs+CDqz10f8fqEkxPH71/eVGKV7z7e/v8fXYr06TTNFFMp4PaoleIBiURfZ8NfPqoVE310lqyVk0OD00e3dlaOfTObBCJxlxpeXCm7kubUYTfqNBxIwBsQ2nOIlwFxUTgFNXkFypPznVfYipHi/35OFatu7P9NYF6nqZxKf3063Tv0Wkt/vA/W+sfPIpdck0YJ9L62NuQTRgUzcly16/WXfLDftot+nHn82wAjEII2J3czm4rq9NhdALzxEZuQOz2lCXX/0BBby+KQERNbzqq30h8bKjqlstXP+AL9MSkp6zlefZNZDl9PKWevJJhq4vDapzefIVa7v7tvPv9W7vFfiN5KlUMuVzqIiczo4JiCebro2Ghfnn68qMMGjq2XrUGJYn3t0619gslvSYx+Q/S3/ZQtUvfBI1AzLepvUEZM59Z1MxEwaj29nFQxLrn17FN3eB1aVrfHF68fnrtne11OaTA5rIe7Pb9w/P1uI/T//zo4lt3Jul6uKm7dd3hFTuzdFO8KiGpP151AzeH11dYLbOGqKgZg5Iyws46l0WnKkCt4ziVCCJ8pjHMQppk80kndZbHsvEBBWpmqd0qUmVdzo0Klc2vfr9Tl7SU0q0OGi8O2+lWGdPyQHZ1FD/c083lu/3V0Z88Ov/wLrraFQxJA4zxkAaVnCAQ2sxC6Ja9Za+7xTVSv+1qE6ZFAKDb6bGWbpGqImZJcoXYLEhv/wZC2zGnhzFmBQQEFEHK1jhiToqqlQvCzDTh17s7KaZ1ThrnmxeP62mNfRwvrxEmWNVxef5ykW4v0h9++OrWfe+GIfOAzqcou7HrTRWtpZKms0x5MSx6kymuNJYsZjfwvym9f3gCdkmsy6lBofRpGvdTrc2egiQpiSCVHig3Zg7a0gwjTOEGyIw3docmDu22vzIbV3012b1+fB3vXdU0Hs6edUevrGg3YVv7J/Xu2cMfXB7fPXSSxPcnQ706wKO3XhURsxYZAFXysB6Ww1gP52WRHA2GhgtEC1Z3uiJ9QlMYBwm6hjOm2RKBoiBTSOPBRosGmAmNTdgzS7KFEFGflBVA5OEXuQyrJFN99fVOr6b9Wndf3x9ubYqWoxjq8/Xtuv3lb727T2+FG2hLsXS5mURMk0J1rrd1Jr7YsBpWw+S1bgfvD00MWScmZenfWofnTqIx3JxsslhRDwqSaGON36ifm7xtdvS4cRD4DYOraZ9cFRE1hv3jPbLXsU5X17e9pKEvvezftSsXX+3rWZc6xXa4mD6QiCkkFvcXm22xvjNTVUi4A+5OujPIYZWTupT9drGmWeO0BonA8nZfIjfisRCWcjZtGBwYXqYxwmMeAjdDBrY8CswVJRlzNmxsbGUg5a5fnAx3P3y/VHcvL7uTEd3JZeWDd+v5pEh5Jyf16HNd/0F3/e3eBgmI6Im+eb4nNCdTUzJqjaZ/igBhXZ4c8N2mWyz6ZRJhFEmmnk7X6uz0NzwMVTVVM5nHB+HTWGowga2YVxIebPVv61hUSaoghBRtAiPw/tn5Zy8/f3dFP+Cb7f3DfrV6VpadP52uPbHbTUMkcf/++49/+9Z+NVWDme9/8tnyZLUYFktVZcBFE1itEWLqftqfT8Jar9NKpVJS3gdDyf72MqLLmL0W5rgqphFNtC5BBIokgopZQD63Vb/5RzDXmRpChgkE+tHLf/OL/fLu78WPXu++Pj3+Wm7d/gwsG8/o1Ff9Wuvw+s2j33n+8NG0WB58SCoXP356hDevDsPJ/TunFsICCzhoYvDp4skXzw9OQi70BN6PglqaHHaxzhH9PFcjJFQEUkUU2hSmgiAdSefm/WbWQ2Ez8oj2B1MjApJKhErqXm8PSY/vbF6+fsm0qherW7vs9ThNaX0oNW7f455Xqz/y9XemfoW9rTKffpp+gNifP37y9Sf3PnhnoQFvKgWX2Jw/+frNVI6iUvYbXy+ufQiMVR1KW57oaL01qLox+G8+GhLgGkGp4UggFN6UIwz+RjIpYm1qJSRDRcEpYXGqr/t+aXX6+pPww/HwJd4v39z6ouuvsDr9ZpLxyfv6Tdn/jbfOf2Ac+vPp6Eg//czW2Nr9vn9xufvm/PKjNU08FIBPrz/5Zop872Qn9VAjqqzgKQzRuG5HQ/gqs+noyVnMqwyBUqBI9BBKJDdRSoR7mym0g66i6g0E9tbj0wHpTo+qHP/xZXxdBn9zNdzS7vjsR+m31qH7cXj0jMl2n9yKVw9/8PJby2m9iGl5Kn/+acaVnV//wcudP9zRt6+QUnRirLE//+KVLq5jezn6cgorzgElN/ICPI5u55BOZ0efNr6RuRUJgSghuYYok4LePKva8NMoZDRfLFcLl5lJAD0+Giy0462VvvjReFit7p9+vFqWQ7pcbjrE9f7tncOmx133Bzy+e+iW3T4v8U9/gVoO3NerSz9+/3RZUDarYCgMcXh2fvl8t7dOMouHBKUfpsRaKK4x3TmzyX6j7wWhQGAOp0LQg0JNzkRpJkSM6oEmgAl4c/9inXVQ4KQPvrP+QqrJ+M0XJ19+GR/eOdw5pL7Pmq6kz5DtF7/XfXKdhNfff2f37ZL6XH1R//GP6zSKR+SdXW+ufu80hfjYZyCZvfnlXz4/0EqlFNsLWErGskREhahFDCtg0c1MZBGRkHa9Z9V4BKNZLKWiAQRqKbXyBn8kiahKwGFBStTyzt+49eRgKtf19ZOvz+Pb7365enpxGyf72k/jermIevmr73745LzWk9+7/LDb9cmK1X/+s+k6FFG7W3+z+3efXv2kexgMrwHr7fJf/ctrWd966/Nvxj4OUEr1XJBtqhQguLjdwQbjDJ4E7MaSRQMMDY8GZAWTtijmDG/Rl2g+WBC1ZCpBkfDg6qg+F9Hy/GQnr+zte7+u61/07yqem2/r9OBkb/L6r+L4+Dx9b+EPDoN1ovyzX+MwLW5d5ZfHi/c/eu8ff3r76tQHdwfzUH/8r6+rPXz43dXluRoAWKmqQ1EAqphw606qKZOzzQAQFG/jkea34oymdkBqUp2YByIziKdCACZmMuvocPX5R5vbxzse4urTK763+oWcvra+366Keeh0effNVmU/Hoq99e7l749Tv+iDP/t6vRddvdVr33tef4f1yDeqpkm6nB5/PBjK/s3y7q1NrRpJ8hQL6WsJFyHQna0ZXSO9NapJS2xEtH8j6F4R0CFpSDN2iWaoheZEBklZRSRJaGi/vXy9efr8d753+OrPP67X+nDxPG4d0uliVY7Es0m5OGZfKTgI7l/f98tl10f9+NdncnTxeeS3vzL83oeqJ+9eLkuuTrXc+xN8/xuN6e2BySrIscv72iG6yWGSCnGyCOTZP65VHLNDA7w5dzi8BNJitUz8a/3iLDWe0UZTUxNTcH94nk4eHN97eHf7p19dV7n/zgUEq5QMOznuA9Ddl+wqGLX78L3dd6fT9VLHX/56SIuzM7mM0+9v3v695Ps3t6EdE9yl7w+XZfHRC3nrh3VfpaOUnAR7y8yoASSwP5GwrLMeD6q1AVitp40gIxzDYjkMw8wTaEyBuWAHRMxEVSFJmacfjae/ff/ktE//4h/+4uGD8dHdePw03YvusB60u/d6v6nVdhbJnOy+e/jeyWXf2/jjXx91YkP+9vlYVmen56NGfxRjmQQOQHj8oB5/186unn913uc9mY0+5fUrFp0EQH93Bc8WIQwJCDrU9nQxPCQqhMyro6FPKSWloZK1yZUbxE21hlmbqkjnh0c//M643d766b+4jP33p1ff9M9jsbxMm0Xcjf2jFxfIDDL76PhoOT7YLPqu/tXPjvpODHrU8XJ/cXg96qPVMV54p0CUArut12W3fX355WFY1H1U6yMmOYrkbhDF+v7gyBZtYkhy8ibeDpdwMii2PFoNnQokkVKDjviNoVLT/kmbxlHicPxHD9bj4Zs3+Wo/8dXz35pubR++qpPkVM+P6t3rnyZoUAB6ffDDq/cmzR3/6t8dDSknsfCFIg7Z8oKbcJOO4SU8UkeUap6WeeC1RLiQLkdaKumrqOk41dTNjbkS4b+ZVEUgQtifHC06E2kAnbt70IvHDU5EoWizA2Rwyr+38IsvPysn5bqsx8cn357s7PVhk7YfbJ5D7dXJbgdQWSu7H6bhERfL/sf/sl+opU6ZQF9iqSl1KSI2XYpABUtanIilute7k9QxT6LujEgyUpR2qP0CTBpkm5VzBnHDmxorZH37uE/N209SeGWt4aVMjoZk4wY3mWv4y6G+vHo83Yo3vjgrd/Rn48mznR3Ong/vTq+u02IsQjKMpm+/9+q9Pi3Wn/0TW5uRopoFrALLKkFeo7OA0GtIGg5VxyrHB47ZTNUQFZ0ipHIq6bijZhB6Y24jbGRqEvSiJ3dPBuVM9UvVnR4MlxYdGlyNG2sRQNReyoD3Vq9+NeaTk3xvvDj2c++6h93T9Vntv/XpQUzc3ZSL3y7DQxv6l39/87BLUIZZgvRRYRahUdzMKYzCEENfqmZE5zWBovRp4kIPU4YGF6fWZJuz69o8iSPhZHU9fXA0GJr4TyQVkwqp/hvLxeYt0LAtVukXi4FHQ7e7flIWPEl9PbynPzsINy9vH9eX7558NX3/6uXWAjXqhw+efrTquul//PJ+JwJPACOLmEmQGVNtg5bwUlwSkSQgiZKSwFThxbUfa8dEOTq1yKmZkalL+yrNwJYROL53PFibjohCUgjnQw1rhdYMNkRIxLA+W6jj8s+/XPDSkPuaxnuHj6+HGvtvpvXtxSeP+beOv7pokv3uO9Nwrx+6//Vf3+qTNWTcKCbqpvOPSM3RtRbSpDAZSNUEipm57B2ngQqpOFkC3exFGE37SYBsE5Dl3eOuiakacJ1qQaVGqyFlngmx+fjChu76cjEd3pwv+wv15d3ro7N0/uKVRaL4a7w4ev7yLWTJAQrrB/cu3llb+vhPkftkCCIsyCYlhsA9VNisxzwkSbioeUBRS2hdOad6OIkqmZGOO6SsTUMFidkiEQQQ3t867mXmb4iqaQonwwnOxnWNwtF8EVTGYjnO38SJ7I9erd7ZXTzqn3+6yPtIhB4usVMg+NYzOq3kb0/5vYVe/A8Xq2xCA6takWRKMS9keAOYyMTqsDafVAF8AuEWPvp0lGCLAxbHitw11VJzwG3aOPEI5/HpYK2ZVZoaRD1qsFaPkGgGDO7RNHCgOxm7i+MPz3Ynq9sfbL6oF7UuNozOVFynaatp8Ne3vp28TvWtW9d3l7n7579etElrFqLSx3FyVgD08FKqh5PiB7cb+jgYY1R3kJPlfs1QseNjcLGYDbFnS9iGkbpzebZMM+SmXU6qojUajdQjmjmRewNTGTGs3AP5vXd3j4/T9Ojl01V+VoaTfVokcfcp3d3WznX3zXf+5gBZfLS3d4bV5/+09kr31lurwj3cI1rvwDo1J8qyc9Ua0aSOZRpFsnjsJ8Gt5E69NZCpmQoSjfwXLaRGpLOjGXuEppSarrOVlQxHmyeS0pzuoi6Pqt1+79H4z3908u3x7OnnJ7f7+8fD2aJ2xlKc5F07vnXnvU9/8dv3w8/u7+/cG7b/4KUOQq8VgDcHoupRahMbl+oejChOlw4EaxCY9kWzqaLsvK4thdtZRrYmBZ2l6cGmuCRXp0NrmSTllEwYVIiaoFWSLSrMOccZZXn0ztnVm8+v3nnrWX75cnl8dHx/L/5AJKKW0t97a82799/6aPXzr+oU31F92Pk//AtBZy0lNfPnYByKuzOaJ4YH6cE66hJtdICyn4LaQ6XuU17VKHV9bMgSsx33bDeLBk92p+vmd6Y5t+gooqJsPVZMjfqnwjnK8GJ8MH69KXn98PzpJi3uyQDZnhyGwyYg2vdu9fb97kU+e/WTFzh7tD19lH/0D12kbxBnI/G4e/ihVC9k1LmkC9bDZIM28npM+50IxcPLLklnmOx0SSSQXhu1dwagKaCsT7rW71lKAlJUNYkEIeLu0boXRZvdqkh9fDJ0CwfflIXvjpHHusnLaw0PGVy0e5K/h28+++KL8fF4/KHVd4fP/4eLHLY4SES4AozaRfHZJbiUqAfttRFWJySrUEawXE4KUxerG2J1fOHHp71oQhPTU1v2plDCPZ+umgIx5dS8/ZSiEmoGJyN8Bh9ETSlSubi1LLv9+fXJdjrdZ98cfzXaXvq4TVksfIzD0dG61+XTr30c/uZ3r1y/+NlrUDXZbIFLULxhHR6gR+wPmtKiMzhIw+gREeVwRSMNIeUSeXUC7291lmx2eg0PmHAm9WG57pUQWEr4zXRKLQsYagpV/f+Bffvb76y3o/oGNd0rUbYLXmK5q8uru/1xHBaM4d75P381fPGrAw6/+ycfvHdrj2cbTSo2hJrOHo9RAwIGPUBUZLN+KYcS3JeqrBFRX20aWSQSr2GykuiPFTbzdluomge1DFsPSgBq1iKaCMCkRSDmltKQ5q63+Z0vpHszSWdWXRf5UF+tvvVaN47dg8/f/nobw+Jq9/r55en2el+tph+s0x2r//bfj6YVkNT0142+BkSkmaSgne5f24p+uN5vDnuQgrh61ty5a6R6XXofkNYLYScN0BIA0czNg2S/zkJCkiFk5p4JUmFQFJKGxZB1HiJCoHJZh35ArYdFqstaNu8uf3nrclpNjzKH67PhoIeXx3d+5+h/mvpa33vQL7T+i3+0VbOg9E1RaYA3ckvjR6raYorzn/d30xKX1+s3ENLpz7cGlQyI1c3YxUKP73aiM7kPITPSBgAhfW+AiJqBaJNzCaRGDVLRvOqzNfE6RIRTFVOoVs0yPt9cHr//tN65YtrWu1x9ZfVbJ59Mb76Vf8IT+slH11ZXX/6Lq6Uy1DJLSrgRbczaFBWIpMU0Ln83yWbqyrQrJjUC21cqVcQEIrbZHdvCjm4ZZ0NymSuTYDOftkU2QjSl1soqgQBS89kMZBs6VaG30n9m3FM7r8/vyLOL8Q8vn613PXy8ePdwizF9lC+/6FbPnwxvHa1OD9+8+k75ey/ySX8xUhP2qtLot67aSGAigKE/LqU3Ten+naNSQ+BRn10DKtZpEWDaaKTFcAwmnTtWSggJYQDB1ETT1rC3xg0gNIEu5hbo+xaUwbkAEkFE1JMn52ud/AfyU8vlQUb31Qf68GySi/fOvjnbvHp7fXL3g6uPcfbof3486CTaTdRmwE0EQkJaLwEKTEW6MfqTKu+vSqEHg9vnIEWtK6aLmC4ZZXkyQIw+j2njpv2OCOTOhDMzown2JRoAx8SAh2adB26qzYscEOHqdNGLL945/dl+Wby62Xb04w+/GV6V3X39+q3v59N7+c/3b//g43/Z51AmmhbLIqbSHuS2U4ZQgCGKXtaP4kJbWU/Gi41FSA/bq4jxkonHdyxEG1NZwHmc2TzeuiSY3bE4g3YKQaqOSRglhTY3Vja5RUDCyfXxq3GF9M7hF5eIIldjn+3FEe5d3fEXY43hd99Op+ufnOP4zd+7TlTkLmotS8NNWCSa9yLgKlEFYnl17KOqIYLU/ddVCLFgQGuNi5L1+EjDGvMNpMzBt+X2lBRipjcSknnErQG4u6vedIezpZnAg7649boU7d6SH10fLc1tGg/TxWHhR/l+/Wp7OPvesSzPNl/i9OTvP7WOXqHOMobUmL2+AqBELR7uAbpo7vq+T0htqMvnF4TAmGaos4yVy0WbbszkcBHMO0MikJJAzf4D2N7Kx8RmIC4pamnsskZ2EYYi37s8LMbhtHtSNGtU6aZ6wIs75G+t/81m8dsL28vKf3SVv/PnH/drVxF0ubCETI1C2CAZJ4opvdHXzZKoqaqUChy+qaoh5p1rcajUaqZdmEHABksLtXk8s3oyFTFr00FSdPbhVx9LU0gzbgibpu2uhDysbzBBXu1f51SKLRe9yPXx7mPK/S+fy7d/+PsfPnq4+smz/O3Lfxfre3QwrA/WylJmuE+aeSHq6HVy91Dl9vn1EZ+d+3ioePGa2onQFOy6HL6bUs4zx6oV78obnhxUoAKzZvM+sxsEECZ3ACKOpi29GU5LUHi2+CqSjK/vWzL1CcPpYXEY/f2vfvJwOH5wdH9855bIX/7SH5393Z0gVTIliggKpmkglC6UpISUMbIlEjGhet+v9jvji77ff1FMC0xS7ZPuRWUae1nkuU+/kbXNHQkYyUTNbr4HblZDSNpD6ox4ikJtXp1ARr398Pn+1t3XY9aPLWk3bvPprubY5vfq7gm/+5FGXugvfiqr7/zrZx02V4Anlg5k0Vrds1Y4jZKol19iseiFGMNpi8Wb55tjwbG9eAloEUnEpBy7YV+3x8OiaV0bGZNsIF1zEtOk1k5I21mBmcyUAioS1NQ16GX2BQvi6Nu7V/39YcfT/evlEZEVK3uVby0vjz76+KXfqbx1Olz8dNf/4OJHOdU4dFXFS29edECpCG9eOp4l5LTst6nLuXoRYfAoL/q03H9yEGOodONSwLq067h+e9kpVJrCQkmwuTE3NytLZtoOj9xEJ0EkKsI1Us4z56GtbwAW741f8Gx4degd2a/6nIY4Hq+WeZ3rp123ePdk92Z7+ovX/XdO/x9TR6AKLEmtvdMn8VqF1Ggrc+S4n6oHcqd9n21Y65vn2iE+f6qiAU1eM0WSinJvKTVjACohGjMfsUGgmlJjKTa795ZeKJxrOgQQPgtHAajq29tf1PXtzWXkeOuzSyyGw2pcPX95di/vpue37t06No4X+58u3n/nHz/rBMSYtOa22wxWuawz+qrqKj2jSRnT0dFSuSQ15fr6V0Uy3BM3uT/YQSSJuKW2+KSJqWltR1PTfFNTSvPpnxlYDRNN6gpITSIRc6lNQTqZnv1sv7yzfFwu/OFpAU6ls9tI692b/OAS7usFx9t3/ln61t1/9RfwrKkGc4lEJqmJU65FRKgQFjWjZlYX47Qt1zZBt6nD9MWb5vIJCxiT9aEJjm4eBc3Rkwr/zTogSUn1N/7jIo0lACR3mVdR1LnZEoj6fvvsoEdvnx+63ebs+pBt9WrQ+4frD785fsK3j/synW7LsOe33/3pn8Ng0kkiewigCRYs+148JQWTBMNETSQcvp2SFUffYXrxJAzmXSjVooZaNY2AEdqK2rY2gGhWWQyGmFlLdwoQDVUBJAldLSiZ1b3mG0bB4RCI2/Eqb67z0YvAQjbbB3fs5cP30uUz/d7kb5bMA9578OwfbUUECGpFFmoVHbts02GriQRTwNvQm6bhtcDA/qSf+OaTHaoWSaiVKSWREDSWHnlTfAhSOHR2b/XazNdaxmNE+HztU4VGuAwLGL3GPDmlmoT224oX0+nhSn3YYru7XU7Pv3U4Sv22ny5WdViVk83/+MoYRokI6tRFiFidaJXTvk8UYTQnb0KF0sNDrevzVOPzcxfRJAYPYW5HIrEqGnX5ZqQpcz0viMYumWsoRrSmECJI2jamLBeRNIIMoQZUVBWHjJ2su6cXWtKoA2R7+kYWq9uvvn4vv3mgtrh//d9/LgiKhog6fEpkssMYXT1AudCYt8C5wgiR3lIKAu72+ovMMIFGW/chgQSQU3TVE2alEdEMk9lgstl1hpBGtJkXBhGp0Ra7fgZZm14/grRULxaL0U8X17Vqvx/73auL3fr6rB/3fnHrbXl194B/9hdtLq9gEOoTM4Vwqxi1rhWCZO1qKixppEW2GCNo8eXVaVEVzVLU0RkUlphJsaDMFPmY46zfRCezxpNjtPq9DRXbdkqBqqiwrXyJmVgTlOm6urtADRYFw2ZzsS2Hxeoudvn06OLq61/9SwIekI4hSld6jQIjK8KuXl5uDuOhCINQs9Qtl11KkK5LSPvHJlSkzCgDJKuZOqYeRDSGIG/KEgUQwaCKDJ0CwYjm1tGgTNIToBUqMTXjMNxQigQU1HI17c7A4VCZvA9//tWDD892x7br0mq1++Zx75PBxaTtxMvOmhEmZCrI1+XYeyZLafY0yuYSIZnF7cVFF2YB2hQRpkmEnbpJlRvPTr9RrDdLTAQpi6xSI9p6OsaMuUZNIhYED11emNxoMmS2GLDz1xpdit5Hj8NhwX578QKyWL7envhyevGNjRCE0JVqArEKN0oYlKWzba1HzEZCqkjUcRBVCqLi8Ma1dBJgOsikmgJ0E9EkApWo/6HXAOntu7inwaRpWVsnHqCHT9evkkZhyLRbDVmgMwUYCrCG4KLKyb5KWiJCLuL4ztVifzng7N51uXvns6OzJ5SAimUqBBqqEtVUPQfZr6/LRXXf5lXfJU+gxCDwkFrrYRKKwABTq6bGRJNUTZTVa7EkUHgbMLdJOUF2q7YlSjhXhOHj5Ysvv36aiigYPnrsVwttaDwYUb3c+t0X79U3q822c5sK9VBO7jzP593JxX5djtPddf/69YVBjLk7lC5LqFGiQkwSrfRLjIzrXR6Ww2KVB++kovepwHd1v1O3edcKyY5iojCL3FvU3db6pOnGyxpCDac7V0uFB4Qzsl92V09+9asXUyQVcbqzTnW9nknxSvdpeue/2P3q/c/K5uiQXTfOyu3th5vL8uad7vF7E2L/5sGH27+8ciW6PDKCqCpURiSpgjEfrbr+OOVFn43ZLKWsEil32C9U16sizUROiwRcYBIR1Xr1/fUUU6dmzCaaWwCnu+No0HBv3FnGeLj6+uNfvzzE0KWspFNjgvo8lSfIGu/80fm/un5QbbzVR180F1Vc9bdep1e/Wp++9esvjhevfv7Dv3Hnf32ZTB3ui7YqMEB6BbWWfbdYn6z63jSbw0QTEJOL6fEw6Adf/+rLc1di9JpEGsVVUtSOvj9oZTFRMbXUKYgQiaAdd/B6M+icthePf/TxNWw4sZQNRPU60mo4eePG/OCDL/+3bf/kspbUbXPtzqrL9vTq+J0f7/fv3tlv9clwq/z07Q/+9v96gawWKSfRCCMgiOSguuqwXHRSlSoCE2/C4NQn2LB++w8e/+Uv34zwICR1YFW6ptrBS0pWWBuYlVJSFUGQtEHpzeOfXqfD+Wdf7JCXR4tId8oUqkVin7eHRZAMjdA7Jz/+3638zmcjxyKTVj2ZLuMLeevZybfvnN4++tHm5PF9Tnevdt3f+l9GZVSGmrB2uSTQs9IEYTnN5DxRiQgpoIhZUpCy/O77T37608dFUAxGhIdE2CA1loZSo7I6IMimMJHw2jXyhwgB3++e/fKnr3Q9rJYS6aPN5fXuoN1U7OL0aDamGYby2S989cHwGFpeHfW+B09K7H58Z/Gk3D86/uLVka93tvzg/U9/8bs/+HcleYWpa8ATvGnkqKVQ1SOBLgmhMSdfVWmWZug+eO8P//2fPT0w12lyAbyW0oVbgvSHqF6kumPU5FDVMom7xkwyicPX/+aTbV6mYRB06ez4+PXmcsp7j3FsdTyt370+19MPPvifL0E977/9q30eT7f9i5d/9Qf3v3r2Rj+5lQOD3x6vr7c/+vDheUu1rVBL7ppcu1I4eRQkl1kwAhfQNCVTE6ckAvruW7/9T/780hFkoMIFyTmkKtpFiuRjserTFExLmc5jF6SoBLxsn/3FT3mSTXqDReoj58u0Z0weQWGIMjZjyOLu2fVjEUHSHnVaLPT4Rffii+O3N6uLfXpbduOD6+6be9+8fPd3/ndMHiKmEsEuRjrNNIxA81eXCDORQJhYSiYRCJUwReTvPnj/T1+U66OoQsA5sFoWSnJl8lRrnaao3q+kxAiBiFFievXNv/8YZ2tQcwoghebcdZejX9cm/gaVdJdv68uvJmGN9XtHHwy37n9d33j/rfqzd9+6+NymF28d6TLixZMPXvz8j995QslJTUAlUg0k8dVBISEKaNA0KQCfd6yCDKGLiND19L+++/95UkevAolJFo6sRpq0PUSWLLvUxQDR9VJnw9vdr//NJ7i/pFuftIyWSp8X2exymw6uKmYENOBn3/t3z64WexHdfL4+/r5+vh3L0QdHf2W3v3wRclIvbq+nbLf+Sfcn//Txd5+WmHlTpARAMx2GMQhBsvCu/433nWaDFxMVhM3aye4/Xvzdz3fTpDlYkrW1yAJRVoWoaHIbs0GHh8dJFKRPb379Ge7dYmE3mENTCqcNAm5ym2/pDBDePd7i0rIyFvw0vb07uB6tp1+8+e7xX3A9be/tqhoW9+998kd/+Ozts8eNZQgTKEK097zA5TiJweIwZGtWAapmIgArVGEMihKk/R7+u714LCDshhDO3pGwgAlQs2om8sn9ZdvEUK5fPJ1unwhMFwMrUq/bbanarW+f9rNeWkSEUHkzfbS58tQPS13afuMre/OrX4394eJkxOD79eHSePX6d+988x19+iE0Zc1ZQYVlE9M0rBdRLNs49UsFTDVEZl1jePVa287dAEn93f+jbnaTjxJdHhtYFWwwm5qYWKfQ/uR2rwAQ4/mzzfHtnikNq4UKTfWz17tJF6frtc1fQlRVRa4fP//pKDiMB25TfXmdH2at9XT9zWcPji6rXGA5Idxu333y2btPP0zl4MGYSnHCx8OBtlj27s6prhZtw703W/dg1CBqePWYiRZk/pv/uUy1HPaHwXxWrmHeJKoqYrnrLJ2cJoUIY//yGW6tiEjL5DWsz/rVi+tJuy6vVvDmKt2ez475GUDn0O8ncx4P5aSrfhfXm1sLZLxQO7+I9a13tn/18OTWyeTFG37DKhKMoesWiLLDuv9rw2U00k5UJ0Ghx40ocvFf/q7GdPBYwxGOZvgjQm1uaGJqeTUkFTDq9smbYS3o8mLQw8QuQa9evp5MU+4XKNTG0gKBLYd7DghOxlI12SIfSopX/dlhvL6fjk8Or3A+1V13tvv63551pxEeIhER2dDpFINojsm1N7ZFXqLajOGaypwRcwtIMkjc/S9PiniJZY0bG9L23DXAncF8fJabM1A9f1oWVizlHrudp2Si9fzZSNOuT7GvnJV8qnb+5vYVEmzwS3J0tUQudfPl3ePY7Y/99vu43PZd2r/0/vm9o7MkCssQk6BXRp9LNEcfcao4G0qu0mhu4cRcwM/s5OBv/cfY1j0Xsx8A5qawwQzuXN5992FHSITvXmz7npLzoPtdRTIPRT1UiqTs1WeksV29N1NRpdomlBGyxoEJaXt+tkqXe9JW4353ela3T07ldBg9kncZgPhU0clgdd91Bp2LvGYGJdpoem3Ticus9RABydWfnExl0mXD0We/dDaCkYee3b17a50VAKfXL6TP0plpKa5JI0ShZVdgyjHa9vbZwjQPg/bUzl0oWYd7Xq+7tMi7x18ftOg4FgS9lLvr/Y77V0h9ajsJlV7ZrXVMvSojqSjUQOfs9NHISq3OnlkAbajz7R+WWnNHAUwhzaYX7XvL6cN7x0NOIkLfPL009dSlJKUgZaltf3mphVGN3kR9ApjQ3rq60Nqn0d1lqmernUyHtMji59OI0RcDdxpvUL8jxT8pGZ34QUBxS7QkvjxKsASlWCMHxbyVua2D9gg45WZPPcO5+uMVo+9KxrzXPQgPegTD7j08G+Z5yPT65SRksmwxhSQjDBplN3nAsgrrjewtCExPJvTdodTQsu++7dX82gxTzdiNu82Ay11Ji/z40R/8YPHq79wl3Ktk4zT2XVqtj1ei0qlAk7BtSuZMw229dNtN0SAbJcCQ33o4+tLcbu5ZAAa6QzjcO150SQFh2by4VDTH7OJhWaAcVeiHogLtrC3QbSpXie15dIv9BIhMy9/S5288yhCl7mTwsn15OIqjoa665UX6rsjofd2Papos3HN/dtIJqvZiEW1bb4M8m4+J3CxxFrZE4u1WnX1PuGyfoRmpz8p6r1yuOxNLAsZ0/nqytg99PKDt7oopURGEinRp1ivPNco2r9OmiILsPrS/+iD6yZZjVSI7re4W38VQ8fHR2fLBX/7j3ck6XJ0eFoFhue4k4hCJCjBsFqeAwhCEtk0mGS4JEmJsBLP8w/9pXDb6TKNVt8kUCV1mhIcKom7PD1lgmpLXiJQkvJZJhU2MLX3fWhb3aSolFCcYiwhUj966/FmNISmkGe6prXcffzq8tSjl8qvX/vrRH//gYQcQq0QNWu6TQHQSS2rKtgcA7pxjUAM/w73NpWYeKQMf3mMnAm1pBiCkba7rekNtk/vp/JxJRNubRBHh7p4ITg6oZaH7jFSKMNaDbqEheerHN4qtqHYXltkJ8kng9Xjn0Vj86JufPTzulh/s39uMU9kewomcFx1g7rnLOp8NkQiBwhhJ2JaCOhJdCMYsjIpbHz0eANJqzI5oTRSq1mtbhUnfvt6gmX95GV1U3VG1S0KOI6G5GaOQqhCadodLFxOI81IWZZp2zHZYplJjwNEzLFK92p6dL+98/fVbn50Wfbe+J5tJDdEN/SJB1afOrBMREZc5WYBAKD2RSiCqEDGbLRLRf/fPlgK6CjA7lwEiBrFAaw2n68vJKeKCcig+JHgFzJIC+6mEiKaGNrcJIxhl1LSnEbJwz7sRskXSbawOUrO/43qdNhHv1b8aunp99HJzUdZ3d3uPbug7FZEyLvKgaPNZY20yIZFmyqBgaBUilNFWTSHShycqYm1J7ewqHiJKORxyhApj2uwYbaYz7aTvEaGhSKoaXmoErUu8+QcSebHI8HDhcLsv9Cl6QsYOlv1kmx5e3+YrpO2XFx++F8MuD4dwvv5kpGvuegPFr6c+J0gSoc6LQ9Ecu9re6TlHNsVtMADGWx/InCWJZsYP0CzJ+cuLTRWhl+1Uw2uIlMtdWpqPE5EUKUWMh4BIWhxuCIQIRyzT8fUukI6OdxeT9IxRlj6ulu6rvJVHr57/DYSky1/88PtP16++DEcwJnf0C+uM8HqNIZkkBWb1UNs2K5yNt0WEVc1buKciJI5/51IP3s8cjIA4xFRMDt9shtOVREyHUr2MkTnW1KcShWHUgwpQ9+6BbpUwTxwhgppslSnL28OrlwVmCvT9YLuljffizWZ69PxHn1zQd+cv1z/8j1TLpHUkR0hKyz4rYn/R9YvUXKGS3pwRKFtyj1nkHD4zXkEB0ndWvHr95mK7n0hVFdOUUt9l3T55TVP4fh+opYqMeyxz1CkoEN+mtqi6MqCNeN5U4zaltqf53COMnSvyYhNJD7tOd2U40u2LZ/Kdjz4rL+x3v/o6Rx0YqMzaW28U8HJzpx8UwpBQJaTtvpvnzE02FBqh8/ADFAl79PXli2nQoRu6bNbMiXNWp1C6JDFdbd1rQfKdDz0LQ1Kq3G0TA2CVjjdsyNn2lxeiiWAAZO5ZptVwvdNcorvYH6ve3Y37688Hs4e7X3xdh4JJokbqum7RqQDlNftVL82suy3fFhHXmLnspAroUAuR0Jk1juPVz157J/3QdV2vnahaTpnT5Hm1ENb91Thttlhgt+8WXXFGGiLK9pCC4lcYModFlPAm6xFVu4L1xYZaRNLq0FX6IZmPan45HN+u8uFzvzv+ejqWqb7SOqYAScurxbpTBl4/O14tGiNEBN5AhkZADGqISbSFZsGY4ReKSrc8r9jHYS8Ly9Z1lpdD7kqMMpwO6uP51X6zHRc27XQ11Apa0orD3hPBMgpF8xpos3cKQJ2ymFiMoPYx9Q4N9GOxheP4yS1e3/3jH10dHcuw/eLVIZdIo1JSTv1ykRA+fTmdHHcQMISiodIYn22teLtabMOatnPsxlzxTo6uutcYhZZSzsenEC6WJQ9Zp6uXb84vp5zKgas1C+EpO30qnpK7kyHaLdNv3F1UdP9msMq0R8hRvmYiGaYoghxPx08edZuz/+jnQ7/47EnZDbnk6gFR6/tFIsy/en56sjZtHnCMtjdbCRChICXQ8uR8SsJIMIy3z57m3muUOhYGwo5v31k/vOtv1kvxw+XrV2+urS8TlkdpU0Uh4l4mSuppqiFqfbZme08IVPcVm12CmyzSTlCqdSMCK59SX1e7x4sjwXevX13ferPvOEoqQqr0w6JnaLz6Uu+dDNL43m3rA9TbwoGgzrwF+IyuWCM9AMTq/efSMTxV4TiWgovz7W89PN3sFh3q/vz5600stNTVqg+yWjKg7kb3tF5JDJOo9tY1nV8bDZd+cdhrpMHGi6QxduJwAdyLLcpq96vxdLUnbuX1m1SKFpJi61U/ZFG9/nT/8N5Rs02djXCl0dhBIbypgcGIZnzAmElMhH7wl1BlqIkmOopPp+/eSRe+SJguv35yEUPPUfPA/RRE18lUisDS21pl5RWKYdnQJIEIPHp0k2g/7DyM5qkfO6jvoWm3XOZ6+Hz48FY3HYa7z8KVDs/ojvJiSLDrX14cPThLKqF6o8mmQBzaRk+ItuaYpDdvZLpQQgl9eHzZmRpm82qyyjLhkBaYLj/72YvapxqyXtl2V0Wk64tHFy7JVLtVYkhaLEqDZ9vF0jELohRH4xrDl3RLBbmflkyL/un04NHZy8vn5oToBLVlzkMWbH7++Pa7DzLBtuOriRuphAUaxa8RMxRkVbXwG9IiKcf3Xg4QWGjyoclZdlM6Wtq4/eWfPSld1smW67S/LkS/6KQ4UkhWR79YdUpoymh+RyJU3WvZw6I6BJIh3FuUw+rMpJweX2vCbb16+bOrt169AMxIJFus0jpbvP7Rk9O37g2YFUUN69Hmed6gDROgaa1AJ7y432jQg937OhUKTC13eXF8fKJbt7P17slP//zrybIeZHWUy6E4LA95Koypdiltbi16C7JWJLTeU4UBSOzDjJB8EAYxnOpBFw7xfOx1XcrS39l/9qTvJgoiJKV1fzT4m2+e1kePThY3jENCZwY7oNH2cnjjlxOBUJQQVwlv26cgby+mKokCEwvrBEdJe79+/MXnT6e86EIWiwWnXYV0q4HFEaOlKW2PFbt+qobQFJz5taKyQ3HJpm7CvLgyTUcbpuHoXAu4v1c3Q3/+4Ysv7+StWgmNGBbdtLm4qCf3T1d9206GEFqTC8Nws/SAbQENIsxVAYlwCb1ZPHN69k2TW1Mlm6lnYWyf/OSrqzENHbE+XeKw39c0dJ1UyLSrnSJtNl225UTTZEAEm++rDs980lY80iWbadEIWz04lOtH2zy9+6vdUvWjl+dCCemA7Of1ADs9O1p2Q2pU3Vn2CWuMHQ2GosIag0jAkKAK4gbpCpCLd74qShMjojPTItevt8//6ivvkBS+OF7rdjOypmGRAfh+yphKqle26kuh0xbbG2qaii4u925RTQxJ8iQSpUxMx/3Vy2r3nr04vfdqWpyffvhxVQmD41B0OFktV4tFTqqCUASaGZQyoAiFUECN5hEbymZeDwEiXESpoKS3rEgnTAh2OUni60/8ybPK1MEnWw7cX2/rpOv1kFBiLFKkjCmN14n7qSai68QbpQBEniZBL2NKkkWWF4GFF5bDYrk/zu893b+8s5sm/enJwy9SjuJIq9WwWHTdIueGwwQDRog32HA+FkqR2gA7BSNEEFVBMEJmdty9o3NzCVOFdkbF/mvfyEJSqdX6wQ67Ec7hqFMtrIVOuKWU6s4PkYa87G2ekAikXlRbHFwgDMXudh6nflfgG33w+fX06K3P3pydfP36rZPXBUXdswyrk8ViocmS3tDVRWEgRKWpCWdqqLGRiyWa/bNFYaNYqRBUntx50bdE0JtqFbCEdF2KKDL0tj3sqclWQ5ejTLvdGKxBDdbdZp+OT84enlqjBkNUXnyxLyEWqqieajmOOE5Qsef33kuX063V+eXtxZdP1Z+KuvUpLU+O1qu+0+ZVLZy9j4Mq4XGz7EAEAtVmQiZNDdm4EhFtzzI9+gf0EAElZ6UI1CQtVoOqdENfttdjdVsdL3ot034/VQaR1H2qhYtVzqu7A9oIKZzd/dXoPSnJYWoXK8lnUOvyLj5Iz55gbZev3/XXVycPjmMqko9P14tOYQ2lDKeIiYdTGpz810cdImrajAE4qyc4Q4Ntz9vbfY0oXkWlbZOWPBwNKtKtOhZHUulOjgaZpnG3m1wgwy1FSBQsJMDV0DiQJGlHOaIgdQbr+mWtx0MUy/3kX5/eLtcn+bTuL9bjXu9/e1VdFsd9VghEdV4MJkJl02mhCRsFArowtO0HgaoKVFijQcMes7/7vZMaLjUQwZRM0PWrRWaNZGU75UW2/uhklTgddpv9oda8vH2cGMI4OANztmq3//L5Pu/KMoQmOSNfrs+mXe5Yp8uX37rQ49WbgZNvRco7f/jjw+kqq2lKkDbVFZ2VhxBhqDc9jMisKVTOBlkGCQXobYUFyRBQjh9dwkcdwtlnp1jKuY61ooxjZBPI8njZRYzXl1eFKun4xDQ0vO4PZEQpCnFRgMpDiLMLj958752cvLV39cCufr78oMo+DhcY9KrsPh3fPz7qskLoKrPejREiomxaYffmp0ifrXh1tqk2BajSrCeBZrlP5kf3Ogl6LbnLybqUF1l8mup+M0qXTHPKiaz7y51Pjv5o3ff/X6lU7Q3We5CWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.PngImagePlugin.PngImageFile image mode=L size=200x200 at 0x2024A904898>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "from torchvision.transforms import ToTensor, ToPILImage\n",
    "to_tensor = ToTensor() # img -> tensor\n",
    "to_pil = ToPILImage()\n",
    "lena = Image.open('imgs/lena.png')\n",
    "lena"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.755771Z",
     "start_time": "2020-12-10T06:06:02.740838Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAADGCAAAAACs8KCBAABTfElEQVR4nFX9Z7MlSZIliJ2jak7uvY8GS06Kdvc0memZHqwIRgRY4AsEfxkCrEAgK72zg9md5tXVRbOqMjMy6GOXuLupHnwwfzmNlJSqlKyoF+HuZqaqhxmfyMPSghBByAUL5hToIFgSoBg0CJalOiRHUBRKJQEAkCkNAJMMGQWKIgCll1kOJInwNImAwOTsxWUyaDEDACZApCc8aFG785qkLd9h60xXElY9YYIpYQxDIPPjsyIkKlMUWM0MoBBBIEgPCfIkkEaQgTQwQRjEJJkyAQoQAgTKJIhMAUTSVC0TJCSSAijJBFkiTRDhUJpgTIoUlICIuZoAG7b3o5FBrO+aUJASQDFH378xSARgApIkBEEwgwQEQXMRICSKBNRepwQBEg0CaQYIgKpEPX6MTArrXxKkEGEpWAIwLZlCggTMjJAbRMEBSCX3IjLsql/WFxEwgRIJgqTkQ5/ffmsQZQYSaF8bJBIhkFISSEGggZJkJCnRAEIIIkWmJEAQYQaRkECABiBTdAKkGTlTNEIEUDIgQWj/b0N7pyYJgOjzbEBquMxZEkhTIlNMIIlAWL+d3sWZwdpbFGBuJKCgyWmU04wkDJKAJCCE1NYNQFBJg+Cg0o0gUiDa4oGRgAFKE9t3DAqUChRORhohmpEAzYG2KgCBLDyKRNXZuAhQSjCQCLZFChvK/sbGYsgkEkLSgES2X1+VmUykEhQgQAYSZuvKI5gspIlmKVh7RTQaEO3VtgXLtm8gSNyKSGeKbjBEEgo5BWYKMBOQBiUEr5MRGeUJJ8EAUjDA2sawsin3r+c0Ga29MBoFERQA64qBnky0I4QwkilkEiTXF61IpdRWEASaFADMQaKteZCgJ0R3Sgmm2kKvopQJku30ISjRSYJGk7kd0oiIcTeFku0LA+1b03u/e89d57Bse5CpBJSCRGrd1eyRQLbtKkGwjLaWle0DE4DEVMKkoIMJUSGsu1dCCmDWBJCQGVIoIOkIWFsB7bQ0hSSkRCHZ1RMghV15BYBcoIRUioEc/P17GwqDJj6uRHJ9p4Ap1U5eIelIQ0Akgu6PR0tm++3NxCRIJamKVIZAEKAiIYFJtpUgy0zAmIEQYMxs50VbKKCSAEQlAJZDJZW1fzItSToFQBkR8L7cTH1vNWXGtmQfT861OEWk6rrWFRAokqK19ZNCytwIKSPb2jFnO4ZpJiUgRbZ9zvb12P5yKWXFgBRNCBJqGzAFUrYepwAKjmZU6qKfpLRibY0m2HW3b1EiU7BiUlvGRoHZikDQDGinC9A2NZAEAYlEO45FyludSDBJrTWwBCC5DNGqDlvZWmsOKUSatX9GezdaS78JrchBkKycth2U0T2Zpw0qAQlpYF/e3BlVYEC1tjtbFWA7RQE6oCIF2xsnZCAMmbkeO/lY10jBTJQCMJFAVrXjK2nSulEECXStlRQUUpnGVPtRa9+TePw0JA2ed0tERl4+i2i9D41i3+/3ZVskUvFYErA+TPsvZbbjzRQpMAURyCRh64lnkMwA0bKVSfOUwLZ1XIlW601oB0/bd9aewxwKwkrm+jqsHT8CuX43iiD7WvtiqLrAKaC28G3sbt+oSytOydxAN7L9JhKSkgSjIlq3IbYSDpFKW2suJCCjImsYzajMbMuCBjNJqfZ6CVjbr6ICJiWRkndEVrXmSJlQO1j4/fmNhFjsIQhlHa6nCQRgsJHv31tPi1bvZFQqtDYeas0RLdX6OxBImFLWKnPKBEmSSHohrT04jAkCTEk1a3up2R43gVxXYKq1lVICaE0C1VYQ2f59itZ+ESD4cjIiU1fbWQIirfO7WxuK2Jl5ClFgSbNWPpLMBKWULChv70UCEVBJ0IRWsEmIKUPrY9cdYwhaakkCDqMbPGEJU5qFPKn1/UNpJLC27khC7W9Q2bY4AdHKYTQhWK419Yj0we/vrBCOCJkAlDQIrhSRMlq2mgDBkCIgowQTLdeDF49dMYSAt18FFUhVSASHklGzJrqubWesZ5HUXgTXpZXW1oDUzrLH/ZkuJJUEwf50OKMyeDaezmk+locbdjSQJsAXlOIJIkNrn9RWs6z9NiQstK7wMAKplGFdPeTaqGT7XlWGMbPvh7QsrjjcH6N0nYFJsL2J9iraNNJqhCllokix1T5jApS1DpN+HJ1U+nNGx667vUXnwdafZlpmEbTuYMjaTxdFrcMHsk1aYq6/BG0oaidnGha6YJnVyw7OOQdfDkU5D+m7a33Qn6YymoUTeBwyubYErYNLU3v/wtpfAaDAaMc1S91fMpGxuTicjf3DrQYkYOlJVpmxBNeKAqxjZyuCUBs9BKYZIFvPXG//C0AZSsqUSCZHM9apROlChCOPi3XHsr2c9ze3Y28S20NAYGX7Pon0Vmv4r2pgW4Va/8NEn0+jlOlP+tIf3qN3QVBWEShUFtKVopRcBzjAAkwjKLbhNUF6hWBgwpLW9r2ktBQdGPxYLIMWc8gjXShgxFy6y+vny7J1EART1qavxQnCaOsrJNrEtG4iCrRsr1W+3I+WmTE8u3+4U+nWVg6o3icAA+AAUsnCtnHbiZ0htHFA1ZkZyHX7JRC1bexlnlB2Ow/3Wiyx6aAlM5eKpDOMyNN+uv5yvNuD5kyVJKGsFJDWJj3icaG39uex6eY6GrJb7osxaz7vX0axSBoyzc0ckYFCt0gqKryN+8G2ZGBdtO4NTLYZGJTkbRQms8o/uN8t81yBI7GoR9AYPckpe0YxhqWm0+6zF0vZWJraZmtVThKU3lohMmWUKYxqD0YyWx/k3WHrgHD17fGMtAomTVXOYLfIiBSRtfrjIUIj4IZcakiJJJVCKxGEPdZdVbvAQXe3J5SiYgzDMVjSvaass+g8a02ldDo++dFwd0oa2kHeajXFx4kFDSEQzdpoCVqb9QwEet3D6Nv69VhM6SYzRNJTsagzN4CMJawVAgkQpUjC1gMLrdkgqWyzOwlEfaGHw2mWRBRKLCrMKptngBFdgVBMKebpXh8/nx4mFfyrURHIXH8cFVqnnXWgVDtpGrrC/rTv++34Tze7lNGZ5KK18yw0KiVEom0ZM0HrHx4GkbkOC1AbWA0NN0B2m1e04qV41FwQrOkJsKorhU7N6bAACVL749Mv+7tTDaH9+ZAg6PAVd2gfJWnfD84QW8uYZCl3cXb2219d94ak0hmCs1ZYX6aihOV8RIcAyYABSE8BlihAtupHh1eQWtpCEKz8qu/IAAQvTAO7JbW4maVZJsKrSM+kSNbD9ofXD5uxDYZoi15A4rG3lQUg2gps/fdflwBHef+LX52ftbHIENGDkscwHB+KRKGSJcsKyIFo9bBNTVRSMkBJtoeTTLlc6DB4hjxLCVaUKHUpNghLYSTAXuiRgrfjJg/9Z0/fbPutS0DCLCGVNvcDzIbQOPPxOZIrokUkz+s/fbPbQTSGARVlyQT6zfywLwI0zQ75ik+iHdpqCERamreTRGkSvb05wj/4bWFNdCLDonaSMKJSVsKQ7gmWbENGmsApdP7JT/X2dDlAWHt5yi1FC1ki2qhMwcg20PKxOy3b29fjSGWhUTXVBURZf9zfXxURmryblCYQLqS5kgSRxRchLQnIzCAoKALpuXk3DZUoVjPd5t5CosKlUgH0sKQtJhJZqFrBLX15sfvU768KFPQ0tb3tARkZ7U3BYkVV2lZnGMy29d2yIWCWkmQdkJA63h83HxgUS6v4DSBocC3SJFgukGFtQhEQ3cwJSJ3fFwBmGV4sCjKkmhYxBEqRAGvP7N7V+bSMF2cdlv27V/bF3T4kQ0JJIteOVOa5LBWSWTsXk/THwXDMb+eNieYUyTTPBaF+E/P4UVdMOTdkWCuOBUmJhMhEqSlLUqAsqRVGcZZ5MVqBwq16Mmkg08BYIFqiLJ4sYTwdudkUzg3mnW6fffhuHpkWRtbS5mjJAC9LhnfIRkikt9ENEIbu7b5HNK5CKStRIfV+uF+eni8FMUXvbXxd63eC8NKmg6DorZKAucLMRhEnGqhEQTqYhazWTmKaeXgSyeJ5qNxtLGaIgi3Guf5Rf3PdCY+HOc3ysQ88+VkrJGrTI60SYDfuX7MIdDAtaimqYF/NbvX8fJksp8WNCtJgBBI0yBuWhrZCIaC0KbedjyqcjDJGlqgla7S+e6nJkt7GoJlJHG+ns+fnmKfaaluX9XSrP+V9tN7OCJAhKgV4Yc4BGK1hx6aEIBv0KnrSOss0iYMtAafjxMudlqXEYkUJT4hBtp0hAVkAjxV3bquNoFHJdJ5kRKZ7wGotJkylqzRRNC7whQVWXufZBWMOloTLQumWOp7/4J9PmwZPpLfGWTBKnS9L31o2iAmSJtPgLw/Pe6/OhV5TRQuKuFB1c57QXJbsSoornNxmSeba/ckyV3AkaIKEoKXbnOYUjAEoyBLZu8mSKLPClMWNt4ftc801OSKV1dI6JoHYf/Ls7dOeslzRQYFEGqSlnS70hDwhSVk29+/6eQ5mkpnpQ40OoU4adqg2DWU20wptCAa4IJq8wgC5QNBjhdxYUyI86ITCDAike1aKWbMnPJZBDmLYv7enu5gRHTOVlhwqaiCtLt0Pru7GnQPpWAEjAKmgWAcx8f2xSdiQb0vncmcSNayrk0mdJh/7kjFjMLRjk44QIWQKmUihai2jihVvsTR6EbsqL5EkFN73lkYhC7xmRPTsENzcvNx8vFmm6mPWOgeHIedlmum9o96V//AX9WZaEesVnjDSgLLso002XPvUvnv30Juso6RI722RG6HKjS+I3NLM25Sdj42nNWx9hWxaH0KtuA+kjL5Gg8EbOZPhHhWWVDGrWCJk/u7+kw/nw+yeAkrp6ml267re2lh3e//03+0eAiu3hASdjWrNpbYmrNVz+nB6bU5vtYTwMqtLYZm7MwtFjqbiRFqCgK1VrtGTViETkm0GJyiDETlvuDdJngKkhDpf2DuSlmlwuNeH9I/6E2mzTG5V0XWzSgisNNSu3pxe/Oh4e2GwpLVVValGHlazFVCSAYN9l9siGKP2UTtbqhkjFr8cl8zoSsgyQThIQ0AKKUljK3ptHFcmiWQmhDp/XE+EohFqmcaoBjIW0YtB0vFBHzxbJgjhMJuXKJ3NBQsyQSeNpdTXT768mVfumSDS4EbRsSwhPRa+rn+33zgEi9IpQMzwRAZ2G8WylCKhGKU0SGtVUzuoLNUhRctsGCQbSWRzv93TJUsZg2YsM4lSZzYaJjIOZ2fDAalgibKIPZCw7PajyaSUFifi/pPL044S7JEEzsYzY3HXHEp27uP8mi6SiUygt2ApuUx5cYFa61gYaUaRxQAFIYrWeCekoGiwlxnh5sqk1enFciwQAwmWzhVh9G46uUOOpPaHp+d5GrhM1uc898NYowoedUhyijB3Eqnj+OcPR7QWTSuk6iZ40f7d62n74tPuWLB8m70T5kgYDFndMuZTf27TVIceAlWYRIj5r7YGADoNLNaEE6JbtIKy2Mcv1SuNWbQAbeyv8kFpRdXm+83VpkZOKhZ36HqL8GptulZIdINqmqLef/TsNdJlCwDIEop0kPnG/uhPbFqe4Hz55buNWxirrKu0SBiWxNkzX5ZaSivWlsiANwCU6yTZumOkFlGRFRlzNprzcOVvoSTQSwaFDF5U+lqtzPs6HS4/ipNjOtiIo29LXcQozKoulzDbeM2mNWBM95+cZpBiI9YgJ4wxvSv/w0/rN3+4CR4udw/GlBFFU4LGUlRnXg91Tut9pf2b/ANYKa+1mCMrZVppY5BORAJRP76LTqRngMWNTMhKvVdfluAhXmyXIerkY72dL8ZY6G2E95gq3Tibp8k8gcjT091NtG46gXbeKB5eX/2PH/7hd3F53hPzD1mpYpFWsB4pUm42WgL9OsLDJBoNjYhD035YG5qyjXpubLiem5ZyfhgKAIQByUVuFtPxOGy4EMfN08E47Q99l7OVWumCWERMi409kphNCyKyCnGyPz4cSVqDj2Am1dubH/+n4Wcvzy48l7rMTy8PUi7spiWLR1Kx5PYS8xLFsiZIyoxI4v8PwlXDahlasdV2gAEs83k5HUNAlE7Jit4156RLn2ocbs9eKHAMp064HE6zOxgWpczV+y5nWIUthJzdUEzLwycvbhRwrFqWqNOb03/4y/3f33y0jbmq1hOeTCAJmTlmmhkDF/2yZCmtpYfMHis1c6UjyYZWi84kxPZtZJ5QvTwFk4D1p5PVMiprarycH5xTvbxQqYfFt6gd51oKVWqiHPfcDVqEUxRZ8c6dWWuiPsw/PO0bukJKyvtX43/64e/+wT/zpQKxZI3dAmXje2saDcuy22ZNFihTgqgCwFLWVTI9mzynISiyx4YNMKZETNP5zGI7eT5oY44IJYvN2vG+Xl0cuqXCu0MZcFqGYZkMPZdj2SjkSxbUztOGeUl1BKG8O//RcTxvkDrj8Pr+x39x8cvX1y/qIi2u3gBUuXGZ5WQRY5780o6hjtkkQq30k/9aHJMrqUy0Qcoob3IIEvJtHpeSyoXnHUDU7Gm0Dd6dnnfzeH+0i7wvQ05drzCXYzn2G83BLAiWhUO8uTcv1gaw5fDJD+PUZAF6eFX+z395/0/vXlzFokynm6ilGBM0R5W7L8e8GuY53LliWZYqK+u2EqcS5IxVXyYoV6UJLUmr43iayKVkVwIzhmNv2WGO/iGebeJgpv6Ou3roitmeAwyH3CqMC0DCqzv2cVVCSEcm4xRnH303WFLTcXr2w4uvHuy5T62SiSB5eK9MOitpgJbjxUdZ01ySGhVjWoHyVqolwFpH4AAQMEcqBCgg5VS6fTUH3OtSMaJ3dqwqD/hwN1VN3fCGO2EglqlHlLKP8wGL+p5MBbs+X9ozVlBO0IGc7s+GCsR0uvyzH9z85ra/4JxJwWSUDDe9uSKkCCnnpbvkEiyeAolGtRQ+ao8IrIC31ZWvcQVoSoBN1zWf17uuXzrTycIHHYvbcuoxlV3MeRrO3+sqD6WLyCjDOB9yU5aln5foNQ+F5vvjVZ9qkzBAX1zzcHbT1UP38fnxVbWLLhcJaSYzgd183FGNUYMY6ZfXUQPIdsQ2mLk0lVKKlLmtbFOrRiXhkSaXrRqR+Xr2zHCor51yLkMcubV9uVxqxOA3OAfgjMSYyuVw1R+FanU4udcyHYt2PmWaw1lN5hacT2O9659e6N0J/ZZzaOVYlTIMD7UfAMua1ild/RNNFY9aF2MmqUJYNpRTMokJp9VGByhoahCMJHLGxZFFHul0RhbmVAbecVfh+83wrpzlMupuYEkN+8prTAURcPUxatlvBuC+6y1KJQQPWPScdufzR3YzW1e86QLU0IYCavOaBYQx4QFkbvsaWRq8nBDTAJR0NpkiG/1tSksGIc20pDHVOg5o6TcPBDMLlxzMofQx7/BkoeqY78tm4dkBfZS+7E+139aoXmBVBbuYcc4w3yWg5MRVrbmwHs697rPYkEtbI02f4RYq42tXiO0sdUW/Y6T5o0ig6SisIAE02KusLAPY9F6rkGRFdyws5FOX7HrOxqSWzk2neCo75nncDf1SdFQdTVPguHHU7OvU0bJ2x+gsGEtPX9wWYOk807BMw7Ys7/riveei9RyW+QEOsuBVOJQRMJZl6nZlqVoldKsYQIQB/v152+QxJqjJQZvahZYULYj0jDqHVboInaoHHvI6stpO77c7eFn2Gnx0eztdb/OOXQwlrLjmLM6IwhCLqnmY5Iz70+7S3x/GoXRY5nQ2TQP7aarpwPbufpdB2xSjYT51I5ZwW0m2hhoQbo7gI3vZZFur5sVoQNJKos2yQBYTtmbFFjE59qW+t2sxcojXm+4EX+axRN3nYbPVwrLAo/jxWIjCeU9yWIVN3BQurHv/6Pxw328Gt6wJJxroWeIY0cdiZ19hAJFTheYFZWc1WVaUVpCUSUTJlfDBqvcDZaJyrSiCGseQEIUIpc9dduU0bzTWZbhGLuXs7uFiTMvgWanHs3zY7eopS8nZupNKTJ0dVIbE0g+TaFF72JR3uNqdHspQmMrFYO0IBUq5XbqLw7vLYr8rIyaWFDO6wa/r3MTm4U0q1yb2Rpagaa3Yeq1V2KEUrIEmDUlCdEFm16WNi7pTmW/Lk+UIH/f7M6/EgsIHnE15xnnxrqIM9TgUjv3hyN5RIhbvIkGExWkaX/jbh34ozDqL3wtJiXJ6KE/4XVcu3txup7rBUovJ6GeogjfZ0CrHk0AUj1Wv6graun5sxXCpNFaSAgNZN1GdJyQ5Fex04lVo6TYPp3FTK5a+LvOZT1n65YBuUyaoaN4sd10PT4aJmumJMmu/DFd6q947qxHtIdbDBsVuhuvpu7OruP754NSi9BJpMQ61ytlIC6IJWExiWbdU4lHOK7Jphq0pkmCWKikycjPBcRpj3vRdjdN4jjlLua/nw1RiGeoybOJBFzqxyzwxB6PVu8QuKjX36akafT/1ccTFdj953zGnYJqteE4DO+/q9XLzgd+8sK/POymcGUH5JucwJpq6tak3TSKKTKIlYQ3DXHsTXyRLUyNmUEWXtM1CK+wXY+LU7R669OEmeg+X81g3vT3YZvF+7scHmysGW8J2uWTKfSkqufQhPGhzfbhbhk0uEcne21pqCjTr5veXWj46vOw++sPxeW/JGKwK2vZTwq31hK6mMgBMKEnLMLOqaIx3U0IkkDSlA1FcTCYUBKkeS2CYDriknfpya2U31z7spH5bb/OsZwQWjeSE4xv/d/fItD7llr2VKcvB8gVvj7uemBPm3sizVTVAljfbbfWv32+/PP9fRm/VyxfO/TaXNGsUhSmtsTmSWKA0bwQoACQTrswAQLFkMxjA+D1usnRdUbdwVxeWzW0MA+S0h7GLabGd4vS2MMKvN7rg51/fsO94c+q1G/Jw558+7Jfd8+V9uSRiDvO2NKhHma53s51Pr/feP/9y/80uM5XMKWTnZY5GmjeUtJHeApMFj+Kv9agFGzGQ2Rh7lZStlImiz35hpkva+ATb3M+bXidDPWygzY1t3G7fnX2xfBT1+Op8W598dFyGvHrm+PaQV/sP//7JF++fjTeHceuxwJxIeyTrJBL9aPeblw+98eLjy/+vzp2gnFaje1YXwJreYAUJ2oEFFaRBHmrQSHM5paQkkwmQclEyhKwOvgyhWsc+7okcD6extwV2zIHIBxuzTPHl0/v7bl7sQGjyF5++eP3bMxtwd5jsaffb/+MXh2/yqp9D6VxlwBLTmEDZjNP7m6O20tnF2Tf/eO0UMljqwvMyV3jDoyyTpvZMgKGQAsKaUQBCG8exqmda466kIPNyug6D14jArXLgcb8dOHE4cIDxwC3r6faif3831no+XD/fp9m7O+jj57/bTe6Jb28/un773eXGRK5iKmt0crpss/HDb+/TrERcXl1u/5f7bQ8KvWlZzl7Eku5NZEVSSHDFdVDkCiJNSmtAZ6sbKZMoz4p1ykp0bz52GhJLuUWv8NPYs5rvYysFbYTs3bi9Pzy5OnxW3uV0c853djx7cZq2171kx82/vfzu7YebOmU2KVhj5gDJec53355gPSy6qxfnw9+9fe4mUFHmnLY2pfn37aCtFbDJjAqTFugS3mo6GnvcoJ2mn4cFmZT6h//TMEfnmJ/2ZRqW41BKGKbs0hinnsL9bjPtL7cV35xSIPpn2s71pPxEHV7v/+r677oP7RiNtFxlFJIZhm33+rfIC/SZ2J0N12//a1x6lwk5lPOwzYxCaaW1GxDf9HxkaTUn/7tTrymLiaZ+NjatOST4+Ho2wOfYPakY6slLqcYDeppi2RB6P4zFB/+OC7IsvNzZPI9x/rCtF7/B/dN/O/zn8ZP6kOyy7bf1dxWHcf/6W7tQVtturPTX9r/hA/dWpI2y86s50LUx4lHbAqA1S7TWikupVbMTq6oz5aRlqrUnpLPkm4uMYMExO9u7dQFMOUgW89jXWuuzPg633+L8fCgRH23szbfv4+H+2TwfHt5e/Pu7v7v6bH6ocLV5s9U82O4Jfv0P8XyYDjx/ftWz7vJ/1vNCJxFpLviZMn31pmAlL1Z/C5CWQGdSGqyxedCjpilrMxymjGmUfPx6NNXRl0k4ZbeV+Tz1oBT9mMXHMhzfvOJnl+/vK5fLM7+5WT5VXr59vyM+/8uvf/HhZzGlF2ajyRortH2++fp/3z/djpeXH338pK/3y5Pxr++fGkoq3UCdTluvEaVV+YasrUYArG6VXBXhmdZ+rNYTt5CE5A3GkpHY3vxfh736sTtu3p+2BvB+OTd5HMPfa3eO7v5Nvbp8f1fOj8AHb+c3/efXv7n8Laby1Wd//pu3L673R5IMyNMaqlcu+2/+oft0sofdZcS0JLuPLv76zbNCKj1ohlONC6vhTUeeTcvO5qlgUsDquKPkXH2YlEBj8wJAzV5WjBB63lx0mcfuvEqjGTN3hrowzzYb3t4Evy7P+jdz/wQLPshXh+6D7v1nt8cf3oxffPyzt59tpjnDTAZGNFNpf1n/37/8+PzTsZs1n2oFx8+e/PWbZz2VATIDcmy6CHWQrRR0k1m1QigYLZuQ0DJyFTG26TAzVleJSCiSRvSbn4+bcjjRH16dcarL7bZTTcybUnaW+685bse0+Vl3j8HfVpydfde/ef1Xb5cvNr88fLqpM92RjWJNMyPw7X85nH8yvDl3P9WAcPbp4f/17mlvRpgLRkbFFkvQhXy0EKOZcBp0LpmZuaUEb9hzG+olmKFN6qzpzXoDOzt8fdn1PJ6V3UZDPQ2mhHPo7l7fLDV83nb3p0P58E3aRTz0Q3n5UL473/3qh4e/G37YHzJBykuGrNlRa2D75cV323dhemBWXnzy9V+fnhWpyg3RSPvS1zl8dU81vgKG5lDDOv01ZSoSzBTZ9NyrEas5Gr2ZTlLqzv7ltIvrfnn//JhF1hODcym1v9xefrSt/fZ+f8ITHMwD8Pl+7/3x+uWfPPvH/jNM0dSjzJCTnvMxvHvyZJp26I5njurnnz/92d/snpRSiikzE3DK+r5O6lY+7/vJ7xEbAVlWrxqUJYslkA4l2uwkWiUoFK21fPfqt3+s2N7N/d2QdSNGnYbxnrt6gxHbszk3yzzczWcv3ngZdx8v0/Orq8Plyyef1tNSZIBlVjOjAmbuNcc+jx/c3I5XiPOzNz+bL0dfOrbFA9CWxAZVg61wZ2ucMh8F+a13wmrRBteJfj3UmpQ+IOVq+QSlcv4dzkLl6UOPg2DL/t7cxif1nYboqL0J4/7hYrOPzNO7+5vTa799c8cnp9MpM2POrOFkPWUp7kj6dsPjVyO/YZf1mz/MVxuoZMrMkGDJQD8sixemVkdHoynMHxvd1n6wgZuJaAowAGFSsyGZGZMwNFsKi6p32FntpZF1GS63Nt9149Aj7lRRErsHfTK/68yo4/KAZ9Npxn0NuUIwdAWWE4ciSRHBzsfl1XO+73r69qKk5CVpULDZbUar9dE6BQpuj7PF6hw30+PJZWwDB0XA0RTEDTR0NAIwlSINXX9z4+JZtz/0O0awe/h22FblFDk8WVhPl+MpYCyYt88e6vOrpxddDh2lIvdlqrHZNPUcM5bi7B+Wy4NVbk9sdYs1mXSTopYxKvuWJJBtcG2ySazuB8katwcpVs8eBElhlius0OCelQOhMVjKa3QJm+o4pmbn6+/m4RA2xNhNXmN88/DppaLG8X7z+auHL8bJFrlFZEZazrLR1/dilCHF/l030137vjgBmXdkVpYIjSVr5zXNmxsttUJozESz+JgxQ4TRDGiVY5UkUdmIfEPCSNQE4LYfhnfTeArO026cj103nA/LB/NkgO7GOm1y4K/K9UfPlA/Dn55ufrB5uOurFasB+cbm0lMRNYFiMKN3MEg+7fp5Us11P6q2dqnb5IIeNZZKc9JtVcM1Dp+NF88mXFpFZk3/ju+3T3t2IzAtniaW8u043AyjWa3nFn54WF4+jP3VMVSsO81nJ+/ms8D7X7zNO/3F+29/MN4M82HRPMnMet1EIWnm1vzEhlJUwON4WzqloS9OiQgIjshxyMk7kah1WZprt1kwGtOExuSLQAC5ZhkQNEWDDRM0cxLSgk/OIyFtH7I/ltIv80gt77TPCVb85GNn6acDFuwuDt89y5jzz06//WB4X0vPcFd6P9YbGwCGmpOakSl0jo6zHU4YB0MVRHMA9IK6bDSra8YSISKTpDWJ2tpgks19TJPsEaJo/ToFmpd1dBexm954UrbNr88Llmq7PJXLJx9fz3Y72XK0cVb2qNRx++TJm2e2n/5o+N1H2wNp1GidDJoeyqimsENjR4mEo1pk7SeHL72UZs3ZIdUcSizeNcUxjYCihmgrX08BAWuka6prs9EjME95aVpKRROXLLdGI2Xjt/iAeRxF8qHevqxPUKrOh/eKQp3V5fLuzdX4ArdfXP5qc/lgpTiO2StM0zIUAErjWrKaaA5u8OiO/Tzm0j7EqpYNXPQ1HGJxawkDUGYs+T11gdXpY0hxydWWY1RLAAgzhFZNSYIbo9HcNvj6wpfB4BuvN/vyBU4l5qd7u6hlRpHVPb773Xf98w+/6q/uIeNytI1OmYGBBVQ2BbaafA1kJ3dT7rVhFWSKkEzyWrHRotLyYMxptsqe6xyNsUh5Ux+jtYHNvLjadJUNzjVrwgV6zKaaEXN59vWb7Dfsoh7eabOf7rajfz3zagrOwzL0r+JJ6X752b/5dn4+zWJiqBgOs0IUV8C7UVqrqcEcnUv04zB1EkQvuaC4Fl3bEsXRXAJuTTwiWUdEXQRjM+CZQXJ+P1iRq7WWzQ+IFgkDphsil6U7vTwFlzrDY/4govTd1OvuE3uALeeH6boMo+M9fn/z8WlRFRe/GiaVfigNy6EyjVq9y1KUDmSa8rTtF0HZNkcogyPncEbTyCdgRtAefeUZC9n6qJQUCcFMa24CmvfWVkgxDZJlyjuXzq9+8pd/NFWwdHfD5Ul2cVvri891jyLvjzjbX/4S+uN4/WlEbyBUL88turHrOmeT1zRO5XGstq6vIhnHzkvZOJFRrXOv6j1rFrT8kPYAXMH0751iEkvrqGAmNNXwCgmvnh0SYUiYmvD2eP0E394NuxLi5u3p491+6N7FgHgZi5Uol6dh9oH7T778b5+dv95qkdgN9uvvrs7G3p3xqLnxRAPsizk7HJyQH7yvOQnez1BQiV2ZsnjTR6+ILVvQDFYPKpioJWVAyyVqPC7Wfdc+P0CDJdVOF8UXN39/OL7XT/HLpb66+ug77v79r13LUjfWd7EpGy1nw6vui386++Ktj/fR08f51/sLLlw22/PB6slVW/ZFklYGz+P+IRJAX323ZHeER4WBGSiKGFcFpNbYCYY1kK55WMRGSram9ntWsHknAkowiiSDZxoskepemV3H2cfHm1Pdjmd5Uy7u+mU+51j6zJg/fqJDf/fwJ3fxk/fY6Ii+Gx6+O3/2RX58vLlLP7s+72qbQylL78v89t1pQG9pqMf/y9gvF2NwzhLwVPlICwv0aONqNa3hIgY8ZmKpSMaWCiPJLB7N3HAjYU3+ajBiYfg2Xj+fjr4/nr55Ns6769+cflC/+SC9f8/h/Dt5vvyIL8u7L3/w2x8sy9DtT+cX21evz85sbx88/PB+f7zP62eN8hVgtMOru8HG5/HRzTGTFjmc0qK3SAMMxU6192a91trZCoY0Pu6VBGUFNFHIdD7ymjSBVq0AxmW1zyWW5Fk/ef9Hef77T54+me/9I/bdl7/s/t1m+XBZzj/4dvFN4Lr/dvjsZ8/10Hf1NDy5/M37K5/x8P6v7pUvrnTUcZBUSFXp9DB+efak5C2W8cRSFV3MTdlhysV3ChRFc+K0D9GOBTVrI4AuQ1aoVLPoWwMLLZmGAJVo0mcBYK22KTmrnHb27M2v8njRXz392WU51v5uu+/YLYcXv5v7ccmy/OnRL95a77M/ufrn96PV2ZbjP70vu35jVrV4hjPoI05n8f7N5B2HrkZsQXinLjOAajjxo6iw70tltlcuNe5ZaMFAts5NSjek2khFAyNBAxGktaCUeW8ffjajouDbv/3N/e3N8xdnT0+d0cvmwRzV6jfPfpj77P3ti7N3n9+z2Fwvrv7+5Vj31eD9sa/7r2IoRgScop1tDt/+7vd3lf1QErMDsXQxZghJeMHCrkaT2gKZsNaO0NQKDlZnYMJqZiZqtF/QJCZqyGcCEZmZUJy2P72+PST6uUqvJ/74y+/Kuz9czxeIotiNnZfpV0++uK7zyX78q2dxFJGbi5/dXiMcLh/+9E83p8Mv3vYN3BN8u/vtv0zDxZMfn51qsZMbLbOfwkzRWBXbqSXhoaWn5OprEJxEUq3mwWTeTEANTmjWy0ackGbu7aBO2PZs/hZy3GzLswNfPPnnevb7w9nIbwec4nbZVt+c/vl3m0udPp6WZ/cDDMP5L24uUHmZw8Mw9R/8+bPug0OEQULYsPv6N32pz65efGBz0qn0JaiOa+c9z8Mu0zwkGI0gvweh/tXUISjCkkYDIW+pbEATvkimR0OqlRGvHr768OqY3Yvdu/v46MXP4uqmH/yBtc9acPeCgdF1c5tPn/3hRw+Tuo7nX99e1TnK9TA855v377ufdmflEKFMOIeH31+dn3evfzP1o8WC7LzLHLL0oWjxV6PPYVpX0eOE1PrJhLSOsKjL0nS4JBQtBqDlwwnujd5AYbkYj3cvb35588EPnn79i7+Zlmt+a08OuekGP8/Fuq48FHWAd4m7zVd2eC0W23776vnl1ZN8OF1+9/7bT59Nx9jdDXMXopPFbvofwzd315qBVOiIktHHXDJZ2CPn0au8/YlXTAD/3YPWMqlQ61TL1prtoJXHx60EKEEZV9HF8fe3fNH/2YtPf/rq1M25eXocOw39Rb/czT1qoouvrUsJOX/4ZP7ycHEx9OOr15fQdnvGh/LZ1U/+otry7dY5oCADpdftoT7dnz75N40cLnDCjma1R0oVqhpqWFmRToilrZEGUikzgVgq+t1YClbJLdeUHT7qXMAmoezGf+Dpp1svF09+/bfvr5bpwx/rzS0+3g/31sGuxn5RdrXP0oH1+PFXn16FUMY/vL4cY3E+vZ7i7El5x9A2SFuMWQjU4fL19WdffHq6v3uw3WxWkFh81PXigURg9yyqo6ZnWsLUrVbvRoMhyawaehpQYJZAtZU1abXFAhKNolhw8+WHH7yb9ex3P+8/ji8//CYvX3HzybsXD+f12bPX13HqOiBrFxH7j56e/vxlZ+Pu96+utxAWdePZ7c3+TPt4sd3M982WHhW5/fiZbh766f1mOGdlpSeW3M4uFIBZrjmrtL6CkLAkkCamWipUSNYXo6QCVmO2zkqP4qRG8Ta95P7Jj5/nOxzuN/nUtP/gs/Li7MUSH/JDy5vt/Lz/mXWeTeB64Jf/9GyX3p9//duLc9ETOjn7cj7qrAwP9x2yZCpcYj1COfp47VtbJEQvS+0Q6aqbrP6RFpZs7LAJqLBEEpaSZbJy6NYBz5iN1MwGTTcrR8Jacp+EJX5wfWV372p/HsvoX//hj87OLlEP/cOX/u3E/s3VpgZEqzWmL/b5dO/js9e/eLJTC9p0HaEBvtt242YLJ8Fwq9ltx824Md+NJD1hkVClJtKtqzM9wtmwtVzdODQq1KwBk29HJ6AAisIkZCQuS7bAonYqGNWC5d5d+ru4nT/u79Vdjtv+56eP3y3lsPn6k89+ON2X69c9lUpLO108/Yfzp6V7/vAPTy7ax6dDCDM6kPQpO0gWj1K0AmVXjYQli1HoaLCqpXJk0FsqyRqy99+JMOQS/ab7Pj3SckUJ3duEvAIj8NU/HsnXX7s+/Y/X76v1tnta7q+2s3f9xx/8/rUdy5/odpEbkx7LD97oycnP+b+NF0Y9IvhdDxCxpGpYyZbnKjAKQ4ZEJjPFjKgaOIXLlbaJivKvxiVkSwNDAhk5nHmmMpESzD1XV6webf+rigPAcpqxvXjy0z/7wdX+JTbdZsH97ecffSPD6fWzH8abn/yHl/dfXNhkQCx3F+dfP1Nur/73+dpDaNp2Q6R51vDCCss1xUT0lCsks9VO01qhfpbJld2odAop0lcDVMZKwmbtNvY9/EkUYuU+2CDHlqUBZiLTxzPX0h//y8unw7KZ0x7Ghw/8q2VwKHflavu7vzv9++27A2BgLJ++0VmMH/zid18O649PE0xcuubB8jCCNM9gGpbiLezOK2hmKhPjYhk2sLmefXCMrs2hrYo1aX0LXQvfeEvgowCgLBR9WXHzpsJo29wAH/122gSP8fE5u/Tnb/zS96f9hlRRnv/m+nj3NKd95iDT4fwn//zsMp7e/v3lsCFT1oJnKPMUyYY9saVqiY4MJwEZItzmi2rJ/cVch5IRI8Ks6U7b4bTWA5EKbQrykSdb02sVtoKeAtspKzJpnO5rt13eTxcXy3iyT/hw1b/5qpzVSlqnZVvO3FSfYU5xPn36rr7Icfgv45lBaVAkGr/L4tnCQpsysqDNN7Y6cRFOS09VHLeCbTJ9iKDnGgG66jqt5VzGMvRYhRXN82RIIKL1I63paqeBqKgqzuP73Q8ub7a+/fz+d34PXR0ZHUl1NQ6d93h7/uLhFMv+yea7s8CLf9k/b5+0oxjUUqtQW4qlIiUlZFjkBHPtlCozg/LoC4cliOyHmkOfLcy1acGQUiYzwwc+6hOaj9ZEmIN6hENbSyVC0W1qnSZ9+dPpt2Pcf/L69fnw7eJn96UnUqr+wf3EY5y++vLPZlCfvJmfxbPjzy82zbIQcLjz8fQLoalkm3NtElmldKMhFQZHcA5bLhSZscVS7Xt+idZC6qzlx43eZAcr6Ka1JycgeNOK0KyJ2uow7uvZJ5/E//x3lz85vHjz+8vzeLoru+5UDCHF6eEyhq7/9Df/9GeX9dhdvNmdbc7+tmyGhCKaRVh0iowmdWRtuAACBegEoqbgqjIHzDTpOGaHqFtlsWkNPl0n0zU4YvGhYY5US09WGhpf0pARNKwUgIJYhu6jy/fvvl1+9OnX4/t3Z2NXnr7P/fP0lOpSrl9095vtxedXv3uLaf/54XSJq394dQ1vqUTrzxK4tN1JrrMbk6aZY0gkYRmgMMiNy8DNSXXp+4yC1gq2DOEW5yJk5ehKtRSJlXQyywhRRG1pGS0NAAJ5Xz85fHW07flH714vw/hkLtXunz1sT0cjWPyU9/2uu7n6PL8+RHf9cnh29eZnT7vwBrM2t7MEclHDJBNrHg1Uw7pGYBsQ1RiomZqGIDBr9KUF7NVVH7kGEJohS6fVE8CmyW2UYAvHZDuB6a2sG4u+fjVdPB2Q726vyv586WaeTud3B9W5dgU2/CF+MLz7h//8anNbcT0tn+7u/8vmjOiIzAywfdRYI0ZUw7A0ixihCvOgQzKeZHIL63hi7TZRbVS0dApX4ypaL0Jl1ejNdGLe5M4grNIdASlDq2Wuwb7zXK6vcJruDk8Oy/VhzLunX2s4+aY+nTEOy5SHs/OL7e5HZdpN0xeXXx358nU5bwxrooloZDJvZtT2QxcjOjeTk4VzI1s4l6J0hGlfrGxisZ1WDBlAhuBNAilkltL0UvYYNSTQOiiDxuQab7UqWyvPPj4/1M6mrvLZROxHHrqL47x5eZXn9WGUXf5g/s8PV9994927T78YzobjM5XOCQxhbo6GVNSkE49Zdui6cXdms6yrRMeQrOwrg0zLrhyLLUMsGCKtORtaWmfTpJsh1VuucVFNG0ZCFiQdNBQ8ZjwAEIerzftvJnSDshbOfL/86O3wkH774f1Hdjt151oe/tu3tR66bliOH97W8x9/+eorJ1KZq2pm7YSQ9XuDcFdwex9d0ax4qAScXu86N0g13SZ1U7+kW0T3CHiyXXCAFh7ifTuIWstuXMV4ekRNbI24Wym1u9d7G1hPx3Gol1u7uxi/e6754vBsOfTL5eZ2mU/bj/7jD38Xnrdn5xq63c//lqMZU8UtMhvD2xgMWXPv9NDdz3/93btFU+V9Z0Badx+dqALC/CQuQ+3Pmi/se6kd0ERizLRG4a5/0KZGUmG2pNJSmlcom6laFf0omKsvWG7r/fzFN3mh6Pen64fxu4fp0+dvNV3Y3/lmmerzDw98Mv/ez4hqLZQsLZuv5HE4gyiU6bj5t736o9WrGb2S6nUowwwhCXSvlsJ+2W0i2gZoenM14k4E0DUthdl/zwYVCiyEkDlbZyCCiIZFWFrJfPu8v9vv/228f6pSw95/jrMZxw8/Hfd2ffMdx4tyPnxz/3n/P+HsrLuJpJX2QYWWualHvRDTNlh8MoxPP7teEqZkeb+4KDQrn6azoOdwSrN4JCya6UotrtcZDUp7dLG39L2wUpxy+z6Ur80dJCLjuMNt2HL8KX8RfvrQyvnvNrzsZ70qm7iavn7xpPLT8e5w/udf1Sdlf1BBIh8LRKbUmBSDIHOUzmrttt3VJoMpoOh+ZAIoYRjc9p5H7y1hGS00ZE2Lb2lMMvs+wrYtG1GkkUXBXFq8mGDm9mjWMtThcjPaZD969rX3UmY/zDr68zx7/7vTs/638WLYPOHPl6u/+vYXH+w6l8t88UbsklYKRcK5RjxCSfPdWT0co4Hfwz57g3rZAtG6Q+nmfhtVtgawU60JXsMUSxtoVyqsPR9pVUsSVdm2FBRCag2IXsbzV8ftXD/rfnF0VT/eHrh9F6ezeO5Hrw/xw6HfnP1+Wq7i/7OxUwY6Y4b7qhZCSsjMFFRDiGzDQBsDAGLg267l/0jpEbZHn0MXreS1maHdW9D87K1/bY0WHrdLa9YbMbDavNaEcCKFubt+i6p8wX+sTwZGn0ZMuF66h0u9me/8E93nRX3bXX72v+psUFZhVixgINpCaqHRDdmgrfpqEOEhgervlwLKoogwyFVj9lLRrN2tSNs6ojZCbJXvgM02Qaxy9RRZMpZc3cyrh8tCz27rdhnPx3e9+dBP2S+1Dneoxx9dvF2Wj7vDt8dN/y8z//Tnr663QS/qCixRKo1YYzKZLW8pM9p76ovR+r5LoPhNTwq+WLWlygpUYFbpa7AAqRYeIgARfLy+RSBaLiJINFOjlJCvS9Cb0JB1eRF3ZfHu3XKzKctiQ2Hy4cny24vy4hffLi+ev+i76ye/O/Y/WX55vnuyBLKiy5bevAIu1rQIzjVYEN0w8IRPzg+589C4PxbrgXRDdl2XmqIYHhXzAhK+StHV1HntRpb2LO01qQWmJVdtHleJqMAElt3mZRarb7vzjSPqsmx0Nkz7H88/e5ivLgf+tv/RF0+/ecknL/7XYchSCS8JmAUzm4tKEtt+d+/GUoaOOd29/u723d3h/W0O5U1XOMPlFcUrYBmoPZuPoG2ORxYDjcdrIENDoxqUQ6IAYZbpCZloK9YIxL7/4tUyflTnzfaXfVU/HfHDvfXLw8VnJ935F5/tbg9l+93Xm/iTfzp+wOmFoTrmzcNstemMWmi6aPLDq34cugxMTKD44d3xEvXF9uG4dZ8pF2diPuuBk5XSIMxcZefrdMf2xh/V3aspqhXAIrolzFDav6Ma3Tvbl4cbf9Ld6jJuz86W9HLsxofNxfXd9U/+6eXUfXa/vZT9YV7+ePOrJ13lMsqQKkTUgeEuMWnMcCaurC5WrKRmmZvtcud5Yd8t5hI47AcIi3XG/XXv68izigVzrRQS5aAa69RgkPW0KoaUBVe3DdRyzLjgj/W1zp++rcPJRt9jg81p9/5whe5av94NuLqa3/zxB/98LJ/96P+hDWBnLi8xL92BiK7Be4Y00JTjyOfKS/f0zyn7YXe8Ka7y5nDhlrSScXlyK6J48tX2IpkAz1x9SlwXWGsK1nZcJiFXBRKbYhjrHShA1k/0W/jlw1S6+mFZTtbPvvTx7jT6NH1zuHjKmvP0+jfDD37yN8sllJzSlpCHIRJa1owPSMiExTQt0DxVH3a70tlxrijLqwEti36PIf0EuSz9+80LgGHrXRCP0tV2Vul7XQVEorRRUbR47F4A+XY4fhN+Pr4eH+qLzanPy8XHi+rn+zN9eHsV9szq8dlf/k/6/JO/+eo8jVbTuik9srCSDR1UNjUTkPDIYMl63MMmdFPXWfdmuQCjZc94luidHqHSQL/HS39aKtqjZp5Maj1Twe8vOUqijRjxKAsCLY7TjXfDx2/zfLm7eD91ZXu/W57sb77g+d2ry+uPLqZnnMv96bMvXv/+6ehmpXSKPkGwoBezCgGasXNkBEE31ETSvePU9fTTfuzMUdwE1hr0xZp0k2YNkyLafVMtOHpNM3tMqOEaOxDKFmElWYZWqwaSXKKU5aK+P1um/mry6pzu6nW+iufj5rvf7yrubk/DZv7w0/3fjB0oRVqi9CoBzoFeOdOUiqYfbXIPo5bj/uHmbtnusiyvrSw5B0sucDciKDKglbJsBJ43hBmQmNHu6vn+NGquEzeDI5baFdj6+5EmeMfUeKC9rdt6GOp23037J/XJy91cnm2momm5m/p35/jPZdeM/ZnijIyoXSqKOrabXPBYjxJQlrEvZXN12U85vJWbWXGaieFo+Rke/ijkWkn6xtM3W+4aP7nO2hLoZqSbpcTsi5dmAUIqG5duC23ybfdq6ms3la0N+ydFxT7Ilxzfle0wfvKDn5/OLZVImoMKp6xkVKE2P2SDUREtBty8bK7ONwVL3S7vziyhQKy2nZBbKud01hZgloCyqmEd68Vp1tw+JoSwXpbWrO9UKTUaLQUCzJTM7B7nrOdXewbGunSnu4fj5Z3j5qiH4SO8R17//ddXvTlbGiucS7QbTSxtwUyuXXMzEVopDDNhqRFl8yZ9cYKlyKSCpDhEKU0/vyoKE2iygxVBbd7NNstQTeetVDZ+BO2uH2ulUyQyZXkEWCmzdFWczajezWN3Fffst3e5v//FtauG0GXQFAalKkwI0zDtI4F2X1kTCbpDiDB3DHx/bmK6Z84byUHjotOgZDTYR+vxlIbV5mBE522YlLjCiBRBg9UUUR8fcKU4jHCHney2Vw5e2ccQvP3dDa5YnsDUnWF5+dmZmbNJFIlwQ5q1Nk/c1EOV2jVoMMJgg6vW9C5juD9uKx2ZFu3OIDB7q6bKZvfGo0pnRdTAhLJr8+tqYG9XGrEMxmKk1Ug3sN2EwzWQI8b5kDGUSj+l6rK52JS4e1jszN+dP/Tl+M9lFhnKuUVFw4R83AlzV+o0Sd4VMxOpmJN0c1MW7b1MHQ1RwiaHV+RCCQ7SnTBvUOeq/29yg8o2kK3QrwQzls1uNC0Sc2bXt7qZZFuPcVzm94qLY9I2IMt97XZ5jjlP3Yd1GjeveXVgyTRYKZRkYTSEuVcGVEbXaYr5dGpnB4iJNEQETehhJjM4BxksPZ3uJmQVIzLSmyqyddwCjMk+0a5/aZ2Jl2E35qtflKaQDHA2e9SRSci42fz4u8/eTB8fT6PZkiw1fEi7Pb/6bjmru7z6o0+Op/ttQVnMYul7JH0xJWnqVJbSpUddqli89CikVZVcqnFx7I9Ls0tHhuQjDJRZtQcDo3qhoFVASMCRVEZXEhJXvyjp5tPbt6euEAlF16J92nFMBud3/B+3b/4PL/f7zbSr3cEUPPZPXjiPX0y//mLpdH/afPTpN8etR2xKIkNWC+UJWFazZRlLDGOiM+jcJGPpkm4bnOjzw+F2NwACu7SFYTIoQuzAmK3SJIsCoERSSiYyto+w7Rqmnqf3t8GzoRgYZKeIdsmpIUkuD/xP0389pjbLZV/Gin7fD/5QNm/7/ZM4/+zu5R9d/eL9T/78T//x9twxl4gCdZQJYoYSqQnWD0ZfTSKZrtUosjnvh7+8/+5Gu9E0K4xstJ11WZFY6mauNJlqUw/L2g0P2FD6fqTNVH1z51tiazK0K1vAdqMcacrT+f/w8q/n85dHRtctXu2c6upuv33xcLc///why/v85ItffHf9F2VWZ12W4jC2m00oB+HyBKTlcDzs52VJZqSwTNM8Hw8P8fTf/ccPTjf7FEl6B0RBpcuFKN4DQiCixlxDsuag7tYgyebiHfu5bgaWbWe5JMxowdqI5kDWevHjX/71pv6byS3Sl6rYjNm/unv6zebzL843F1/V8e2U+2fvfz3/xTS7FKqgWUVRR4RZmkNmK3tmdJPAUIIszpz3b97Yn/zVx8vbe3iKRYYql2xAxOB97wZmSLnUiKWG1j/8mjxHqvD4dir9MHRh20JVlLrU04ImV4r+fPuHbzb2Zf+qL3zflbrEPGrUV+PlV2+uL59+/XCeZyf6Jz/yf9j88H6qCnMLJSuyxnrblirMMkhlOpIrG4kW2UzMN2/yx//DFw8v7+fIpbarBeLYRcAiO7q7OVRVq2KeTss86/HuVcDLGF99tYwDwKXauOt7o1VwzU9w+BC3x37z9E/+2+JZ7o4/ygWHDc6H+qsnn8RXv7755uMROdTN7fsT/ubqYnZas2oSdCS7lDNRlVWWK2yfaNcQmJHOzCSW++/mn/6np/cPUWw9ah2lqi+NtHcrZCGzZoZ7Hg/z9zIjG+zNu+5q57RMs7DBvbCw5X9nStovBXk1nt7tSqHPy/72xnuca3vz8vCDD36S5bDZTXfndvb6iye39Y8PWTMAs8LI0rVLKkrj4jNgxszvMTOaEe1il4Tq3Sv++38z3y0nMkUzxZhLm+kEuXVd6ZxUtaEfWclVF1By/vrt2eXo67WvkV4GZ8kQW0CogYzDi6vjvxRXXfofn330/IsXb/DuFD/uf/XtxfTVLl91Oyud3fzix/m322fHufSFBsAkM0PB1Aex7pCs7VJBBNbcOD7ejmea3t18/lf29iAjaFi0qehWk8cq5rauMytuUfuuwc5Wyv7XL+26W6YwM84GevEO1TLNm5nfjFP50fvbepUS73/32/6PPvzdYsfNT599O3z63csSF7jvL4/WffCPb/7T61/9ZK7talZAiSoFme4hAOlFYT2ygX0JN0WVoHZ1K5j7N5d/ZTenXBYlojLU0mYetQki0RnE0EWRDClY3u/Pn26b4V0NZ0UxQwsYb4iCEJfb2/5hLF3n5+PLn3/9i/tZmyf5i/tnT177BR6upzdzRHn2wd9/8Gffld0+GVUpc5lXsEvari6VpqIDHIhonVALPpC0Rt8B0Ont+V/6becY6eqGMD16KdkE6HQWRoVvFWhY8HLktj+FGkdmJlYFx6HJb9eTRPB7+8vTIawrG396vXHf2e1Xv+d53D05YafTde5H3n71xxe/+ol+/YOTl966ziiXdcUKgK5TlM6nqRsluXs2WQfbqdkM8KAIzG/P/92rmyWwdFH8RDU4AliF0IhkBlDGVe2Feqy7HcNhfUcErCrpAz3i+1h+M9jx1d3PVHyaDrrv8v3RnrPrypPdd7//dPNu8Xs/j94tuvNvf/ujn38+LXNmxlxrCDHPU8rNqoRl2fSVICNaTyoo2n0OuSa+g5hvPv7JzQmI5dizdtYosgAaxwTQSlGUPtbpdnnwM59zVqcayc5q8y+VPrPhJoIAO+su3/WENIxL9NDWDlelzs/9dLzsauGbYbw/cDN8lL/+o+1mnJiiFXOgmsGyB3sxj9qWyNZotwM/E0AmWyb644VZx9ufPj8oq+UuExFtkmvcJwW1C1ztkUXjwzQMNYt1nddQgS0ziyB0yHanplGSnbB5IQhxOUulQ/HDZHg3XhyXuxc5bo9vxwPqHa/z5r9eHDeTMoHIzM4x2Bx9ouQi71FpQTaitw20TXDQLDwNioEOyx+fDp2pjlO7sU9skiW1yVqxVPYhgbBS7zFqIlAwz3KazXeAydw0JUJr3kD/cPvinYo45J1hbgT0tiy/+2Cbp+PF4fxTHpddZ6cbe/7qAz8zc5m3RLtYUDtfUsiCZMoZjW80w6obX6+xfiSRgHpz+fn9gVOOuWbqrVqfloeSdeZ4da4QJeMhh1Ll7DjNgls1g5NpJdYNt4Ip/q7OxoAfrEARY87wGObb55eb+ykXbaPuL67j9M1zXZVTLD4Xl8SoYpeFceyKicwINexGzY3+qK9C2JrZRgKYDz9djggO0cD+Fd9Y+/KIYei8jYJm077vLYvAENyZsqJZLNSMYo833ojycedjoE8UWQ8+mbEfOXTzq280qJ+PE8lYlheb/YPHnVtn5jBmgWpyg6OVREYxmmwFAyORLXINaM4ktnuZKeD+8qP3Ge7RwhKamKgJ54noNh1Xl7tpv/RczGCMpBlW41AoalFmEilKjtRn9w9l7suCTJsPY7nvc7audMOJUVK7jS893vvDZ7d3+Jc5a4c8UWlZPKDl2G8YcDFk1lZPY7ge8QCJgdWXIUE5n368KLty6tolYVhR7IQUdbtpGAzoPh/NlQZrES5MFNOcELyAjWLjiqbotaH0U6a8PtQvKkc8OFU1+qSYtro9zmXsf/3shz8YXv7fzxcIQTctU+9ZunEIqAiwgnwUBjbF44r6N3SGIOEikfcfDrfLiOrf20wIb+H6id0q9SKQx6WHDNYum3bAYGQNh+SMRkI1CQOXPcrmVI20o/2E7+4S81hDJ4zCcjufHYZuutjuvnr1nNdeuzrV4nRXVmrsFQj2aRF6TOhsfkJbs88b598+RaM2T/hwv/TRFjXXaxfTjFlradoqALT5qJI1U1gWM4dUrMBgsEhvIpvkCuLue7/YpxNZ9UX3Tz+2kdxMOQBd1TCyfDn57D9/tvnhFz/7LxcczmE1pILMXh2UmNLqsB6ZjS3G9xEzJNPahRrCaleIw+fTuKntAaWkLMVgEjkiMtMMspw1ukmlpZgboKqCJFJppWDtASqt0mOXm3TKtPlw/750GwKmHFhS2sXv+Xz3sPTL8PR6+jf/t3sexuPgu6i+ZPNIs8yk0VrKSoDRyNRENjX86hD5noxUHp71e0/Cl/VeZTUxMugOtBgis9OpmJIIb4lhEoACBkwGS0S0Iw60jKswGBPdPMz3xZ44Ndx0nfeCnS/9cf/0430uT/JvPhj7Jz8+/sfb0yzUSIkqnnJkZ1LYarJtSk1PlTZMi9nybx+DdKiTv/hVF5A8m5gFsGBAREGzXcLyFCOiXcVdm77Q5lJgNQpgIefqIKdJXdzC3ICKW5156IiuO+5MkZFnXwuzjnfP31365mH3sxfH+PzwQ+1v0pnudIiuJVIlYWJw1cuuHXRLmbL1AiKsXCTq9OH/sxdUTVijhBqSnSqybAdcnSkQUahQ7RwZbRarSqFRIY/kJzIjZutrIoENw5dK7OnDHMOBs5+en5dpeyr46MfLxdPL4Xq5/Ze//e75Tktac3a55gGd1kXTLoY0ZpPTtquTEHqEvtmOoGc5C03c0sKlwISMXtc7QYjaLhpWUhV9J6CstCzQnHhNGqAUmD50lqHQgquhQpUbyI59sCxXD/78zeX0jbrl5bsPr97qfbmI7e5y+SoYTeWexuPSod0QuiZxtBfU0kzEBkAxHVRmu432sLs+0MzB75PkgDTQj5PWC/uWNknQtJ98YC4VchoRlWZAiyJfVcasQ2x8Vmr3oW4XDNI+hzJh8KnvTscn96//7E/Oyk6/v/9sev50kQzeuQDvgkiKJytNwQFThkCt92UbHwMmEdluvKSJSMz5xcligpVS1gzvBJFWdHe/r9ZON2VWdZYohdkIzc5gOUOCD+vx0X5uLRossXk6vn3fMoTQlcEfdn74IB+O+w9f/ePL6LXU++2XfzpbXUos5ouJ2bkTlvuBXSNWUNgoCbVs6ta5RrarWFaLLkTm4QNZXeqSKaebtWggN+91f3AnGekOpXmdNFhmhdr1gIRb1JqrFq0dHWUJLknWd2/DO5aFsP4wDWU+mZ9kw9mLh/e/evfTnu9/s3v5hzHVkxYpWZgq2E2nDawRyplGmhvM7TGUA1z7RTbWCxAY01V3mmpdalUFihOGpBm9c++MxFzNTdZhVu8ZBhQXF2uX9pWiMFulWoBgd04XaH1BWLE69yOmJSq2d6fNctpsh52/eTv4F/r262EjzoGa9E5OkV4ezFrLC1qT7AGsyEc6rAUQrld0YXUxLP321RJLXXJZllxEmhUSsST7kiAWs/mwuM+LulIjgx2kxQA+2EArXa3rLAaaDSdDP2HDELg9mRCHYqg03W3PvjzOL/b1efnVjbG7XrYx9a0WqGNHSN3p/c69CjI3WrSQvqaBUEIyZn4vbFG7b0CW2Lw3X5Z5OUWdlzkj6X3Pepjq1iDO8kx1XZwwdEsSZkyPMDDDU5ljrKOGIGVfRzk8F0Z2WT1YpM6CY+LZN6Xc55/ETXfx4Qd58zXGBV2SNBPMQZi/1bZjEopmBmucO7WyYEo9jn7rw7SxY76Sb4fBgGU5LfNU5wqo+uVohFmdoi619DlrGDOMYaUSKVOtQkLssM6VgKHMt9sS6o6WuesO7CGFd0VSZ9/a15U3+GF3Njx798/dfhiyLJkJrPB8P76/3XX+GGoOWQudedT+Y30erYkVq6FbiXkznFBK527KmKf94XA6vY2zYelLShm1TlHiULsBpxabJkBm6jokCV9VGk3cU+a0/dQjlUN3LFwqhsglNrbEyKu89z5vP+jf/e7JzopN1smRkaR5Devmt37ulqtwpM3V1l540zSbrfHKq5KhPRJR8fwYpHvpShPmnu7e63o0OgRqqWGDp7rB0zzTTMRcZduxt5PRPK0NTu3CgaXfLFOp6HZxC8Pcl4ogkJrrGDu9fD/sloe4OvO5j4oKCegLjPLRX9WL0ZeWxPioRFlVNW1lZa7quhVjwGpUyuOLmusQUgY3A+f66YanqaMXP81hY48p0GnORDQQ0mQD+7KEWL3XYzwVoBxyqOSmNc6etjkVc57k/XEZxpHvv9bTD/o7PO8yC1JpwhDuwNa+PQ3bLsBswT/tAzdnGZPtDuT1Ou51k3x/rdl8WY7RJOjGUsw9OSge0NPL8V2lWz1pHLxWAbIO0tCZLc0KkiylJYG3riHLaSCz3i80AbEs0XklQzZOqByv99++LNd+/6arAj0ElEgjN+Xl282Tba2QoSknVxpe/z1hpgG0xOON6S0PEtJUzvdtXE+EeUkVHJZlPHe3+28W887mKINNJ6Gic1Sx1LSk9d6UxdHGTUDmpxITPdsd9R2cR9b5OJyLp934nlyedsv+F9NP728K3AW4vNTBxh1+8+78bMj62FpDmXLTYzLDmiezmo6TrjUEliQVy4t5aaMDiGBnfe4Tu53i/vXcu+OoYWANkoT7nNSy9MVKItt9go/1lQiy2EJ4Av0REUB/DqFEHSufjf02TmfdD1/84en1TgQUsBy0G4Bv3t8/ueoVDWMCYE0V1WQfIIAQHRQYJckF66WhTc95eopa2oToTNrYL8+ts3o67CtL7zU6FKmSsL5kqGAuVqyrR0sBlZ5Yffukna6k6m4zobI5GPPigfKucq46fjLmwK9/cPH+h0MttkTRstkSh9u7OL96orU6o3XZSQneoKfmTs8EGshM0Jp4EC3AaNqNpydq9g2T3AArlsc3kxlHj6UfHTUDLKCCJaccvFTUpU83ekuCaU4CbTCruGk2S8roNltGlnPw/qNvNH3+L7615bP+/VaW7NEb7k97Ha52xRdbAT+0TBO5rWFUCGO7Ht2Stspbm+2+9aVC1bOXYesJRqf5MoeOr+at1Q0V3necFisTBzBhmnNktXmhUxaJfq1MpBO7wx29LkkLg6WYS1T4sNsg9HR6G08xnb/bfb4sVDVmHO7vdPHiw0tnI+ASUOTjy2+7HaLa5MdWch1Qg45XPsOk5elcA5I7gjR6f7x98+1UMjvDrM5znmosGDqnAVWlRrWYqciadaEhhMb+YaP0HPuahV60ycSZkHFguXN9OMf7s75Cf9d9sJS+z7lmd3Z1vnMJqkttTss0ga1CrPtO66ELQEascrCGU2Pt16fzbubj/cmkbMDD0caxGw0VVjwWlEzvMxBGukjYwFColdUkmx/K/eT9qCVAVeND53nq7yNxOj7F7XH3bP+g8+P+/MObYI2lFi/uZsVcrYeVmtPIRLpB1m6HYpM4tXMkH0VTtUbVmuoizN3ZoU0L6U6kaPRipaNqluLTUs2NDnRMLEsQNQ2oy2LjMFz08YhDW3nz+3lJeJohasn5fFnOjMbx26vndlyuNg/7Z7tv3g/5B1Bdj0AxMyFSxCqchTXAM6PdwfU45rPpz9YxFu1BgdV0mDUvTwlrCmElICuAuaXSvKvLFFHTO7rFKhxWmoJA32XgjKsOKxPjR2dzjpmyKsLf76ouK1DGef44336XvT+8/YHvjxdXwzQvqW5wQ9bmWo0107Vmgg1Ue7x+qMUNklzZSWRjC9C0dRKJ+VkuQF0DB5prgL0J8KEgwc4SfTEtQsZ69RqcyTGr5t6j6cEA+FkRKqwz0W1Tlw3n2a2byx/Oz+f9ha6w3FxS5erDfJgr+3YbX4sk4+qRa/os0dsEDkDREl/Xj26QU/UxiG1lDY5n3UkqanGUpIJwZ0aWEqdaBhNL54CAUGQ131iCaJGIwPeeCLt/PfWaCxBk8a689SfHqRTJltdfeL0YHtiHTrfHt/anO216BemFIE1BFj4ylGx+Laxa1CbQbjePZW0EBR6n8VUOWcvFkrlUIsMcgHs3KJUl5zlASKU3gqjzLPPMrvz/AL8n+AMGG/NIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=198x198 at 0x2024561F358>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 输入是一个batch，batch_size＝1\n",
    "input = to_tensor(lena).unsqueeze(0) \n",
    "\n",
    "# 锐化卷积核\n",
    "kernel = t.ones(3, 3)/-9.\n",
    "kernel[1][1] = 1\n",
    "conv = nn.Conv2d(1, 1, (3, 3), 1, bias=False)\n",
    "conv.weight.data = kernel.view(1, 1, 3, 3)\n",
    "\n",
    "out = conv(input)\n",
    "to_pil(out.data.squeeze(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.761755Z",
     "start_time": "2020-12-10T06:06:02.756769Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1111, -0.1111, -0.1111],\n",
       "        [-0.1111,  1.0000, -0.1111],\n",
       "        [-0.1111, -0.1111, -0.1111]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Module能够自动检测到自己的`Parameter`，并将其作为学习参数。除了`parameter`之外，Module还包含子`Module`，主Module能够递归查找子`Module`中的`parameter`。下面再来看看稍微复杂一点的网络，多层感知机。\n",
    "\n",
    "多层感知机的网络结构如图4-1所示，它由两个全连接层组成，采用$sigmoid$函数作为激活函数，图中没有画出。\n",
    "![图4-1；多层感知机](imgs/multi_perceptron.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.767739Z",
     "start_time": "2020-12-10T06:06:02.762753Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 1, 200, 200])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "池化层可以看作是一种特殊的卷积层，用来下采样。但池化层没有可学习参数，其weight是固定的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.773723Z",
     "start_time": "2020-12-10T06:06:02.768736Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool = nn.AvgPool2d(2,2)\n",
    "list(pool.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.781702Z",
     "start_time": "2020-12-10T06:06:02.774721Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAAAAABVicqIAAAZt0lEQVR4nEV6Sa9lWXbW96299znnNq9/8eK96DIiMiOzsquiqih32FbJNjAwMp5YDJAQA5gwYMIY8Q8QI4QsMfCkQJjGSBjbYKrswmVX48rKqsiK7CIjMzKjyXgRr73NafZeazG4YXMHV1dXV3efs9c63/qazd+hmrnUlfRPPmyb6KSHOkYJhCGAhIkL3IsQEPeQaTQ6SumGCzvj0FFE4UIyvT6xb789rkg4hRpd8952LDQ3GAgdvBQxguJ0E4BuQndXKpywoIRR3d0AOBzeZQ3B4AQc8O68CdfuqIoQhNEZ5eP/LTCQjMFVzeEW6Eo3pxlIVwNAd4cTBvD58iwghXG+zB5iTFFIV9UHHa69nIu7uxkIxsd/+UTcCUgFLYV0Ak44ACfhpACkgySFBBhIgrBECCXYIpeYggQJIBznh1p9eVrUGSgMVfrwRz3EhUIR10IhXGy1isEdbu5mZg5XLcXVzZ5/62YwBuNQ3Al3GAE6Hsx994tubu4IKXx4xxBFQISKXiBW1I0AHCDgZgDc3DzQclZTs1KccCfdQRdhKPZ8a0gRon2cw+u7xYXu8Hd+ko0iJKUKdAfIQDzfLJiv3kAXV4O7GszcHauqA4CCsFLMnQ6nUISfner6L4ScPQR5/x0PiRARiWJFCTMtjmIA4OrmMHNzBWHmAlgpZv5X9wkDGGmmpZibwwnAkD8ruHazL8rw7k8s1cQgAGPuB2B1eU7CHSThBqGbwBxOCVKPa+SianCKgAZzumkp6mYGdS1FcXhk1VeanP3Ddy1Fc/VID9qWMHGKCAxiXD2CIEgQEHNvQorjtRh9dvy0j4xiBqdT3RwFoBsBteKGcn9jevFvvJU+umMhktklepB2kddIAOZugAACQFzECHf1OGlymFZMqpsXvnD0wcID4W5mcJA0JWEGV1PQnz66WX/l8Tt3LET0Jm7CShfHbRSSsOKqVuhm7mqmbjpaH6nYktL3Q6uBiFd++WbOpZhbMTUA4l7MSynmDnfaRwvbvHG7MImbEEEqdiePlSRI6qBqMFMzdbhbiuXs3HVou0E9u9TBLcc3fyEscimrX0KEbsVZ1AARhtDd0/ydXhKZDYUiKc8fn8FBqJqRgTA3e35VcrToYVB61/aMLE4Cvv1r1+bzvpg5nRKCl76jAAQgkOrp4Te+FYUoknJqTqO2x08DSHeHggI6VkAlxrXT0EWJFqPTy9COEqMIneFn198/HkNpihDE2zZILMEBoVDCf/y2S+2DN7kaH31JutkjE7oLV93kTkJWkMXSZqHCtMDc1VOguufM8cbX/+nmbMjFCCejHz+ZQ8xJAowP/0IZvViTsT5c+jVZPjkVAQCYklSBrOCREHlmVQwciguAwd1d2Lb11lrQNvxGbrWompsztPeedgICAKT907lX0bUaOEm7X09y9FAC4e5cbakJnAYQkHkWUtUDA2NMpiUvziYHI1VV7a7++rwr5k7CQ3z62bnTAYD8/nlsAkrtNpqs/czmXB71CWbucDoYAszoJN0l5ySiRauYWNWelXaEa+MM6xXU/LWvLOeDEoCQ5elxK3CA8cHtIYp3Il7JxmtbeV2OAoTmK4A0dcIFDoCxje6Oqo6Mbjm7leP9XZXS5oCSfci/9Q+sVQdIQTi7f5IJY+z+LLMWQ9WH0Nzc9kktCGAQ+F/Du0AdTht3fQkwYxUnElBKWcqNDe3mAxJDCKq5fPXvzAZzdzcJ5eGDucMpbz2NyV3rPkJe3S+xgUTA1QG4rHrLhUJovroQFEqMSNq753m5cZ2tZaMIDKpa3L/+8lIdBBCku/upX6iqB7dDim7Bg7QvvqjV2CzC3N2wGigr8Aows9FahkQTiZUKbV62Xlpf1oOOaijEjBAUn/76vzWYQAh4++Wf1+B/lFOCGDUtrn3JwzoU0Shubi50z6pOhxmsvLB0klECS65k1t46sCVmo9oLkZkiPUez5a2rnQvdNeeL/+xa2Ujpq+8DyLSo06+O8oYoXEIIMThEBHC6iASBBLkyz0DxkPo26qx/Y7/vB29ywJB7iZ7V3IHBfrOYCLVdfulfbaXpMOteA0xtgPibl/KEBY4YGGzFSUIIIQkoCOK+cVe8b2LoMJbWX2nmIg206dmHZjV3hiqItFe//qmKLvt/9IvdXmpzTfEgnYcw339Nm8rcTWIkSMAhIhSQpAJo4jms9t7q2GfexJlMKg3ehflIvE2s7FlT04uWW3JY8sVfqO2CtRYTngZjzMHwlVFuHA5oJJwxEMJAQuA0V+S9ZaxtYGPWYrRjg05cBmvkdGreSq77oZEhGGTe7nxev7jeVql3CUJ8EoMVIN26lms6aO7RxUhIEAbCYeJG47CPJngIEi3KLpc6jn46TUuMVHqJaR7qXPmQgyvTjY3B15FdJAO85y4ews4blAR3AJRVC7sI3OkON7i7bVq1NSIy6Ttl6JswW45tHpKxYOj7yL4u/aIty/PMYJNmKArmZ6onH6k5U76+YzWdpDBEA9zwHCJDoBgd7rGMB+37SZF9xfl2MJS5r4k6tIvdEOrY21APZ9wWroUBdAl8tMy7d5ZJMmz/utVhxYDEBWYGEZiDWJE0QEuwECduJewN+WQqfWZYxq6YLdT6bFh2XTssumYD44m2mUIJZ3P73L5lUSHTL048Pcd2IAJwrgaKCAADYdARYzVMtd4puthOfW7CMM1h2YdUMDIbMHeWZttHsYNIiEDEY7RVfj+pRN+7bBVdIeZ0RAQLFR0EaWpOC07VRkbZh/PNnu2W52UK3drJPI/RPHh51Fr081TZfkgj7z2JRELi4/nH7a+8PZ+6hHAzWsKqwHCPhAiFpK9IIyMImFW1+2i6qBfbyF0Vl9PTpcw+ib8aP+05OZ63p7+1m9eawoAQIE4++cH78rP7/7pyrcrBQRmJwelwh0cGZRChr+iiwwkynI/XHh55jic7krtKuumJ8lTH7U/LXYyPj5b1L1bVtmW6B7rTP/7Jw3byxmv/+agi2L8YWIEQd9AdMdCxamenkwBocD68ao7JcL7R5LZiHs365mg4PGIt69XD2KW/d2F7VFYFZAjPvvlA1196+cq/eye51rhwYGti9NUyQIwWXOFGBwS++hCqD3/OpmepH0+GRc1uzeejD878tH01/vhKvxxP/v7ajaqrYSJCefb9ki7v7k42f+entRQP/kqDaiWnnE5oJFalAAA+ZxMu6XAZd8oT27TjqffVUSg/HvJZmeCnlzbvLa//avMF9KF3VuDwR+9eDa9MQ9r43e/VwS3m8bUyJVasAnSnQOCqIEnC3GHmCCn95Thvnm2U0zXkCt2dDy2fH7TbH9ni2YWtV5o3ksEHNfHhm2+txXRqo/jb366iwMkrwRvQSQGFhAtAz8VW9JPqBCkMox893YqjPNSBkhfHqbYn07K2QN2Wza/pFzBkU42hWP3iVjqrS/Po39xJDX1gGF9aVgG2Qi2u/s8dRd2dpJg7aABjCnfXGilLWXgoTx4/Y197XR/1i2U6Tde75bzVIrlpmnjl4OH5o/Tw956wkugabHvDR6u5T4qDQDTQzZ9rLAmBhIOCVJrj4zSe7/jJe6VdxGo9VzvLvLy+d+HF7TQsK5lN1mu6VLu4f0VtISKIA51XpYGt1K07CK56YIUqTllpGglgqI54Nw9lenR4cshzOVPWn+6Nlld/7o0X1NlanFejSKma6e54ftqzDQHimb61z/V+MSjcV8V3xOdiF6S4CBnopru5VCcnfZObMHt8JKVJ/XG8VC8PL/7yzv6isHg8aeokZJDqwkY8fnRx1IYQe1T5YkrNoB2CiAhgDkSaqa/AHohR3MnQeUS5vTHb5OzzuS+3BZK6Nuver0z2xv6sPu+HuhIBhQjrm+OzY6laETdBvOmTyGzmvTMKHHBRd/pKgtLMAQqliJZ499laHD4405RO9vqSJs/KO39rcqFjM1qcaaoDFKrqmOxs+LknSvAcyuUpa0McjaoE7xftYCvShRXZptMNgMGJOLl4Wnl3No8NuT0bwrDWnH0p3BgVVuGHRzmmaEPRUooitT4fNkYSMoVXvKK7e4ixChTPXTcIKEJ3CCBOPrc9KHuPv/Pu1c/e3z6/1PGsjzb1buP6a+MU0tGH8vhP/uTjbH3RIfcP/u+dTJ1tmlgBmt0ycTXN2Uxi3VT0vIgQFDOHUSCkUJ3McmBbX2refhz7gwcb7eLq/TIft796c7r2+e6d++P5fp795fx6rdFP3/l4GO31pVsrSU2420h0d3cXuAvpVjxCXYsDwkJRNQNM93clxpe+e7Z76eHupxfq46MXP5odf/XC1pqsf+etrhvGy1dv2dl0Mw0//uGztokqQ53YO7p9GeGvjB+H0dSDSclD3/cuAMyMrqp99Utus/vfPLyK97Yjzprp6fyirL10TRr84M8fd+36P3zj3sz7IpPv/Pfjr+0vFy6tjFjo9UWZrNwrN4eV0ucua5Tn0oESwBiTQIZP3n6h+7OPFtcfjXzcIT/bap9mvLGVN+Wdp5vdK4drr974b7mtMT7/5JLvlwcF9VBPjyHlwlpY2XzidDOzknNYjwLj6mGkhCqm6B/Wr+1t/8F7wxufhcmmbI/bpSYfbly7tDF6+7MLVz6uD26V+Y1ZKRg/2j4Y7d0elVK1o1BUeJE1zCzEImZqxYpc3JxGYS6ucEpY0fl682+W9/7Hh5cO8mwzdKN0cFeyZHn9cpr88N5eHV/uZw8fHGwuo+ey+dLR/NvnFx+59BueEdIVrc0d5q4oWhTbu5OIqHnQbjAAJM1M9JfO3rLDapHSeFZdPmtKVMDeXN/bvPPWpTSOLFuxlqFin/Noq9CnZ4dZS6UObcaB7kYoYGZFru5FN8S+5FzM3Vdz3sLoWG6+55f2lt2z6Xa7OU6jZZHJrUtrh7//QnBUVaXitqwzhzytm36MgRFt0sKyLw1tZd+alcIbF2N2SBxU1QCsTFYbb6TZu/l8I47C+6n7+ODz+38Xd5nfbK7w309qumkI0X1gMil91CBMwYJ0ZZyTHkizwlt392JXL4iTIcZetbj+tVfZmS42h5g63svR20fBhhvvaX312vQ/Pd0W0TIMManmYq6pD/Qg6NRkOWx5hc2qHoDVvmjZ2A9GSZFSDJodZmqOsp4nm0PTcNmnUZ1KHpW1gK+Ul3nzs2+NVAHzoR9US26z6mI1r+dDRFc2abtNWk0mN9MSrtTurBpxEYFpWTE903AQn85jN9u/FLaK9WF/49YXPjrxm5ftt3Ws5qamXTsUy6VY7svI3MupV8j9BP1+qNQcQsLNdjeBMKqEjKLQXIwUF+IsxYHzfmqdLwvH0r4of3i//xr3/svjagoz10KziL5vp46+1AO0PWIwnU/Hw34IK7YAQsNBUtbRXCgxIIRIOMiwvbnQszBZhlMs0uVY9/bO4s78hd945fjtIDUFBjM3Vc8hS2kXCyvl81agPK+m9Vak+8pNdlufOFMwBynFJEw3K8DBcH5oGaJnBz7YxY1uI9298nloXhvv/oejxLpxw8pYRxjz84fHz54eQ/P9EIKHZ5hcraK5O9yKuW9GxIqMgYju4Np6glLgEFRnk/eufpImJV18e/PmN+flZ1548uQHT3YW3nSBZhqc4jJhNfKXmlPPj+eMI4tHGO8h+mrw0SxMIUkcdGdUh3otDoEQLhvJbj57utPw2cEtf7LV/e3LH8jZW2Ote4+BJIskA9jUV2VjyFruiddgPOH0gothZbO7pwohAjQhpZRsNCcdhNr2mV98OpvY2cmhXpl+uvvqHu6v/25US6W3Qrq5dkOfi4zWpxGDHZ4giJt1w2QUVvGNO1BSkooQJ4HYe4IbVpGBXzgZD+0MMmUbPnj5k6u/sjf6X1e/W3aO2HgfBgvuMLQxINq9zT4fV+8jMCCFTseBgDudbu5BquCEiBhiF2hBJES66w4wzQ+YlmuT+fn2R9M3dl76r/X8ETon2MfOUChV+zFGacij9aPR2uFhyBPz3CyXG2klpUAjLKVqlWuBiO6G+rkkmsqnrx01dnxhujbH7lE8aGbv9y98o+4KWeoebe+BUo2vt5amu83avVK++dwVrTmsizgQFE53r+tgJEl3jwYHi7sANe5c70/X8tHENssz3cV+Xr/7xW+UlDTCUyFaqyhDrG3w7lQeI75/FCEamBCtdkDcxRVummpxhJXvLaqZKQ8GsJ/p9snTyaxqZ7vlVldtPpylV/7wSEiKqPS2OGv7Yrkg1qGcnDEevueeaktCRpUVewugmxVEShDkghCjq21Mq+JGMMoih3ZedViOm6tPriy4+cfvBY0O62NcSjmlJINJasIk5dLd7hliseRCKVUWN3cC7mpB4FATMXhUQXyeEsJ4NgwTicPoyafcGOzgo+99ly7RgQxK9uX5gaYoMU5iQpr1jxgTiLoUrW1VCwfdzDGVohBR9ZUV1fVR6CRR/HG3E8UWG90DuTisjX80CBgYNGgQZ5h/0mFtSg6JinyoqYYMIZTARNeOTndTc5UpDUItunz0bjTqIi/pRmh/dX+4fxJtUHnx0+Vivb5664+fhroyybTciOfdzk9N1WKLUnIfCDoE0cJQ94cYVSGKA6WMpmpQL7MP/uynOY5sWc7rAhjzC/hztjtDHreznc8/fFMe8s5v/v7DldUGL8G69ZsXmhiZxBZhfOmNr/zFu52WBgayXvbaUUJT0a3sVYPSyuz27x1Od+OLz8RmYaFuYfvsT6/9qC95uPn4J1++sHfw9nhjsnhz5oWRFrwkJ8ZBjVEHQzUNzauvPf7Wd/vkRel13kDfd+UkBpEexYRY3P6D053tKK+9+cJOPmvdGbsnr2CG8wuzyd7RvZTfG9fVXvfuGzY44JG0EQfNRnMUDXWSnO3yP/4X13KXc9YQJuPJxtak6s+O/eTYIDZ75/e6g61KRC6/ernJIFGGW5MPWJqtn+kX23gwPJg16xvv7Z9d7EMIZEJskjtoEsUQk5eSi9oX/uXrx+1gfZIU63q0vr65NlmvdkISe+cbabder1zacOHqhoNRiKvDrF6z248+XdgnG58EHHeyV23vDwSDSKimG22UXCVzpkjt+5yL2cY/v7woLoEEQ6iayfoat2+E4Cc/ibvV+hoauX02PthxUoLIk/vLbCpPLobDnXf3hiXbxSXIhmT3bOjL2lYuOg2gC91yybmYmq3/k9jO4io9EYYQm2pnR6D3n1ySeqOwkbuP02hzcILkbAJMFirrOc7Gi63dxdEQ5YdXNsoqEdOmztbQnUKaWlY3czO7+hv9slFx54pG+ORGQ+8+CKGZdotaZDjxSgZzgvHMY8zFZNo2C+SFTM9SOMVOTrES7zk2VpQiMMOKqKkXB81+bbqYiJCAmaluv3y9oj0+HFU1lpEmed7DzQgKtp5K3Wrejb1NZs/aeT337mDz8XZyDSqy1kyjhEjY6uXFDOpwXf/FYbpip2ZZ5fWDStB+kL2ue68B8aVKTSXgXM5GS+fowmOItnX7VDYD49cflLhkgK5dGpc6mAhBNxhUV6cC3PTnMQGeRyQ63kmRevgkMea+EluKF/WmUtgwFK63CKPRR1ZTg9R867o8u/tELl/ZmjplfSQ5peAUFjM1MyvqbmauN/Yah6k7xTkJg7P7RELCoFGHHNl1UgU3JobQimEQPQl90+a1tjmDjn94ht0tn7cbGzG0kxEDVla1azDVnMzEvf5iQ0AKAMZQEO34qdGHeQyd1ZL6VsPIzOHeNMhpV72lIAi26iiPN65vyfGDjwefjmGn0yoF+f/muxVdlUd/Pg4ZTCII1fLusejpcsg400lXAoWlLXFi5gb6JKTR8ZAYG60mZyW++8A+uHXNtM9DvTbivNuIpJDigLuWbJrNQPjr9SefHrUmkupp+Pgk5aO81DxMy0CcRw1DaoKaC2QuQealif30LGtMa1e/9/7W+r33XW2YbK6LP2w2CIJ0WRkC9JxKcAc3l59xNJ42TTNtTrHN88PT2fR0HYM058eRuQ+jaTEHuajJISVVbXKVy4CffX/r5MfI1DDamsrZJ19e+SYAXElQNRR1DQLuNjIsHyNMb1za7zbKxx8/XluMxi3q0vayfREyHtFhDPMyWBWHQvdaHffvb7+Gqh3ENK5vQG/vXgiEm4is4lq3XLwUNXV/Ja5vraXlw9vf276yWZ/d+RTFtvrBYp5bvInG0ubCKWIh5BS6yioVM4nxXvXK1INaqbY2Jrjtr1R4Hh2YuCghtCGYqSgOxmiE9JNh09e6H3zf6nJxOJdJyqkW3Zo6xoQLWZaiQwjW18MFGdfV+vFn3QMPLtPNtdn3+9enz8WaQ1Zqqrjr0Gd1eLrWeYjTzYOd5Zb9n//Z17YTTrWedpareLo960OgEc68LA0hEevtxixXEm3rrpbA0MyP/GB/ugrYAIqYKE3MSvBStAT3l36aYgrc5nD0p+/mTVyYHM0m69QZLJ5+jn40CgoRyUdBK0kopR9d/+kLH758PBw8jaD0zeXtSYo0BTU4QVAIh7karAQVv+a91FrX5Xb/sKm78ejkzHYm2g+5j+Gsks3dJyTQPwvihI6fpb7diqPp07WHEzWm7YOtKiXh6riB0whBMBe4uWaREIKtb8zAOoHzfjOcjMZHC9najOenfTEp7XJ77WrtbmaTPohIZfUkVecv3dteVJ/bFcXW/kZKFDcJukq8HCRDogQHTM2sQG7kkoMEidOdkCZ9CaPLo+XJaQl7Yrq0AiGI/txNQtXkF2e1HY5Hm/1MH9ubO9tVCBJAcSdWh8ugcEEg6aZqpup2qxmGITTV5ELs8snQVDsb/uzzBXYP/h8CwMEc4WwTQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=100x100 at 0x2024AB8A668>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = pool(input)\n",
    "to_pil(out.data.squeeze(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了卷积层和池化层，深度学习中还将常用到以下几个层：\n",
    "- Linear：全连接层。\n",
    "- BatchNorm：批规范化层，分为1D、2D和3D。除了标准的BatchNorm之外，还有在风格迁移中常用到的InstanceNorm层。\n",
    "- Dropout：dropout层，用来防止过拟合，同样分为1D、2D和3D。\n",
    "下面通过例子来说明它们的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.788684Z",
     "start_time": "2020-12-10T06:06:02.782699Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.8090, -0.1497,  0.0473,  0.6063],\n",
       "        [-0.7525,  0.4366, -0.5472, -0.0942]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 输入 batch_size=2，维度3\n",
    "input = t.randn(2, 3)\n",
    "linear = nn.Linear(3, 4)\n",
    "h = linear(input)\n",
    "h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.797660Z",
     "start_time": "2020-12-10T06:06:02.790682Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0., 0., 0., 0.], grad_fn=<MeanBackward1>),\n",
       " tensor([15.9997, 15.9981, 15.9982, 15.9987], grad_fn=<VarBackward1>))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4 channel，初始化标准差为4，均值为0\n",
    "bn = nn.BatchNorm1d(4)\n",
    "bn.weight.data = t.ones(4) * 4\n",
    "bn.bias.data = t.zeros(4)\n",
    "\n",
    "bn_out = bn(h)\n",
    "# 注意输出的均值和方差\n",
    "# 方差是标准差的平方，计算无偏方差分母会减1\n",
    "# 使用unbiased=False 分母不减1\n",
    "bn_out.mean(0), bn_out.var(0, unbiased=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.803643Z",
     "start_time": "2020-12-10T06:06:02.798657Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 7.9999, -7.9995,  7.9995,  7.9997],\n",
       "        [-7.9999,  7.9995, -7.9995, -7.9997]], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 每个元素以0.5的概率舍弃\n",
    "dropout = nn.Dropout(0.5)\n",
    "o = dropout(bn_out)\n",
    "o # 有一半左右的数变为0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上很多例子中都对module的属性直接操作，其大多数是可学习参数，一般会随着学习的进行而不断改变。实际使用中除非需要使用特殊的初始化，应尽量不要直接修改这些参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.2 激活函数\n",
    "PyTorch实现了常见的激活函数，其具体的接口信息可参见官方文档[^3]，这些激活函数可作为独立的layer使用。这里将介绍最常用的激活函数ReLU，其数学表达式为：\n",
    "$$ReLU(x)=max(0,x)$$\n",
    "[^3]: http://pytorch.org/docs/nn.html#non-linear-activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.810625Z",
     "start_time": "2020-12-10T06:06:02.804641Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.2325,  0.2955,  1.0767],\n",
      "        [-0.9411, -0.9849, -0.1090]])\n",
      "tensor([[0.2325, 0.2955, 1.0767],\n",
      "        [0.0000, 0.0000, 0.0000]])\n"
     ]
    }
   ],
   "source": [
    "relu = nn.ReLU(inplace=True)\n",
    "input = t.randn(2, 3)\n",
    "print(input)\n",
    "output = relu(input)\n",
    "print(output) # 小于0的都被截断为0\n",
    "# 等价于input.clamp(min=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU函数有个inplace参数，如果设为True，它会把输出直接覆盖到输入中，这样可以节省内存/显存。之所以可以覆盖是因为在计算ReLU的反向传播时，只需根据输出就能够推算出反向传播的梯度。但是只有少数的autograd操作支持inplace操作（如tensor.sigmoid_()），除非你明确地知道自己在做什么，否则一般不要使用inplace操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在以上的例子中，基本上都是将每一层的输出直接作为下一层的输入，这种网络称为前馈传播网络（feedforward neural network）。对于此类网络如果每次都写复杂的forward函数会有些麻烦，在此就有两种简化方式，ModuleList和Sequential。其中Sequential是一个特殊的module，它包含几个子Module，前向传播时会将输入一层接一层的传递下去。ModuleList也是一个特殊的module，可以包含几个子module，可以像用list一样使用它，但不能直接把输入传给ModuleList。下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.819601Z",
     "start_time": "2020-12-10T06:06:02.811622Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "net1: Sequential(\n",
      "  (conv): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (batchnorm): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (activation_layer): ReLU()\n",
      ")\n",
      "net2: Sequential(\n",
      "  (0): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (2): ReLU()\n",
      ")\n",
      "net3: Sequential(\n",
      "  (conv1): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (bn1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (relu1): ReLU()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Sequential的三种写法\n",
    "net1 = nn.Sequential()\n",
    "net1.add_module('conv', nn.Conv2d(3, 3, 3))\n",
    "net1.add_module('batchnorm', nn.BatchNorm2d(3))\n",
    "net1.add_module('activation_layer', nn.ReLU())\n",
    "\n",
    "net2 = nn.Sequential(\n",
    "        nn.Conv2d(3, 3, 3),\n",
    "        nn.BatchNorm2d(3),\n",
    "        nn.ReLU()\n",
    "        )\n",
    "\n",
    "from collections import OrderedDict\n",
    "net3= nn.Sequential(OrderedDict([\n",
    "          ('conv1', nn.Conv2d(3, 3, 3)),\n",
    "          ('bn1', nn.BatchNorm2d(3)),\n",
    "          ('relu1', nn.ReLU())\n",
    "        ]))\n",
    "print('net1:', net1)\n",
    "print('net2:', net2)\n",
    "print('net3:', net3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.825585Z",
     "start_time": "2020-12-10T06:06:02.820598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)),\n",
       " Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)),\n",
       " Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可根据名字或序号取出子module\n",
    "net1.conv, net2[0], net3.conv1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.837553Z",
     "start_time": "2020-12-10T06:06:02.826582Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "net1:  tensor([[[[0.0000, 0.0000],\n",
      "          [1.6621, 0.0000]],\n",
      "\n",
      "         [[0.4993, 0.3580],\n",
      "          [0.8461, 0.0000]],\n",
      "\n",
      "         [[0.0000, 0.0000],\n",
      "          [0.3035, 1.3322]]]], grad_fn=<ReluBackward0>)\n",
      "net2:  tensor([[[[0.0000, 1.3669],\n",
      "          [0.5347, 0.0000]],\n",
      "\n",
      "         [[0.0000, 0.0000],\n",
      "          [1.6539, 0.0000]],\n",
      "\n",
      "         [[0.9862, 0.8709],\n",
      "          [0.0000, 0.0000]]]], grad_fn=<ReluBackward0>)\n",
      "net3:  tensor([[[[1.4161, 0.2165],\n",
      "          [0.0000, 0.0000]],\n",
      "\n",
      "         [[0.8232, 0.0000],\n",
      "          [1.0589, 0.0000]],\n",
      "\n",
      "         [[0.0000, 0.3910],\n",
      "          [0.0000, 1.3766]]]], grad_fn=<ReluBackward0>)\n",
      "net all:  tensor([[[[0.0000, 0.0000],\n",
      "          [1.6621, 0.0000]],\n",
      "\n",
      "         [[0.4993, 0.3580],\n",
      "          [0.8461, 0.0000]],\n",
      "\n",
      "         [[0.0000, 0.0000],\n",
      "          [0.3035, 1.3322]]]], grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "input = t.rand(1, 3, 4, 4)\n",
    "output = net1(input)\n",
    "print('net1: ',output)\n",
    "output = net2(input)\n",
    "print('net2: ',output)\n",
    "output = net3(input)\n",
    "print('net3: ',output)\n",
    "output = net3.relu1(net1.batchnorm(net1.conv(input)))\n",
    "print('net all: ',output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.843537Z",
     "start_time": "2020-12-10T06:06:02.838550Z"
    }
   },
   "outputs": [],
   "source": [
    "modellist = nn.ModuleList([nn.Linear(3,4), nn.ReLU(), nn.Linear(4,2)])\n",
    "input = t.randn(1, 3)\n",
    "for model in modellist:\n",
    "    input = model(input)\n",
    "# 下面会报错,因为modellist没有实现forward方法\n",
    "# output = modelist(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看到这里，读者可能会问，为何不直接使用Python中自带的list，而非要多此一举呢？这是因为`ModuleList`是`Module`的子类，当在`Module`中使用它的时候，就能自动识别为子module。\n",
    "\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.850518Z",
     "start_time": "2020-12-10T06:06:02.844534Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MyModule(\n",
       "  (module_list): ModuleList(\n",
       "    (0): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
       "    (1): ReLU()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyModule(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyModule, self).__init__()\n",
    "        self.list = [nn.Linear(3, 4), nn.ReLU()]\n",
    "        self.module_list = nn.ModuleList([nn.Conv2d(3, 3, 3), nn.ReLU()])\n",
    "    def forward(self):\n",
    "        pass\n",
    "model = MyModule()\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.857502Z",
     "start_time": "2020-12-10T06:06:02.851517Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "module_list.0.weight torch.Size([3, 3, 3, 3])\n",
      "module_list.0.bias torch.Size([3])\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，list中的子module并不能被主module所识别，而ModuleList中的子module能够被主module所识别。这意味着如果用list保存子module，将无法调整其参数，因其未加入到主module的参数中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除ModuleList之外还有ParameterList，其是一个可以包含多个parameter的类list对象。在实际应用中，使用方式与ModuleList类似。如果在构造函数`__init__`中用到list、tuple、dict等对象时，一定要思考是否应该用ModuleList或ParameterList代替。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.3 循环神经网络层(RNN)\n",
    "近些年随着深度学习和自然语言处理的结合加深，RNN的使用也越来越多，关于RNN的基础知识，推荐阅读colah的文章[^4]入门。PyTorch中实现了如今最常用的三种RNN：RNN（vanilla RNN）、LSTM和GRU。此外还有对应的三种RNNCell。\n",
    "\n",
    "RNN和RNNCell层的区别在于前者一次能够处理整个序列，而后者一次只处理序列中一个时间点的数据，前者封装更完备更易于使用，后者更具灵活性。实际上RNN层的一种后端实现方式就是调用RNNCell来实现的。\n",
    "[^4]: http://colah.github.io/posts/2015-08-Understanding-LSTMs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.870465Z",
     "start_time": "2020-12-10T06:06:02.861489Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3610, -0.1643,  0.1631],\n",
       "         [-0.0613, -0.4937, -0.1642],\n",
       "         [ 0.5080, -0.4175,  0.2502]],\n",
       "\n",
       "        [[-0.0703, -0.0393, -0.0429],\n",
       "         [ 0.2085, -0.3005, -0.2686],\n",
       "         [ 0.1482, -0.4728,  0.1425]]], grad_fn=<StackBackward>)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.manual_seed(1000)\n",
    "# 输入：batch_size=3，序列长度都为2，序列中每个元素占4维\n",
    "input = t.randn(2, 3, 4)\n",
    "# lstm输入向量4维，隐藏元3，1层\n",
    "lstm = nn.LSTM(4, 3, 1)\n",
    "# 初始状态：1层，batch_size=3，3个隐藏元\n",
    "h0 = t.randn(1, 3, 3)\n",
    "c0 = t.randn(1, 3, 3)\n",
    "out, hn = lstm(input, (h0, c0))\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.879441Z",
     "start_time": "2020-12-10T06:06:02.871462Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3610, -0.1643,  0.1631],\n",
       "         [-0.0613, -0.4937, -0.1642],\n",
       "         [ 0.5080, -0.4175,  0.2502]],\n",
       "\n",
       "        [[-0.0703, -0.0393, -0.0429],\n",
       "         [ 0.2085, -0.3005, -0.2686],\n",
       "         [ 0.1482, -0.4728,  0.1425]]], grad_fn=<StackBackward>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.manual_seed(1000)\n",
    "input = t.randn(2, 3, 4)\n",
    "# 一个LSTMCell对应的层数只能是一层\n",
    "lstm = nn.LSTMCell(4, 3)\n",
    "hx = t.randn(3, 3)\n",
    "cx = t.randn(3, 3)\n",
    "out = []\n",
    "for i_ in input:\n",
    "    hx, cx=lstm(i_, (hx, cx))\n",
    "    out.append(hx)\n",
    "t.stack(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词向量在自然语言中应用十分普及，PyTorch同样提供了Embedding层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.884428Z",
     "start_time": "2020-12-10T06:06:02.880438Z"
    }
   },
   "outputs": [],
   "source": [
    "# 有4个词，每个词用5维的向量表示\n",
    "embedding = nn.Embedding(4, 5)\n",
    "# 可以用预训练好的词向量初始化embedding\n",
    "embedding.weight.data = t.arange(0,20).view(4,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.890412Z",
     "start_time": "2020-12-10T06:06:02.885425Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15, 16, 17, 18, 19],\n",
       "        [10, 11, 12, 13, 14],\n",
       "        [ 5,  6,  7,  8,  9]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.arange(3, 0, -1).long()\n",
    "output = embedding(input)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.895398Z",
     "start_time": "2020-12-10T06:06:02.891409Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(4, 5)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.901382Z",
     "start_time": "2020-12-10T06:06:02.896396Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3, 2, 1])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.906369Z",
     "start_time": "2020-12-10T06:06:02.902380Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0,  1,  2,  3,  4],\n",
       "        [ 5,  6,  7,  8,  9],\n",
       "        [10, 11, 12, 13, 14],\n",
       "        [15, 16, 17, 18, 19]], requires_grad=True)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.4 损失函数\n",
    "在深度学习中要用到各种各样的损失函数（loss function），这些损失函数可看作是一种特殊的layer，PyTorch也将这些损失函数实现为`nn.Module`的子类。然而在实际使用中通常将这些loss function专门提取出来，和主模型互相独立。详细的loss使用请参照文档[^5]，这里以分类中最常用的交叉熵损失CrossEntropyloss为例说明。\n",
    "[^5]: http://pytorch.org/docs/nn.html#loss-functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.913350Z",
     "start_time": "2020-12-10T06:06:02.907366Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5944)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# batch_size=3，计算对应每个类别的分数（只有两个类别）\n",
    "score = t.randn(3, 2)\n",
    "# 三个样本分别属于1，0，1类，label必须是LongTensor\n",
    "label = t.Tensor([1, 0, 1]).long()\n",
    "\n",
    "# loss与普通的layer无差异\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "loss = criterion(score, label)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 优化器\n",
    "\n",
    "PyTorch将深度学习中常用的优化方法全部封装在`torch.optim`中，其设计十分灵活，能够很方便的扩展成自定义的优化方法。\n",
    "\n",
    "所有的优化方法都是继承基类`optim.Optimizer`，并实现了自己的优化步骤。下面就以最基本的优化方法——随机梯度下降法（SGD）举例说明。这里需重点掌握：\n",
    "\n",
    "- 优化方法的基本使用方法\n",
    "- 如何对模型的不同部分设置不同的学习率\n",
    "- 如何调整学习率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.921328Z",
     "start_time": "2020-12-10T06:06:02.914348Z"
    }
   },
   "outputs": [],
   "source": [
    "# 首先定义一个LeNet网络\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.features = nn.Sequential(\n",
    "                    nn.Conv2d(3, 6, 5),\n",
    "                    nn.ReLU(),\n",
    "                    nn.MaxPool2d(2,2),\n",
    "                    nn.Conv2d(6, 16, 5),\n",
    "                    nn.ReLU(),\n",
    "                    nn.MaxPool2d(2,2)\n",
    "        )\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(16 * 5 * 5, 120),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(120, 84),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(84, 10)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "\n",
    "net = Net()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.932299Z",
     "start_time": "2020-12-10T06:06:02.922326Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch import  optim\n",
    "optimizer = optim.SGD(params=net.parameters(), lr=1)\n",
    "optimizer.zero_grad() # 梯度清零，等价于net.zero_grad()\n",
    "\n",
    "input = t.randn(1, 3, 32, 32)\n",
    "output = net(input)\n",
    "output.backward(output) # fake backward\n",
    "\n",
    "optimizer.step() # 执行优化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.938284Z",
     "start_time": "2020-12-10T06:06:02.933297Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 1e-05\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.01\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为不同子网络设置不同的学习率，在finetune中经常用到\n",
    "# 如果对某个参数不指定学习率，就使用最外层的默认学习率\n",
    "optimizer =optim.SGD([\n",
    "                {'params': net.features.parameters()}, # 学习率为1e-5\n",
    "                {'params': net.classifier.parameters(), 'lr': 1e-2}\n",
    "            ], lr=1e-5)\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.951249Z",
     "start_time": "2020-12-10T06:06:02.939282Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function id(obj, /)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.958230Z",
     "start_time": "2020-12-10T06:06:02.952246Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 0.001\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.01\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 只为两个全连接层设置较大的学习率，其余层的学习率较小\n",
    "special_layers = nn.ModuleList([net.classifier[0], net.classifier[3]])\n",
    "special_layers_params = list(map(id, special_layers.parameters()))\n",
    "base_params = filter(lambda p: id(p) not in special_layers_params,\n",
    "                     net.parameters())\n",
    "\n",
    "optimizer = t.optim.SGD([\n",
    "            {'params': base_params},\n",
    "            {'params': special_layers.parameters(), 'lr': 0.01}\n",
    "        ], lr=0.001 )\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于如何调整学习率，主要有两种做法。一种是修改optimizer.param_groups中对应的学习率，另一种是更简单也是较为推荐的做法——新建优化器，由于optimizer十分轻量级，构建开销很小，故而可以构建新的optimizer。但是后者对于使用动量的优化器（如Adam），会丢失动量等状态信息，可能会造成损失函数的收敛出现震荡等情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.964214Z",
     "start_time": "2020-12-10T06:06:02.959227Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 1e-05\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.010000000000000002\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 方法1: 调整学习率，新建一个optimizer\n",
    "old_lr = 0.1\n",
    "optimizer1 =optim.SGD([\n",
    "                {'params': net.features.parameters()},\n",
    "                {'params': net.classifier.parameters(), 'lr': old_lr*0.1}\n",
    "            ], lr=1e-5)\n",
    "optimizer1### 4.3 nn.functional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.970198Z",
     "start_time": "2020-12-10T06:06:02.965211Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 0.0001\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.001\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 方法2: 调整学习率, 手动decay, 保存动量\n",
    "for param_group in optimizer.param_groups:\n",
    "    param_group['lr'] *= 0.1 # 学习率为之前的0.1倍\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 nn.functional\n",
    "\n",
    "nn中还有一个很常用的模块：`nn.functional`，nn中的大多数layer，在`functional`中都有一个与之相对应的函数。`nn.functional`中的函数和`nn.Module`的主要区别在于，用nn.Module实现的layers是一个特殊的类，都是由`class layer(nn.Module)`定义，会自动提取可学习的参数。而`nn.functional`中的函数更像是纯函数，由`def function(input)`定义。下面举例说明functional的使用，并指出二者的不同之处。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.976182Z",
     "start_time": "2020-12-10T06:06:02.971196Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[True, True, True, True],\n",
       "        [True, True, True, True]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.randn(2, 3)\n",
    "model = nn.Linear(3, 4)\n",
    "output1 = model(input)\n",
    "output2 = nn.functional.linear(input, model.weight, model.bias)\n",
    "output1 == output2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.982166Z",
     "start_time": "2020-12-10T06:06:02.977179Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[True, True, True],\n",
       "        [True, True, True]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = nn.functional.relu(input)\n",
    "b2 = nn.ReLU()(input)\n",
    "b == b2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此时读者可能会问，应该什么时候使用nn.Module，什么时候使用nn.functional呢？答案很简单:\n",
    "- 如果模型有可学习的参数，最好用nn.Module;\n",
    "- 否则既可以使用nn.functional也可以使用nn.Module，二者在性能上没有太大差异，具体的使用取决于个人的喜好。如激活函数（ReLU、sigmoid、tanh），池化（MaxPool）等层由于没有可学习参数，则可以使用对应的functional函数代替，而对于卷积、全连接等具有可学习参数的网络建议使用nn.Module。\n",
    "- 另外虽然dropout操作也没有可学习操作，但建议还是使用`nn.Dropout`而不是`nn.functional.dropout`，因为dropout在训练和测试两个阶段的行为有所差别，使用`nn.Module`对象能够通过`model.eval`操作加以区分。\n",
    "\n",
    "下面举例说明，如何在模型中搭配使用nn.Module和nn.functional。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.989148Z",
     "start_time": "2020-12-10T06:06:02.983163Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.pool(F.relu(self.conv1(x)), 2)\n",
    "        x = F.pool(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于不具备可学习参数的层（激活层、池化层等），将它们用函数代替，这样则可以不用放置在构造函数`__init__`中。对于有可学习参数的模块，也可以用functional来代替，只不过实现起来较为繁琐，需要手动定义参数parameter，如前面实现自定义的全连接层，就可将weight和bias两个参数单独拿出来，在构造函数中初始化为parameter。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:02.995132Z",
     "start_time": "2020-12-10T06:06:02.990145Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyLinear(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyLinear, self).__init__()\n",
    "        self.weight = nn.Parameter(t.randn(3, 4))\n",
    "        self.bias = nn.Parameter(t.zeros(3))\n",
    "    def forward(self):\n",
    "        return F.linear(input, weight, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于nn.functional的设计初衷，以及它和nn.Module更多的比较说明，可参看论坛的讨论和作者说明[^6]。\n",
    "[^6]: https://discuss.pytorch.org/search?q=nn.functional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 初始化策略\n",
    "在深度学习中参数的初始化十分重要，良好的初始化能让模型更快收敛，并达到更高水平，而糟糕的初始化则可能使得模型迅速瘫痪。PyTorch中nn.Module的模块参数都采取了较为合理的初始化策略，因此一般不用我们考虑，当然我们也可以用自定义初始化去代替系统的默认初始化。而当我们在使用Parameter时，自定义初始化则尤为重要，因t.Tensor()返回的是内存中的随机数，很可能会有极大值，这在实际训练网络中会造成溢出或者梯度消失。PyTorch中`nn.init`模块就是专门为初始化而设计，如果某种初始化策略`nn.init`不提供，用户也可以自己直接初始化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.003110Z",
     "start_time": "2020-12-10T06:06:02.996129Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.3535,  0.1427,  0.0330],\n",
       "        [ 0.3321, -0.2416, -0.0888],\n",
       "        [-0.8140,  0.2040, -0.5493],\n",
       "        [-0.3010, -0.4769, -0.0311]], requires_grad=True)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 利用nn.init初始化\n",
    "from torch.nn import init\n",
    "linear = nn.Linear(3, 4)\n",
    "\n",
    "t.manual_seed(1)\n",
    "# 等价于 linear.weight.data.normal_(0, std)\n",
    "init.xavier_normal_(linear.weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.010091Z",
     "start_time": "2020-12-10T06:06:03.004107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.3535,  0.1427,  0.0330],\n",
       "        [ 0.3321, -0.2416, -0.0888],\n",
       "        [-0.8140,  0.2040, -0.5493],\n",
       "        [-0.3010, -0.4769, -0.0311]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 直接初始化\n",
    "import math\n",
    "t.manual_seed(1)\n",
    "\n",
    "# xavier初始化的计算公式\n",
    "std = math.sqrt(2)/math.sqrt(7.)\n",
    "linear.weight.data.normal_(0,std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.015079Z",
     "start_time": "2020-12-10T06:06:03.011089Z"
    }
   },
   "outputs": [],
   "source": [
    "# 对模型的所有参数进行初始化\n",
    "for name, params in net.named_parameters():\n",
    "    if name.find('linear') != -1:\n",
    "        # init linear\n",
    "        params[0] # weight\n",
    "        params[1] # bias\n",
    "    elif name.find('conv') != -1:\n",
    "        pass\n",
    "    elif name.find('norm') != -1:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 nn.Module深入分析\n",
    "\n",
    "如果想要更深入地理解nn.Module，究其原理是很有必要的。首先来看看nn.Module基类的构造函数：\n",
    "```python\n",
    "def __init__(self):\n",
    "    self._parameters = OrderedDict()\n",
    "    self._modules = OrderedDict()\n",
    "    self._buffers = OrderedDict()\n",
    "    self._backward_hooks = OrderedDict()\n",
    "    self._forward_hooks = OrderedDict()\n",
    "    self.training = True\n",
    "```\n",
    "其中每个属性的解释如下：\n",
    "\n",
    "- `_parameters`：字典，保存用户直接设置的parameter，`self.param1 = nn.Parameter(t.randn(3, 3))`会被检测到，在字典中加入一个key为'param'，value为对应parameter的item。而self.submodule = nn.Linear(3, 4)中的parameter则不会存于此。\n",
    "- `_modules`：子module，通过`self.submodel = nn.Linear(3, 4)`指定的子module会保存于此。\n",
    "- `_buffers`：缓存。如batchnorm使用momentum机制，每次前向传播需用到上一次前向传播的结果。\n",
    "- `_backward_hooks`与`_forward_hooks`：钩子技术，用来提取中间变量，类似variable的hook。\n",
    "- `training`：BatchNorm与Dropout层在训练阶段和测试阶段中采取的策略不同，通过判断training值来决定前向传播策略。\n",
    "\n",
    "上述几个属性中，`_parameters`、`_modules`和`_buffers`这三个字典中的键值，都可以通过`self.key`方式获得，效果等价于`self._parameters['key']`.\n",
    "\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.022060Z",
     "start_time": "2020-12-10T06:06:03.016076Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # 等价与self.register_parameter('param1' ,nn.Parameter(t.randn(3, 3)))\n",
    "        self.param1 = nn.Parameter(t.rand(3, 3))\n",
    "        self.submodel1 = nn.Linear(3, 4) \n",
    "    def forward(self, input):\n",
    "        x = self.param1.mm(input)\n",
    "        x = self.submodel1(x)\n",
    "        return x\n",
    "net = Net()\n",
    "net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.031036Z",
     "start_time": "2020-12-10T06:06:03.023057Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('submodel1', Linear(in_features=3, out_features=4, bias=True))])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net._modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.039014Z",
     "start_time": "2020-12-10T06:06:03.032033Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('param1',\n",
       "              Parameter containing:\n",
       "              tensor([[0.3398, 0.5239, 0.7981],\n",
       "                      [0.7718, 0.0112, 0.8100],\n",
       "                      [0.6397, 0.9743, 0.8300]], requires_grad=True))])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net._parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.044998Z",
     "start_time": "2020-12-10T06:06:03.040012Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[0.3398, 0.5239, 0.7981],\n",
       "        [0.7718, 0.0112, 0.8100],\n",
       "        [0.6397, 0.9743, 0.8300]], requires_grad=True)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.param1 # 等价于net._parameters['param1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.049985Z",
     "start_time": "2020-12-10T06:06:03.045996Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "param1 torch.Size([3, 3])\n",
      "submodel1.weight torch.Size([4, 3])\n",
      "submodel1.bias torch.Size([4])\n"
     ]
    }
   ],
   "source": [
    "for name, param in net.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.055969Z",
     "start_time": "2020-12-10T06:06:03.050982Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Net(\n",
      "  (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
      ")\n",
      "submodel1 Linear(in_features=3, out_features=4, bias=True)\n"
     ]
    }
   ],
   "source": [
    "for name, submodel in net.named_modules():\n",
    "    print(name, submodel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.065942Z",
     "start_time": "2020-12-10T06:06:03.056966Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('running_mean', tensor([0.0362, 0.0596])),\n",
       "             ('running_var', tensor([0.9009, 0.9262])),\n",
       "             ('num_batches_tracked', tensor(1))])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn = nn.BatchNorm1d(2)\n",
    "input = t.rand(3, 2)\n",
    "output = bn(input)\n",
    "bn._buffers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Module在实际使用中可能层层嵌套，一个module包含若干个子module，每一个子module又包含了更多的子module。为方便用户访问各个子module，nn.Module实现了很多方法，如函数`children`可以查看直接子module，函数`module`可以查看所有的子module（包括当前module）。与之相对应的还有函数`named_childen`和`named_modules`，其能够在返回module列表的同时返回它们的名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.069932Z",
     "start_time": "2020-12-10T06:06:03.066940Z"
    }
   },
   "outputs": [],
   "source": [
    "# input = t.arange(0, 12).view(3, 4)\n",
    "# model = nn.Dropout()\n",
    "# # 在训练阶段，会有一半左右的数被随机置为0\n",
    "# model(input)\n",
    "\n",
    "\n",
    "# model.training  = False\n",
    "# # 在测试阶段，dropout什么都不做\n",
    "# model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.074918Z",
     "start_time": "2020-12-10T06:06:03.070929Z"
    }
   },
   "outputs": [],
   "source": [
    "input1 = t.arange(0, 12).view(3, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.079905Z",
     "start_time": "2020-12-10T06:06:03.075916Z"
    }
   },
   "outputs": [],
   "source": [
    "model = nn.Dropout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:06:03.085889Z",
     "start_time": "2020-12-10T06:06:03.080902Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dropout(p=0.5, inplace=False)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于batchnorm、dropout、instancenorm等在训练和测试阶段行为差距巨大的层，如果在测试时不将其training值设为True，则可能会有很大影响，这在实际使用中要千万注意。虽然可通过直接设置`training`属性，来将子module设为train和eval模式，但这种方式较为繁琐，因如果一个模型具有多个dropout层，就需要为每个dropout层指定training属性。更为推荐的做法是调用`model.train()`函数，它会将当前module及其子module中的所有training属性都设为True，相应的，`model.eval()`函数会把training属性都设为False。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:07:43.217532Z",
     "start_time": "2020-12-10T06:07:43.212546Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(False, False)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(net.training, net.submodel1.training)\n",
    "net.eval()\n",
    "net.training, net.submodel1.training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:07:50.529548Z",
     "start_time": "2020-12-10T06:07:50.524562Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('',\n",
       "  Net(\n",
       "    (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
       "  )),\n",
       " ('submodel1', Linear(in_features=3, out_features=4, bias=True))]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(net.named_modules())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`register_forward_hook`与`register_backward_hook`，这两个函数的功能类似于variable函数的`register_hook`，可在module前向传播或反向传播时注册钩子。每次前向传播执行结束后会执行钩子函数（hook）。前向传播的钩子函数具有如下形式：`hook(module, input, output) -> None`，而反向传播则具有如下形式：`hook(module, grad_input, grad_output) -> Tensor or None`。钩子函数不应修改输入和输出，并且在使用后应及时删除，以避免每次都运行钩子增加运行负载。钩子函数主要用在获取某些中间结果的情景，如中间某一层的输出或某一层的梯度。这些结果本应写在forward函数中，但如果在forward函数中专门加上这些处理，可能会使处理逻辑比较复杂，这时候使用钩子技术就更合适一些。下面考虑一种场景，有一个预训练好的模型，需要提取模型的某一层（不是最后一层）的输出作为特征进行分类，但又不希望修改其原有的模型定义文件，这时就可以利用钩子函数。下面给出实现的伪代码。\n",
    "```python\n",
    "model = VGG()\n",
    "features = t.Tensor()\n",
    "def hook(module, input, output):\n",
    "    '''把这层的输出拷贝到features中'''\n",
    "    features.copy_(output.data)\n",
    "    \n",
    "handle = model.layer8.register_forward_hook(hook)\n",
    "_ = model(input)\n",
    "# 用完hook后删除\n",
    "handle.remove()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nn.Module`对象在构造函数中的行为看起来有些怪异，如果想要真正掌握其原理，就需要看两个魔法方法`__getattr__`和`__setattr__`。在Python中有两个常用的buildin方法`getattr`和`setattr`，`getattr(obj, 'attr1')`等价于`obj.attr`，如果`getattr`函数无法找到所需属性，Python会转而调用`obj.__getattr__('attr1')`方法，即`getattr`函数无法找到的交给`__getattr__`函数处理，没有实现`__getattr__`或者`__getattr__`也无法处理的就会raise AttributeError。`setattr(obj, 'name', value)`等价于`obj.name=value`，如果obj对象实现了`__setattr__`方法，setattr会直接调用`obj.__setattr__('name', value)`，否则调用buildin方法。总结一下：\n",
    "- result  = obj.name会调用buildin函数`getattr(obj, 'name')`，如果该属性找不到，会调用`obj.__getattr__('name')`\n",
    "- obj.name = value会调用buildin函数`setattr(obj, 'name', value)`，如果obj对象实现了`__setattr__`方法，`setattr`会直接调用`obj.__setattr__('name', value')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Module实现了自定义的`__setattr__`函数，当执行`module.name=value`时，会在`__setattr__`中判断value是否为`Parameter`或`nn.Module`对象，如果是则将这些对象加到`_parameters`和`_modules`两个字典中，而如果是其它类型的对象，如`Variable`、`list`、`dict`等，则调用默认的操作，将这个值保存在`__dict__`中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:13:03.413024Z",
     "start_time": "2020-12-10T06:13:03.407042Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('param',\n",
       "              Parameter containing:\n",
       "              tensor([[1., 1.],\n",
       "                      [1., 1.]], requires_grad=True))])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module = nn.Module()\n",
    "module.param = nn.Parameter(t.ones(2, 2))\n",
    "module._parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:13:23.681616Z",
     "start_time": "2020-12-10T06:13:23.676656Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_modules:  OrderedDict()\n",
      "__dict__['submodules']: [Linear(in_features=2, out_features=2, bias=True), Linear(in_features=2, out_features=2, bias=True)]\n"
     ]
    }
   ],
   "source": [
    "submodule1 = nn.Linear(2, 2)\n",
    "submodule2 = nn.Linear(2, 2)\n",
    "module_list =  [submodule1, submodule2]\n",
    "# 对于list对象，调用buildin函数，保存在__dict__中\n",
    "module.submodules = module_list\n",
    "print('_modules: ', module._modules)\n",
    "print(\"__dict__['submodules']:\",module.__dict__.get('submodules'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:14:23.396630Z",
     "start_time": "2020-12-10T06:14:23.391673Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ModuleList is instance of nn.Module:  True\n",
      "_modules:  OrderedDict([('submodules', ModuleList(\n",
      "  (0): Linear(in_features=2, out_features=2, bias=True)\n",
      "  (1): Linear(in_features=2, out_features=2, bias=True)\n",
      "))])\n",
      "__dict__['submodules']: None\n"
     ]
    }
   ],
   "source": [
    "module_list = nn.ModuleList(module_list)\n",
    "module.submodules = module_list\n",
    "print('ModuleList is instance of nn.Module: ', isinstance(module_list, nn.Module))\n",
    "print('_modules: ', module._modules)\n",
    "print(\"__dict__['submodules']:\", module.__dict__.get('submodules'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因`_modules`和`_parameters`中的item未保存在`__dict__`中，所以默认的getattr方法无法获取它，因而`nn.Module`实现了自定义的`__getattr__`方法，如果默认的`getattr`无法处理，就调用自定义的`__getattr__`方法，尝试从`_modules`、`_parameters`和`_buffers`这三个字典中获取。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:15:44.739150Z",
     "start_time": "2020-12-10T06:15:44.735132Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getattr(module, 'training') # 等价于module.training\n",
    "# error\n",
    "# module.__getattr__('training')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:15:53.104266Z",
     "start_time": "2020-12-10T06:15:53.100274Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.attr1 = 2\n",
    "getattr(module, 'attr1')\n",
    "# 报错\n",
    "# module.__getattr__('attr1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:16:04.595014Z",
     "start_time": "2020-12-10T06:16:04.590052Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[1., 1.],\n",
       "        [1., 1.]], requires_grad=True)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 即module.param, 会调用module.__getattr__('param')\n",
    "getattr(module, 'param')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在PyTorch中保存模型十分简单，所有的Module对象都具有state_dict()函数，返回当前Module所有的状态数据。将这些状态数据保存后，下次使用模型时即可利用`model.load_state_dict()`函数将状态加载进来。优化器（optimizer）也有类似的机制，不过一般并不需要保存优化器的运行状态。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:16:54.087103Z",
     "start_time": "2020-12-10T06:16:54.078128Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 保存模型\n",
    "t.save(net.state_dict(), 'net.pth')\n",
    "\n",
    "# 加载已保存的模型\n",
    "net2 = Net()\n",
    "net2.load_state_dict(t.load('net.pth'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实际上还有另外一种保存方法，但因其严重依赖模型定义方式及文件路径结构等，很容易出问题，因而不建议使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:17:17.255208Z",
     "start_time": "2020-12-10T06:17:17.245236Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\DELL\\.conda\\envs\\pytorch\\lib\\site-packages\\torch\\serialization.py:256: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.save(net, 'net_all.pth')\n",
    "net2 = t.load('net_all.pth')\n",
    "net2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将Module放在GPU上运行也十分简单，只需两步：\n",
    "- model = model.cuda()：将模型的所有参数转存到GPU\n",
    "- input.cuda()：将输入数据也放置到GPU上\n",
    "\n",
    "至于如何在多个GPU上并行计算，PyTorch也提供了两个函数，可实现简单高效的并行GPU计算\n",
    "- nn.parallel.data_parallel(module, inputs, device_ids=None, output_device=None, dim=0, module_kwargs=None)\n",
    "- class torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)\n",
    "\n",
    "可见二者的参数十分相似，通过`device_ids`参数可以指定在哪些GPU上进行优化，output_device指定输出到哪个GPU上。唯一的不同就在于前者直接利用多GPU并行计算得出结果，而后者则返回一个新的module，能够自动在多GPU上进行并行加速。\n",
    "\n",
    "```\n",
    "# method 1\n",
    "new_net = nn.DataParallel(net, device_ids=[0, 1])\n",
    "output = new_net(input)\n",
    "\n",
    "# method 2\n",
    "output = nn.parallel.data_parallel(new_net, input, device_ids=[0, 1])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataParallel并行的方式，是将输入一个batch的数据均分成多份，分别送到对应的GPU进行计算，各个GPU得到的梯度累加。与Module相关的所有数据也都会以浅复制的方式复制多份，在此需要注意，在module中属性应该是只读的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6 nn和autograd的关系\n",
    "nn.Module利用的也是autograd技术，其主要工作是实现前向传播。在forward函数中，nn.Module对输入的tensor进行的各种操作，本质上都是用到了autograd技术。这里需要对比autograd.Function和nn.Module之间的区别：\n",
    "- autograd.Function利用了Tensor对autograd技术的扩展，为autograd实现了新的运算op，不仅要实现前向传播还要手动实现反向传播\n",
    "- nn.Module利用了autograd技术，对nn的功能进行扩展，实现了深度学习中更多的层。只需实现前向传播功能，autograd即会自动实现反向传播\n",
    "- nn.functional是一些autograd操作的集合，是经过封装的函数\n",
    "\n",
    "作为两大类扩充PyTorch接口的方法，我们在实际使用中应该如何选择呢？如果某一个操作，在autograd中尚未支持，那么只能实现Function接口对应的前向传播和反向传播。如果某些时候利用autograd接口比较复杂，则可以利用Function将多个操作聚合，实现优化，正如第三章所实现的`Sigmoid`一样，比直接利用autograd低级别的操作要快。而如果只是想在深度学习中增加某一层，使用nn.Module进行封装则更为简单高效。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.7 小试牛刀：搭建ResNet\n",
    "Kaiming He的深度残差网络（ResNet）[^7]在深度学习的发展中起到了很重要的作用，ResNet不仅一举拿下了当年CV下多个比赛项目的冠军，更重要的是这一结构解决了训练极深网络时的梯度消失问题。\n",
    "\n",
    "首先来看看ResNet的网络结构，这里选取的是ResNet的一个变种：ResNet34。ResNet的网络结构如图4-2所示，可见除了最开始的卷积池化和最后的池化全连接之外，网络中有很多结构相似的单元，这些重复单元的共同点就是有个跨层直连的shortcut。ResNet中将一个跨层直连的单元称为Residual block，其结构如图4-3所示，左边部分是普通的卷积网络结构，右边是直连，但如果输入和输出的通道数不一致，或其步长不为1，那么就需要有一个专门的单元将二者转成一致，使其可以相加。\n",
    "\n",
    "另外我们可以发现Residual block的大小也是有规律的，在最开始的pool之后有连续的几个一模一样的Residual block单元，这些单元的通道数一样，在这里我们将这几个拥有多个Residual block单元的结构称之为layer，注意和之前讲的layer区分开来，这里的layer是几个层的集合。\n",
    "\n",
    "考虑到Residual block和layer出现了多次，我们可以把它们实现为一个子Module或函数。这里我们将Residual block实现为一个子moduke，而将layer实现为一个函数。下面是实现代码，规律总结如下：\n",
    "\n",
    "- 对于模型中的重复部分，实现为子module或用函数生成相应的module`make_layer`\n",
    "- nn.Module和nn.Functional结合使用\n",
    "- 尽量使用`nn.Seqential`\n",
    "\n",
    "![图4-2: ResNet34网络结构](imgs/resnet1.png)\n",
    "![图4-3: Residual block 结构图](imgs/residual.png)\n",
    " [^7]: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:25:42.005221Z",
     "start_time": "2020-12-10T06:25:42.002227Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch import  nn\n",
    "import torch as t\n",
    "from torch.nn import  functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:29:15.867567Z",
     "start_time": "2020-12-10T06:29:15.855570Z"
    }
   },
   "outputs": [],
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "    '''\n",
    "    实现子module: Residual Block\n",
    "    '''\n",
    "    def __init__(self, inchannel, outchannel, stride=1, shortcut=None):\n",
    "        super(ResidualBlock, self).__init__()\n",
    "        self.left = nn.Sequential(\n",
    "                nn.Conv2d(inchannel,outchannel,3,stride, 1,bias=False),\n",
    "                nn.BatchNorm2d(outchannel),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.Conv2d(outchannel,outchannel,3,1,1,bias=False),\n",
    "                nn.BatchNorm2d(outchannel) )\n",
    "        self.right = shortcut\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.left(x)\n",
    "        residual = x if self.right is None else self.right(x)\n",
    "        out += residual\n",
    "        return F.relu(out)\n",
    "\n",
    "class ResNet(nn.Module):\n",
    "    '''\n",
    "    实现主module：ResNet34\n",
    "    ResNet34 包含多个layer，每个layer又包含多个residual block\n",
    "    用子module来实现residual block，用_make_layer函数来实现layer\n",
    "    '''\n",
    "    def __init__(self, num_classes=1000):\n",
    "        super(ResNet, self).__init__()\n",
    "        # 前几层图像转换\n",
    "        self.pre = nn.Sequential(\n",
    "                nn.Conv2d(3, 64, 7, 2, 3, bias=False),\n",
    "                nn.BatchNorm2d(64),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(3, 2, 1))\n",
    "        \n",
    "        # 重复的layer，分别有3，4，6，3个residual block\n",
    "        self.layer1 = self._make_layer( 64, 64, 3)\n",
    "        self.layer2 = self._make_layer( 64, 128, 4, stride=2)\n",
    "        self.layer3 = self._make_layer( 128, 256, 6, stride=2)\n",
    "        self.layer4 = self._make_layer( 256, 512, 3, stride=2)\n",
    "\n",
    "        #分类用的全连接\n",
    "        self.fc = nn.Linear(512, num_classes)\n",
    "    \n",
    "    def _make_layer(self,  inchannel, outchannel, block_num, stride=1):\n",
    "        '''\n",
    "        构建layer,包含多个residual block\n",
    "        '''\n",
    "        shortcut = nn.Sequential(\n",
    "                nn.Conv2d(inchannel,outchannel,1,stride, bias=False),\n",
    "                nn.BatchNorm2d(outchannel))\n",
    "        \n",
    "        layers = []\n",
    "        layers.append(ResidualBlock(inchannel, outchannel, stride, shortcut))\n",
    "        \n",
    "        for i in range(1, block_num):\n",
    "            layers.append(ResidualBlock(outchannel, outchannel))\n",
    "        return nn.Sequential(*layers)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.pre(x)\n",
    "        \n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.layer3(x)\n",
    "        x = self.layer4(x)\n",
    "\n",
    "        x = F.avg_pool2d(x, 7)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:29:30.398629Z",
     "start_time": "2020-12-10T06:29:30.064523Z"
    }
   },
   "outputs": [],
   "source": [
    "model = ResNet()\n",
    "input  = t.randn(1, 3, 224, 224)\n",
    "o = model(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "感兴趣的读者可以尝试实现Google的Inception网络结构或ResNet的其它变体，看看如何能够简洁明了地实现它，实现代码尽量控制在80行以内（本例去掉空行和注释总共不超过50行）。另外，与PyTorch配套的图像工具包`torchvision`已经实现了深度学习中大多数经典的模型，其中就包括ResNet34，读者可以通过下面两行代码使用：\n",
    "```python\n",
    "from torchvision import models\n",
    "model = models.resnet34()\n",
    "```\n",
    "本例中ResNet34的实现就是参考了torchvision中的实现并做了简化，感兴趣的读者可以阅读相应的源码，比较这里的实现和torchvision中实现的不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:30:13.197391Z",
     "start_time": "2020-12-10T06:30:12.944102Z"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision import models\n",
    "model = models.resnet34()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T06:30:19.572427Z",
     "start_time": "2020-12-10T06:30:19.569434Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ResNet(\n",
      "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
      "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (relu): ReLU(inplace=True)\n",
      "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
      "  (layer1): Sequential(\n",
      "    (0): BasicBlock(\n",
      "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (1): BasicBlock(\n",
      "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (2): BasicBlock(\n",
      "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (layer2): Sequential(\n",
      "    (0): BasicBlock(\n",
      "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (downsample): Sequential(\n",
      "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (1): BasicBlock(\n",
      "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (2): BasicBlock(\n",
      "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (3): BasicBlock(\n",
      "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (layer3): Sequential(\n",
      "    (0): BasicBlock(\n",
      "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (downsample): Sequential(\n",
      "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (1): BasicBlock(\n",
      "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (2): BasicBlock(\n",
      "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (3): BasicBlock(\n",
      "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (4): BasicBlock(\n",
      "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (5): BasicBlock(\n",
      "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (layer4): Sequential(\n",
      "    (0): BasicBlock(\n",
      "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (downsample): Sequential(\n",
      "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (1): BasicBlock(\n",
      "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "    (2): BasicBlock(\n",
      "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
      "  (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-10T07:00:13.405403Z",
     "start_time": "2020-12-10T07:00:13.395426Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Conv2d in module torch.nn.modules.conv:\n",
      "\n",
      "class Conv2d(_ConvNd)\n",
      " |  Applies a 2D convolution over an input signal composed of several input\n",
      " |  planes.\n",
      " |  \n",
      " |  In the simplest case, the output value of the layer with input size\n",
      " |  :math:`(N, C_{\\text{in}}, H, W)` and output :math:`(N, C_{\\text{out}}, H_{\\text{out}}, W_{\\text{out}})`\n",
      " |  can be precisely described as:\n",
      " |  \n",
      " |  .. math::\n",
      " |      \\text{out}(N_i, C_{\\text{out}_j}) = \\text{bias}(C_{\\text{out}_j}) +\n",
      " |      \\sum_{k = 0}^{C_{\\text{in}} - 1} \\text{weight}(C_{\\text{out}_j}, k) \\star \\text{input}(N_i, k)\n",
      " |  \n",
      " |  \n",
      " |  where :math:`\\star` is the valid 2D `cross-correlation`_ operator,\n",
      " |  :math:`N` is a batch size, :math:`C` denotes a number of channels,\n",
      " |  :math:`H` is a height of input planes in pixels, and :math:`W` is\n",
      " |  width in pixels.\n",
      " |  \n",
      " |  * :attr:`stride` controls the stride for the cross-correlation, a single\n",
      " |    number or a tuple.\n",
      " |  \n",
      " |  * :attr:`padding` controls the amount of implicit zero-paddings on both\n",
      " |    sides for :attr:`padding` number of points for each dimension.\n",
      " |  \n",
      " |  * :attr:`dilation` controls the spacing between the kernel points; also\n",
      " |    known as the à trous algorithm. It is harder to describe, but this `link`_\n",
      " |    has a nice visualization of what :attr:`dilation` does.\n",
      " |  \n",
      " |  * :attr:`groups` controls the connections between inputs and outputs.\n",
      " |    :attr:`in_channels` and :attr:`out_channels` must both be divisible by\n",
      " |    :attr:`groups`. For example,\n",
      " |  \n",
      " |      * At groups=1, all inputs are convolved to all outputs.\n",
      " |      * At groups=2, the operation becomes equivalent to having two conv\n",
      " |        layers side by side, each seeing half the input channels,\n",
      " |        and producing half the output channels, and both subsequently\n",
      " |        concatenated.\n",
      " |      * At groups= :attr:`in_channels`, each input channel is convolved with\n",
      " |        its own set of filters, of size:\n",
      " |        :math:`\\left\\lfloor\\frac{out\\_channels}{in\\_channels}\\right\\rfloor`.\n",
      " |  \n",
      " |  The parameters :attr:`kernel_size`, :attr:`stride`, :attr:`padding`, :attr:`dilation` can either be:\n",
      " |  \n",
      " |      - a single ``int`` -- in which case the same value is used for the height and width dimension\n",
      " |      - a ``tuple`` of two ints -- in which case, the first `int` is used for the height dimension,\n",
      " |        and the second `int` for the width dimension\n",
      " |  \n",
      " |  .. note::\n",
      " |  \n",
      " |       Depending of the size of your kernel, several (of the last)\n",
      " |       columns of the input might be lost, because it is a valid `cross-correlation`_,\n",
      " |       and not a full `cross-correlation`_.\n",
      " |       It is up to the user to add proper padding.\n",
      " |  \n",
      " |  .. note::\n",
      " |  \n",
      " |      When `groups == in_channels` and `out_channels == K * in_channels`,\n",
      " |      where `K` is a positive integer, this operation is also termed in\n",
      " |      literature as depthwise convolution.\n",
      " |  \n",
      " |      In other words, for an input of size :math:`(N, C_{in}, H_{in}, W_{in})`,\n",
      " |      a depthwise convolution with a depthwise multiplier `K`, can be constructed by arguments\n",
      " |      :math:`(in\\_channels=C_{in}, out\\_channels=C_{in} \\times K, ..., groups=C_{in})`.\n",
      " |  \n",
      " |  .. include:: cudnn_deterministic.rst\n",
      " |  \n",
      " |  Args:\n",
      " |      in_channels (int): Number of channels in the input image\n",
      " |      out_channels (int): Number of channels produced by the convolution\n",
      " |      kernel_size (int or tuple): Size of the convolving kernel\n",
      " |      stride (int or tuple, optional): Stride of the convolution. Default: 1\n",
      " |      padding (int or tuple, optional): Zero-padding added to both sides of the input. Default: 0\n",
      " |      padding_mode (string, optional). Accepted values `zeros` and `circular` Default: `zeros`\n",
      " |      dilation (int or tuple, optional): Spacing between kernel elements. Default: 1\n",
      " |      groups (int, optional): Number of blocked connections from input channels to output channels. Default: 1\n",
      " |      bias (bool, optional): If ``True``, adds a learnable bias to the output. Default: ``True``\n",
      " |  \n",
      " |  Shape:\n",
      " |      - Input: :math:`(N, C_{in}, H_{in}, W_{in})`\n",
      " |      - Output: :math:`(N, C_{out}, H_{out}, W_{out})` where\n",
      " |  \n",
      " |        .. math::\n",
      " |            H_{out} = \\left\\lfloor\\frac{H_{in}  + 2 \\times \\text{padding}[0] - \\text{dilation}[0]\n",
      " |                      \\times (\\text{kernel\\_size}[0] - 1) - 1}{\\text{stride}[0]} + 1\\right\\rfloor\n",
      " |  \n",
      " |        .. math::\n",
      " |            W_{out} = \\left\\lfloor\\frac{W_{in}  + 2 \\times \\text{padding}[1] - \\text{dilation}[1]\n",
      " |                      \\times (\\text{kernel\\_size}[1] - 1) - 1}{\\text{stride}[1]} + 1\\right\\rfloor\n",
      " |  \n",
      " |  Attributes:\n",
      " |      weight (Tensor): the learnable weights of the module of shape\n",
      " |                       :math:`(\\text{out\\_channels}, \\frac{\\text{in\\_channels}}{\\text{groups}},`\n",
      " |                       :math:`\\text{kernel\\_size[0]}, \\text{kernel\\_size[1]})`.\n",
      " |                       The values of these weights are sampled from\n",
      " |                       :math:`\\mathcal{U}(-\\sqrt{k}, \\sqrt{k})` where\n",
      " |                       :math:`k = \\frac{1}{C_\\text{in} * \\prod_{i=0}^{1}\\text{kernel\\_size}[i]}`\n",
      " |      bias (Tensor):   the learnable bias of the module of shape (out_channels). If :attr:`bias` is ``True``,\n",
      " |                       then the values of these weights are\n",
      " |                       sampled from :math:`\\mathcal{U}(-\\sqrt{k}, \\sqrt{k})` where\n",
      " |                       :math:`k = \\frac{1}{C_\\text{in} * \\prod_{i=0}^{1}\\text{kernel\\_size}[i]}`\n",
      " |  \n",
      " |  Examples::\n",
      " |  \n",
      " |      >>> # With square kernels and equal stride\n",
      " |      >>> m = nn.Conv2d(16, 33, 3, stride=2)\n",
      " |      >>> # non-square kernels and unequal stride and with padding\n",
      " |      >>> m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))\n",
      " |      >>> # non-square kernels and unequal stride and with padding and dilation\n",
      " |      >>> m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))\n",
      " |      >>> input = torch.randn(20, 16, 50, 100)\n",
      " |      >>> output = m(input)\n",
      " |  \n",
      " |  .. _cross-correlation:\n",
      " |      https://en.wikipedia.org/wiki/Cross-correlation\n",
      " |  \n",
      " |  .. _link:\n",
      " |      https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      Conv2d\n",
      " |      _ConvNd\n",
      " |      torch.nn.modules.module.Module\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  conv2d_forward(self, input, weight)\n",
      " |  \n",
      " |  forward(self, input)\n",
      " |      Defines the computation performed at every call.\n",
      " |      \n",
      " |      Should be overridden by all subclasses.\n",
      " |      \n",
      " |      .. note::\n",
      " |          Although the recipe for forward pass needs to be defined within\n",
      " |          this function, one should call the :class:`Module` instance afterwards\n",
      " |          instead of this since the former takes care of running the\n",
      " |          registered hooks while the latter silently ignores them.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from _ConvNd:\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  extra_repr(self)\n",
      " |      Set the extra representation of the module\n",
      " |      \n",
      " |      To print customized extra information, you should reimplement\n",
      " |      this method in your own modules. Both single-line and multi-line\n",
      " |      strings are acceptable.\n",
      " |  \n",
      " |  reset_parameters(self)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from _ConvNd:\n",
      " |  \n",
      " |  __constants__ = ['stride', 'padding', 'dilation', 'groups', 'bias', 'p...\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from torch.nn.modules.module.Module:\n",
      " |  \n",
      " |  __call__(self, *input, **kwargs)\n",
      " |      Call self as a function.\n",
      " |  \n",
      " |  __delattr__(self, name)\n",
      " |      Implement delattr(self, name).\n",
      " |  \n",
      " |  __dir__(self)\n",
      " |      __dir__() -> list\n",
      " |      default dir() implementation\n",
      " |  \n",
      " |  __getattr__(self, name)\n",
      " |  \n",
      " |  __repr__(self)\n",
      " |      Return repr(self).\n",
      " |  \n",
      " |  __setattr__(self, name, value)\n",
      " |      Implement setattr(self, name, value).\n",
      " |  \n",
      " |  add_module(self, name, module)\n",
      " |      Adds a child module to the current module.\n",
      " |      \n",
      " |      The module can be accessed as an attribute using the given name.\n",
      " |      \n",
      " |      Args:\n",
      " |          name (string): name of the child module. The child module can be\n",
      " |              accessed from this module using the given name\n",
      " |          module (Module): child module to be added to the module.\n",
      " |  \n",
      " |  apply(self, fn)\n",
      " |      Applies ``fn`` recursively to every submodule (as returned by ``.children()``)\n",
      " |      as well as self. Typical use includes initializing the parameters of a model\n",
      " |      (see also :ref:`torch-nn-init`).\n",
      " |      \n",
      " |      Args:\n",
      " |          fn (:class:`Module` -> None): function to be applied to each submodule\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> def init_weights(m):\n",
      " |          >>>     print(m)\n",
      " |          >>>     if type(m) == nn.Linear:\n",
      " |          >>>         m.weight.data.fill_(1.0)\n",
      " |          >>>         print(m.weight)\n",
      " |          >>> net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))\n",
      " |          >>> net.apply(init_weights)\n",
      " |          Linear(in_features=2, out_features=2, bias=True)\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 1.,  1.],\n",
      " |                  [ 1.,  1.]])\n",
      " |          Linear(in_features=2, out_features=2, bias=True)\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 1.,  1.],\n",
      " |                  [ 1.,  1.]])\n",
      " |          Sequential(\n",
      " |            (0): Linear(in_features=2, out_features=2, bias=True)\n",
      " |            (1): Linear(in_features=2, out_features=2, bias=True)\n",
      " |          )\n",
      " |          Sequential(\n",
      " |            (0): Linear(in_features=2, out_features=2, bias=True)\n",
      " |            (1): Linear(in_features=2, out_features=2, bias=True)\n",
      " |          )\n",
      " |  \n",
      " |  buffers(self, recurse=True)\n",
      " |      Returns an iterator over module buffers.\n",
      " |      \n",
      " |      Args:\n",
      " |          recurse (bool): if True, then yields buffers of this module\n",
      " |              and all submodules. Otherwise, yields only buffers that\n",
      " |              are direct members of this module.\n",
      " |      \n",
      " |      Yields:\n",
      " |          torch.Tensor: module buffer\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> for buf in model.buffers():\n",
      " |          >>>     print(type(buf.data), buf.size())\n",
      " |          <class 'torch.FloatTensor'> (20L,)\n",
      " |          <class 'torch.FloatTensor'> (20L, 1L, 5L, 5L)\n",
      " |  \n",
      " |  children(self)\n",
      " |      Returns an iterator over immediate children modules.\n",
      " |      \n",
      " |      Yields:\n",
      " |          Module: a child module\n",
      " |  \n",
      " |  cpu(self)\n",
      " |      Moves all model parameters and buffers to the CPU.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  cuda(self, device=None)\n",
      " |      Moves all model parameters and buffers to the GPU.\n",
      " |      \n",
      " |      This also makes associated parameters and buffers different objects. So\n",
      " |      it should be called before constructing optimizer if the module will\n",
      " |      live on GPU while being optimized.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          device (int, optional): if specified, all parameters will be\n",
      " |              copied to that device\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  double(self)\n",
      " |      Casts all floating point parameters and buffers to ``double`` datatype.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  eval(self)\n",
      " |      Sets the module in evaluation mode.\n",
      " |      \n",
      " |      This has any effect only on certain modules. See documentations of\n",
      " |      particular modules for details of their behaviors in training/evaluation\n",
      " |      mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,\n",
      " |      etc.\n",
      " |      \n",
      " |      This is equivalent with :meth:`self.train(False) <torch.nn.Module.train>`.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  float(self)\n",
      " |      Casts all floating point parameters and buffers to float datatype.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  half(self)\n",
      " |      Casts all floating point parameters and buffers to ``half`` datatype.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  load_state_dict(self, state_dict, strict=True)\n",
      " |      Copies parameters and buffers from :attr:`state_dict` into\n",
      " |      this module and its descendants. If :attr:`strict` is ``True``, then\n",
      " |      the keys of :attr:`state_dict` must exactly match the keys returned\n",
      " |      by this module's :meth:`~torch.nn.Module.state_dict` function.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          state_dict (dict): a dict containing parameters and\n",
      " |              persistent buffers.\n",
      " |          strict (bool, optional): whether to strictly enforce that the keys\n",
      " |              in :attr:`state_dict` match the keys returned by this module's\n",
      " |              :meth:`~torch.nn.Module.state_dict` function. Default: ``True``\n",
      " |      \n",
      " |      Returns:\n",
      " |          ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields:\n",
      " |              * **missing_keys** is a list of str containing the missing keys\n",
      " |              * **unexpected_keys** is a list of str containing the unexpected keys\n",
      " |  \n",
      " |  modules(self)\n",
      " |      Returns an iterator over all modules in the network.\n",
      " |      \n",
      " |      Yields:\n",
      " |          Module: a module in the network\n",
      " |      \n",
      " |      Note:\n",
      " |          Duplicate modules are returned only once. In the following\n",
      " |          example, ``l`` will be returned only once.\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> l = nn.Linear(2, 2)\n",
      " |          >>> net = nn.Sequential(l, l)\n",
      " |          >>> for idx, m in enumerate(net.modules()):\n",
      " |                  print(idx, '->', m)\n",
      " |      \n",
      " |          0 -> Sequential(\n",
      " |            (0): Linear(in_features=2, out_features=2, bias=True)\n",
      " |            (1): Linear(in_features=2, out_features=2, bias=True)\n",
      " |          )\n",
      " |          1 -> Linear(in_features=2, out_features=2, bias=True)\n",
      " |  \n",
      " |  named_buffers(self, prefix='', recurse=True)\n",
      " |      Returns an iterator over module buffers, yielding both the\n",
      " |      name of the buffer as well as the buffer itself.\n",
      " |      \n",
      " |      Args:\n",
      " |          prefix (str): prefix to prepend to all buffer names.\n",
      " |          recurse (bool): if True, then yields buffers of this module\n",
      " |              and all submodules. Otherwise, yields only buffers that\n",
      " |              are direct members of this module.\n",
      " |      \n",
      " |      Yields:\n",
      " |          (string, torch.Tensor): Tuple containing the name and buffer\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> for name, buf in self.named_buffers():\n",
      " |          >>>    if name in ['running_var']:\n",
      " |          >>>        print(buf.size())\n",
      " |  \n",
      " |  named_children(self)\n",
      " |      Returns an iterator over immediate children modules, yielding both\n",
      " |      the name of the module as well as the module itself.\n",
      " |      \n",
      " |      Yields:\n",
      " |          (string, Module): Tuple containing a name and child module\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> for name, module in model.named_children():\n",
      " |          >>>     if name in ['conv4', 'conv5']:\n",
      " |          >>>         print(module)\n",
      " |  \n",
      " |  named_modules(self, memo=None, prefix='')\n",
      " |      Returns an iterator over all modules in the network, yielding\n",
      " |      both the name of the module as well as the module itself.\n",
      " |      \n",
      " |      Yields:\n",
      " |          (string, Module): Tuple of name and module\n",
      " |      \n",
      " |      Note:\n",
      " |          Duplicate modules are returned only once. In the following\n",
      " |          example, ``l`` will be returned only once.\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> l = nn.Linear(2, 2)\n",
      " |          >>> net = nn.Sequential(l, l)\n",
      " |          >>> for idx, m in enumerate(net.named_modules()):\n",
      " |                  print(idx, '->', m)\n",
      " |      \n",
      " |          0 -> ('', Sequential(\n",
      " |            (0): Linear(in_features=2, out_features=2, bias=True)\n",
      " |            (1): Linear(in_features=2, out_features=2, bias=True)\n",
      " |          ))\n",
      " |          1 -> ('0', Linear(in_features=2, out_features=2, bias=True))\n",
      " |  \n",
      " |  named_parameters(self, prefix='', recurse=True)\n",
      " |      Returns an iterator over module parameters, yielding both the\n",
      " |      name of the parameter as well as the parameter itself.\n",
      " |      \n",
      " |      Args:\n",
      " |          prefix (str): prefix to prepend to all parameter names.\n",
      " |          recurse (bool): if True, then yields parameters of this module\n",
      " |              and all submodules. Otherwise, yields only parameters that\n",
      " |              are direct members of this module.\n",
      " |      \n",
      " |      Yields:\n",
      " |          (string, Parameter): Tuple containing the name and parameter\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> for name, param in self.named_parameters():\n",
      " |          >>>    if name in ['bias']:\n",
      " |          >>>        print(param.size())\n",
      " |  \n",
      " |  parameters(self, recurse=True)\n",
      " |      Returns an iterator over module parameters.\n",
      " |      \n",
      " |      This is typically passed to an optimizer.\n",
      " |      \n",
      " |      Args:\n",
      " |          recurse (bool): if True, then yields parameters of this module\n",
      " |              and all submodules. Otherwise, yields only parameters that\n",
      " |              are direct members of this module.\n",
      " |      \n",
      " |      Yields:\n",
      " |          Parameter: module parameter\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> for param in model.parameters():\n",
      " |          >>>     print(type(param.data), param.size())\n",
      " |          <class 'torch.FloatTensor'> (20L,)\n",
      " |          <class 'torch.FloatTensor'> (20L, 1L, 5L, 5L)\n",
      " |  \n",
      " |  register_backward_hook(self, hook)\n",
      " |      Registers a backward hook on the module.\n",
      " |      \n",
      " |      The hook will be called every time the gradients with respect to module\n",
      " |      inputs are computed. The hook should have the following signature::\n",
      " |      \n",
      " |          hook(module, grad_input, grad_output) -> Tensor or None\n",
      " |      \n",
      " |      The :attr:`grad_input` and :attr:`grad_output` may be tuples if the\n",
      " |      module has multiple inputs or outputs. The hook should not modify its\n",
      " |      arguments, but it can optionally return a new gradient with respect to\n",
      " |      input that will be used in place of :attr:`grad_input` in subsequent\n",
      " |      computations.\n",
      " |      \n",
      " |      Returns:\n",
      " |          :class:`torch.utils.hooks.RemovableHandle`:\n",
      " |              a handle that can be used to remove the added hook by calling\n",
      " |              ``handle.remove()``\n",
      " |      \n",
      " |      .. warning ::\n",
      " |      \n",
      " |          The current implementation will not have the presented behavior\n",
      " |          for complex :class:`Module` that perform many operations.\n",
      " |          In some failure cases, :attr:`grad_input` and :attr:`grad_output` will only\n",
      " |          contain the gradients for a subset of the inputs and outputs.\n",
      " |          For such :class:`Module`, you should use :func:`torch.Tensor.register_hook`\n",
      " |          directly on a specific input or output to get the required gradients.\n",
      " |  \n",
      " |  register_buffer(self, name, tensor)\n",
      " |      Adds a persistent buffer to the module.\n",
      " |      \n",
      " |      This is typically used to register a buffer that should not to be\n",
      " |      considered a model parameter. For example, BatchNorm's ``running_mean``\n",
      " |      is not a parameter, but is part of the persistent state.\n",
      " |      \n",
      " |      Buffers can be accessed as attributes using given names.\n",
      " |      \n",
      " |      Args:\n",
      " |          name (string): name of the buffer. The buffer can be accessed\n",
      " |              from this module using the given name\n",
      " |          tensor (Tensor): buffer to be registered.\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> self.register_buffer('running_mean', torch.zeros(num_features))\n",
      " |  \n",
      " |  register_forward_hook(self, hook)\n",
      " |      Registers a forward hook on the module.\n",
      " |      \n",
      " |      The hook will be called every time after :func:`forward` has computed an output.\n",
      " |      It should have the following signature::\n",
      " |      \n",
      " |          hook(module, input, output) -> None or modified output\n",
      " |      \n",
      " |      The hook can modify the output. It can modify the input inplace but\n",
      " |      it will not have effect on forward since this is called after\n",
      " |      :func:`forward` is called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          :class:`torch.utils.hooks.RemovableHandle`:\n",
      " |              a handle that can be used to remove the added hook by calling\n",
      " |              ``handle.remove()``\n",
      " |  \n",
      " |  register_forward_pre_hook(self, hook)\n",
      " |      Registers a forward pre-hook on the module.\n",
      " |      \n",
      " |      The hook will be called every time before :func:`forward` is invoked.\n",
      " |      It should have the following signature::\n",
      " |      \n",
      " |          hook(module, input) -> None or modified input\n",
      " |      \n",
      " |      The hook can modify the input. User can either return a tuple or a\n",
      " |      single modified value in the hook. We will wrap the value into a tuple\n",
      " |      if a single value is returned(unless that value is already a tuple).\n",
      " |      \n",
      " |      Returns:\n",
      " |          :class:`torch.utils.hooks.RemovableHandle`:\n",
      " |              a handle that can be used to remove the added hook by calling\n",
      " |              ``handle.remove()``\n",
      " |  \n",
      " |  register_parameter(self, name, param)\n",
      " |      Adds a parameter to the module.\n",
      " |      \n",
      " |      The parameter can be accessed as an attribute using given name.\n",
      " |      \n",
      " |      Args:\n",
      " |          name (string): name of the parameter. The parameter can be accessed\n",
      " |              from this module using the given name\n",
      " |          param (Parameter): parameter to be added to the module.\n",
      " |  \n",
      " |  requires_grad_(self, requires_grad=True)\n",
      " |      Change if autograd should record operations on parameters in this\n",
      " |      module.\n",
      " |      \n",
      " |      This method sets the parameters' :attr:`requires_grad` attributes\n",
      " |      in-place.\n",
      " |      \n",
      " |      This method is helpful for freezing part of the module for finetuning\n",
      " |      or training parts of a model individually (e.g., GAN training).\n",
      " |      \n",
      " |      Args:\n",
      " |          requires_grad (bool): whether autograd should record operations on\n",
      " |                                parameters in this module. Default: ``True``.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  share_memory(self)\n",
      " |  \n",
      " |  state_dict(self, destination=None, prefix='', keep_vars=False)\n",
      " |      Returns a dictionary containing a whole state of the module.\n",
      " |      \n",
      " |      Both parameters and persistent buffers (e.g. running averages) are\n",
      " |      included. Keys are corresponding parameter and buffer names.\n",
      " |      \n",
      " |      Returns:\n",
      " |          dict:\n",
      " |              a dictionary containing a whole state of the module\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> module.state_dict().keys()\n",
      " |          ['bias', 'weight']\n",
      " |  \n",
      " |  to(self, *args, **kwargs)\n",
      " |      Moves and/or casts the parameters and buffers.\n",
      " |      \n",
      " |      This can be called as\n",
      " |      \n",
      " |      .. function:: to(device=None, dtype=None, non_blocking=False)\n",
      " |      \n",
      " |      .. function:: to(dtype, non_blocking=False)\n",
      " |      \n",
      " |      .. function:: to(tensor, non_blocking=False)\n",
      " |      \n",
      " |      Its signature is similar to :meth:`torch.Tensor.to`, but only accepts\n",
      " |      floating point desired :attr:`dtype` s. In addition, this method will\n",
      " |      only cast the floating point parameters and buffers to :attr:`dtype`\n",
      " |      (if given). The integral parameters and buffers will be moved\n",
      " |      :attr:`device`, if that is given, but with dtypes unchanged. When\n",
      " |      :attr:`non_blocking` is set, it tries to convert/move asynchronously\n",
      " |      with respect to the host if possible, e.g., moving CPU Tensors with\n",
      " |      pinned memory to CUDA devices.\n",
      " |      \n",
      " |      See below for examples.\n",
      " |      \n",
      " |      .. note::\n",
      " |          This method modifies the module in-place.\n",
      " |      \n",
      " |      Args:\n",
      " |          device (:class:`torch.device`): the desired device of the parameters\n",
      " |              and buffers in this module\n",
      " |          dtype (:class:`torch.dtype`): the desired floating point type of\n",
      " |              the floating point parameters and buffers in this module\n",
      " |          tensor (torch.Tensor): Tensor whose dtype and device are the desired\n",
      " |              dtype and device for all parameters and buffers in this module\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |      \n",
      " |      Example::\n",
      " |      \n",
      " |          >>> linear = nn.Linear(2, 2)\n",
      " |          >>> linear.weight\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 0.1913, -0.3420],\n",
      " |                  [-0.5113, -0.2325]])\n",
      " |          >>> linear.to(torch.double)\n",
      " |          Linear(in_features=2, out_features=2, bias=True)\n",
      " |          >>> linear.weight\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 0.1913, -0.3420],\n",
      " |                  [-0.5113, -0.2325]], dtype=torch.float64)\n",
      " |          >>> gpu1 = torch.device(\"cuda:1\")\n",
      " |          >>> linear.to(gpu1, dtype=torch.half, non_blocking=True)\n",
      " |          Linear(in_features=2, out_features=2, bias=True)\n",
      " |          >>> linear.weight\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 0.1914, -0.3420],\n",
      " |                  [-0.5112, -0.2324]], dtype=torch.float16, device='cuda:1')\n",
      " |          >>> cpu = torch.device(\"cpu\")\n",
      " |          >>> linear.to(cpu)\n",
      " |          Linear(in_features=2, out_features=2, bias=True)\n",
      " |          >>> linear.weight\n",
      " |          Parameter containing:\n",
      " |          tensor([[ 0.1914, -0.3420],\n",
      " |                  [-0.5112, -0.2324]], dtype=torch.float16)\n",
      " |  \n",
      " |  train(self, mode=True)\n",
      " |      Sets the module in training mode.\n",
      " |      \n",
      " |      This has any effect only on certain modules. See documentations of\n",
      " |      particular modules for details of their behaviors in training/evaluation\n",
      " |      mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,\n",
      " |      etc.\n",
      " |      \n",
      " |      Args:\n",
      " |          mode (bool): whether to set training mode (``True``) or evaluation\n",
      " |                       mode (``False``). Default: ``True``.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  type(self, dst_type)\n",
      " |      Casts all parameters and buffers to :attr:`dst_type`.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          dst_type (type or string): the desired type\n",
      " |      \n",
      " |      Returns:\n",
      " |          Module: self\n",
      " |  \n",
      " |  zero_grad(self)\n",
      " |      Sets gradients of all model parameters to zero.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from torch.nn.modules.module.Module:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from torch.nn.modules.module.Module:\n",
      " |  \n",
      " |  dump_patches = False\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(nn.Conv2d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-pytorch] *",
   "language": "python",
   "name": "conda-env-.conda-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}