09.Perceptron_and_neural_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 感知机(Perceptron)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "感知机是由美国学者 Frank Rosenblatt 在1957 年提出来的。感知机是作为神经网络（深度学习）的起源的算法，学习感知机的构造也是学习通向神经网络和深度学习的一种重要思想。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "感知机接收多个输入信号，输出一个信号。感知机的输出信号只有 **激活** 和 **抑制** (**未激活**) 两种状态，使用1代表激活，0代表未激活。下图是一个接收两个输入信号的感知机的例子。x1、x2 是输入信号，y 是输出信号，w1、w2 是权重。\n",
    "\n",
    "![9.perceptron.png](images/chapter09/perceptron.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "图中的 ○ 称为“神经元”或者“节点”。输入信号被送往神经元时，会被分别乘以固定的权重（$w_1 x_1$、$w_2 x_2$）。神经元会计算传送过来的信号的总和，只有当这个总和超过了某个界限值时，才会输出1。这也称为“神经元被激活” 。这里将这个界限值称为阈值，用符号 θ 表示。\n",
    "\n",
    "$$y=\n",
    "\\begin{cases}\n",
    "0&w_1 x_1 + w_2 x2 <= \\theta \\\\\n",
    "1&w_1 x_1 + w_2 x2 > \\theta\n",
    "\\end{cases}$$\n",
    "\n",
    "如果将阈值 θ 改记为 -b，我们将 b 称为偏置(bias)，则\n",
    "\n",
    "$$y=\n",
    "\\begin{cases}\n",
    "0&w_1 x_1 + w_2 x2 + b <= 0 \\\\\n",
    "1&w_1 x_1 + w_2 x2 + b > 0\n",
    "\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 实现逻辑运算\n",
    "\n",
    "现在让我们考虑用感知机来解决简单的逻辑电路问题。这里首先以逻辑电路为题材来思考一下**与门**（AND gate）。与门是有两个输入和一个输出的门电路。\n",
    "\n",
    "|输入x1|输入x2|输出y|\n",
    "|--|--|-|\n",
    "| 0| 0|0|\n",
    "| 1| 0|0|\n",
    "| 0| 1|0|\n",
    "| 1| 1|1|\n",
    "\n",
    "与门仅在两个输入均为 1 时输出 1，其他时候则输出 0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用 Python 和 Numpy 实现一个 AND() 函数，这个函数接受两个参数 x1 和 x2，并在函数内初始化参数 w1、w2 和 b："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def AND(x1, x2):\n",
    "    x = np.array([x1, x2])\n",
    "    w = np.array([0.5, 0.5])\n",
    "    b = -0.7\n",
    "    tmp = np.sum(w*x) + b           # np.sum(w*x)计算相乘后的各个元素的总和\n",
    "    if tmp <= 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "0\n",
      "0\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(AND(0, 0)) # 输出0\n",
    "print(AND(1, 0)) # 输出0\n",
    "print(AND(0, 1)) # 输出0\n",
    "print(AND(1, 1)) # 输出1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了“与”门，逻辑电路还有两种常用的门：**或门** (OR gate) 和**非门** (NOT gate)。其中**或门**是“只要有一个输入信号是1，输出就为1”的逻辑电路。\n",
    "\n",
    "|输入x1|输入x2|输出y|\n",
    "|--|--|-|\n",
    "| 0| 0|0|\n",
    "| 1| 0|1|\n",
    "| 0| 1|1|\n",
    "| 1| 1|1|\n",
    "\n",
    "**非门**只有一个输入端，并且输入端和输出端的电平状态总是反相的：\n",
    "\n",
    "|输入x|输出y|\n",
    "|--|--|\n",
    "| 0| 1|\n",
    "| 1| 0|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们继续使用 Python 实现或门和非门，其中在或门的实现中，仅设置权重和偏置的值这一点和与门的实现不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def OR(x1, x2):\n",
    "    x = np.array([x1, x2])\n",
    "    w = np.array([0.5, 0.5])   # 仅权重和偏置与AND不同\n",
    "    b = -0.2\n",
    "    tmp = np.sum(w*x) + b\n",
    "    if tmp <= 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def NOT(x):\n",
    "    return 1 if x <= 0 else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 感知机的局限性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们来考虑一下**异或门**（XOR gate）。仅当x1 或x2 中的一方为1 时，才会输出1（“异或”是拒绝其他的意思）。\n",
    "\n",
    "|输入x1|输入x2|输出y|\n",
    "|--|--|-|\n",
    "| 0| 0|0|\n",
    "| 1| 0|1|\n",
    "| 0| 1|1|\n",
    "| 1| 1|0|\n",
    "\n",
    "那么，要用感知机实现这个异或门的话，应该设定什么样的权重参数呢？\n",
    "\n",
    "**实际上，用前面介绍的感知机是无法实现这个异或门的。**\n",
    "\n",
    "只拥有一层功能神经元，其学习能力非常有限。与、或、非问题都是线性可分(linearly separable)的问题，即存在一个线性超平面能将它们分开。而异或属于非线性可分问题。\n",
    "\n",
    "![img](images/chapter09/linearly_separable.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 多层感知机\n",
    "\n",
    "感知机不能表示异或门让人深感遗憾，但通过叠加层来构造感知机可以解决非线性可分问题。\n",
    "\n",
    "下面我们使用之前定义的AND函数、OR函数，可以轻松实现异或门。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def XOR(x1, x2):\n",
    "    s1 = not AND(x1, x2)\n",
    "    s2 = OR(x1, x2)\n",
    "    y = AND(s1, s2)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "1\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "print(XOR(0, 0)) # 输出0\n",
    "print(XOR(1, 0)) # 输出1\n",
    "print(XOR(0, 1)) # 输出1\n",
    "print(XOR(1, 1)) # 输出0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与门、或门是单层感知机，而异或门是一种**多层感知机**（MLP, multi-layered perceptron）。\n",
    "\n",
    "![img](images/chapter09/multi_layer_perceptron.png)\n",
    "\n",
    "- 第0 层的两个神经元接收输入信号，并将信号发送至第1 层的神经元。\n",
    "- 第1 层的神经元将信号发送至第2 层的神经元，第2 层的神经元输出y。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. 神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "更一般的，常见的神经网络是如下图所示的层级结构，每层神经元与下一层神经元全互连，神经元之间不存在同层连接，也不存在跨层连接。这样的网络也通常称为 **多层前馈神经网络**(multi-layer feedforward neural network)。\n",
    "\n",
    "![img](images/chapter09/neural_network.png)\n",
    "\n",
    "最左边的一列称为**输入层**，最右边的一列称为**输出层**，中间的一列称为**中间层**，也称为隐藏层。中间层和输出层神经元都是拥有激活函数的功能神经元。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 激活函数\n",
    "\n",
    "神经网络的激活函数必须使用非线性函数，否则加深神经网络的层数就没有意义了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.1 阶跃函数\n",
    "\n",
    "理想中的激活函数是阶跃函数，如上一节所述它将输入值映射为输出值 0 或者 1 ，0 对应神经元抑制，1 对应神经元激活。因此阶跃函数具有不连续、不光滑等性质。可以像下面这样简单地实现阶跃函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def step_function(x):\n",
    "    if x > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个实现简单、易于理解，但是参数 x 只能接受实数，但不允许参数取 NumPy 数组，例如 step_function(np.array([1.0, 2.0]))，我们将它修改为支持 NumPy 数组的实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step_function(x):\n",
    "    y = x > 0\n",
    "    return y.astype(np.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Devon\\AppData\\Local\\Temp/ipykernel_7284/821751868.py:3: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  return y.astype(np.int)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step_function(np.array([-1.0, 1.0, 2.0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pylab as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Devon\\AppData\\Local\\Temp/ipykernel_7284/4084177707.py:2: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  return np.array(x > 0, dtype=np.int)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQ0klEQVR4nO3db4wd91XG8efxbkNBSUmLl1D8J7aQA7UggWbrRqoggdDGTqNaSLxIAg0EKstSglKJigQq/kjlDYqAqkpaY0VWqBphISVQE7mEIkH6ogp4E/KnTki0ckm8cSCbFrWoEbLuzOHFvd5cNjNzx/bdOz53vx/Jys6dWe+5ytxHx785O+OIEAAgvw1dFwAAGA8CHQCmBIEOAFOCQAeAKUGgA8CUmO3qB2/cuDG2bdvW1Y8HgJSefPLJNyJirmpfZ4G+bds2LSwsdPXjASAl2y/X7WPJBQCmBIEOAFOCQAeAKUGgA8CUINABYEqMDHTbh2y/bvsbNftt+3O2F20/a/v94y8TADBKmw79QUm7G/bvkbRj8GefpC+cf1kAgLM1cg49Ir5me1vDIXslfTH69+F9wvaltt8bEa+Nq0igS6986009/NSSuNU0xmV+23v0c1dU/m7QeRnHLxZtknRyaHtp8NrbAt32PvW7eG3dunUMPxpYew/968v6i8dPyO66EkyL/df+2AUb6FWneWUrExEHJR2UpPn5edodpHC6V+qSd87quT+6oetSgEbjmHJZkrRlaHuzpFNj+HuBC0JRhmY20J7jwjeOQD8i6bbBtMs1kr7D+jmmSa8MzRLoSGDkkovtv5J0naSNtpck/aGkd0hSRByQdFTSjZIWJb0p6fa1KhboQlHQoSOHNlMut4zYH5LuGFtFwAWmiNDsBn4HDxc+zlJgBNbQkQWBDozAGjqyINCBEYqypENHCgQ6MEKPi6JIgkAHRijK0OwMgY4LH4EOjNArQzNMuSABzlJghIKLokiCQAdG6HFRFEkQ6MAIdOjIgkAHRujxi0VIgkAHRqBDRxYEOjBCfw6djwoufJylwAh06MiCQAdG6JWlZvjFIiRAoAMj0KEjCwIdGIEpF2RBoAMj0KEjCwIdGIF7uSALzlJgBDp0ZEGgAyP0Cu7lghwIdGAEOnRkQaADI/TKYA4dKRDowAh06MiCQAcaRARTLkiDsxRoUEb/v3ToyIBABxr0ylKSmHJBCgQ60KAYtOh06MiAQAca9AaBToeODAh0oEFR0KEjDwIdaLDSoc/wUcGFj7MUaMAaOjJpFei2d9t+0fai7Xsq9v+g7b+z/Yzt47ZvH3+pwOQx5YJMRga67RlJ90vaI2mnpFts71x12B2Sno+IqyRdJ+lPbV805lqBiaNDRyZtOvRdkhYj4kREnJZ0WNLeVceEpEtsW9LFkr4tqTfWSoEOMOWCTNoE+iZJJ4e2lwavDbtP0vsknZL0nKS7IqJc/RfZ3md7wfbC8vLyOZYMTM5bHTqXm3Dha3OWVrUmsWr7BklPS/pRST8t6T7b73rbN0UcjIj5iJifm5s7y1KByesVdOjIo02gL0naMrS9Wf1OfNjtkh6JvkVJ35T0E+MpEegOa+jIpE2gH5O0w/b2wYXOmyUdWXXMK5KulyTbl0n6cUknxlko0IWVKRfuh44EZkcdEBE923dKekzSjKRDEXHc9v7B/gOSPiPpQdvPqb9Ec3dEvLGGdQMTQYeOTEYGuiRFxFFJR1e9dmDo61OSPjLe0oDuMeWCTLh0DzRgygWZcJYCDejQkQmBDjQoBhdFWUNHBgQ60IA5dGRCoAMNVtbQGVtEAgQ60KDH2CISIdCBBsXKRVE+KrjwcZYCDejQkQmBDjQoeMAFEiHQgQbMoSMTAh1oUBDoSIRABxpwcy5kQqADDejQkQmBDjTocXMuJMJZCjSgQ0cmBDrQ4My9XFhDRwYEOtCgKEvZ0gYCHQkQ6ECDXhl050iDQAcaFGWwfo40CHSgQb9D52OCHDhTgQZ06MiEQAca9MqSNXSkQaADDejQkQmBDjToFUy5IA8CHWhQlKEZnieKJAh0oAFTLsiEMxVowBo6MiHQgQZMuSATAh1oQIeOTAh0oAH3ckEmrQLd9m7bL9petH1PzTHX2X7a9nHbj4+3TKAbdOjIZHbUAbZnJN0v6cOSliQds30kIp4fOuZSSZ+XtDsiXrH9w2tULzBR/Tl0/iGLHNqcqbskLUbEiYg4LemwpL2rjrlV0iMR8YokRcTr4y0T6AYdOjJpE+ibJJ0c2l4avDbsCknvtv3Ptp+0fVvVX2R7n+0F2wvLy8vnVjEwQb2y1Cy/WIQk2gR61dkcq7ZnJV0t6aOSbpD0+7aveNs3RRyMiPmImJ+bmzvrYoFJo0NHJiPX0NXvyLcMbW+WdKrimDci4nuSvmf7a5KukvTSWKoEOsKUCzJp06Efk7TD9nbbF0m6WdKRVcd8WdLP2p61/QOSPijphfGWCkweHToyGdmhR0TP9p2SHpM0I+lQRBy3vX+w/0BEvGD77yU9K6mU9EBEfGMtCwcmgXu5IJM2Sy6KiKOSjq567cCq7Xsl3Tu+0oDu0aEjE1oPoAH3ckEmBDrQoCjo0JEHgQ406JXBHDrSINCBBqyhIxMCHWjAlAsy4UwFGtChIxMCHWjAlAsyIdCBBnToyIRABxpwLxdkQqADNcoyFCHNcFEUSXCmAjV6Zf8u0cyhIwsCHahRDAKdNXRkQaADNXplKUmsoSMNAh2oQYeObAh0oMbKGjqBjiQIdKDGWx06HxPkwJkK1KBDRzYEOlCjKFhDRy4EOlBjZcqFOXQkQaADNZhyQTYEOlCDNXRkQ6ADNZhyQTacqUCN3kqgd1wI0BKnKlCjGFwUpUNHFpypQI1ewRo6ciHQgRpFMOWCXAh0oEbBlAuSIdCBGj3m0JEMgQ7UKFbW0PmYIAfOVKAGHTqyIdCBGgXPFEUyrQLd9m7bL9petH1Pw3EfsF3Y/uXxlQh0o7cyh06gI4eRgW57RtL9kvZI2inpFts7a477E0mPjbtIoAtMuSCbNh36LkmLEXEiIk5LOixpb8VxvyXpYUmvj7E+oDOsoSObNoG+SdLJoe2lwWsrbG+S9EuSDjT9Rbb32V6wvbC8vHy2tQIT9VaHzqUm5NDmTK1qT2LV9mcl3R0RRdNfFBEHI2I+Iubn5uZalgh0gw4d2cy2OGZJ0pah7c2STq06Zl7SYduStFHSjbZ7EfG34ygS6EJRDJ5YRKAjiTaBfkzSDtvbJb0q6WZJtw4fEBHbz3xt+0FJjxLmyG6lQ2dsEUmMDPSI6Nm+U/3plRlJhyLiuO39g/2N6+ZAVky5IJs2Hboi4qiko6teqwzyiPj18y8L6B5r6MiGy/dADaZckA1nKlDjTIdOg44sCHSgRlGWmt1gDaa3gAsegQ7U6JXB+jlSIdCBGkURTLggFQIdqEGHjmwIdKBGUYZmZ/iIIA/OVqAGHTqyIdCBGmemXIAsCHSgBh06siHQgRpFyZQLciHQgRp06MiGQAdq9OfQ+YggD85WoAYdOrIh0IEaRVlqlodbIBECHahBh45sCHSgBlMuyIZAB2rQoSMbAh2o0e/Q+YggD85WoAYdOrIh0IEa3MsF2RDoQI1eQYeOXAh0oEb/fugEOvIg0IEaRRma4aIoEuFsBWr0mENHMgQ6UKNgygXJEOhAjR5TLkiGQAdq0KEjGwIdqMEaOrIh0IEaRcGUC3Jpdbba3m37RduLtu+p2P8rtp8d/Pm67avGXyowWT3m0JHMyEC3PSPpfkl7JO2UdIvtnasO+6akayPiSkmfkXRw3IUCk8YaOrJp06HvkrQYESci4rSkw5L2Dh8QEV+PiP8ebD4hafN4ywQmjykXZNMm0DdJOjm0vTR4rc5vSvpK1Q7b+2wv2F5YXl5uXyUwYWUZKkN06EilTaBXndFReaD98+oH+t1V+yPiYETMR8T83Nxc+yqBCSuif4rToSOT2RbHLEnaMrS9WdKp1QfZvlLSA5L2RMS3xlMe0I2i7Af6BgIdibTp0I9J2mF7u+2LJN0s6cjwAba3SnpE0scj4qXxlwlMVq+kQ0c+Izv0iOjZvlPSY5JmJB2KiOO29w/2H5D0B5J+SNLnbUtSLyLm165sYG2d6dCZQ0cmbZZcFBFHJR1d9dqBoa8/IekT4y0N6E5Bh46EaD+ACr2ylMSUC3Ih0IEKdOjIiEAHKvSKM2voBDryINCBCisdOvdyQSIEOlChx5QLEuJsBSqwho6MCHSgAlMuyIhAByrQoSMjAh2o8NYaOoGOPAh0oMJbHTofEeTB2QpUYA4dGRHoQAXm0JERgQ5UYMoFGRHoQAWmXJARgQ5UYMoFGRHoQAWmXJARZytQgQ4dGRHoQIVicFGUNXRkQqADFZhDR0YEOlCBOXRkRKADFVhDR0YEOlCBKRdkxNkKVKBDR0YEOlCBKRdkRKADFejQkRGBDlQoCu7lgnwIdKACHToyItCBCkUZmtlg2QQ68iDQgQq9QaADmRDoQIWiLFk/RzoEOlCBDh0ZtQp027ttv2h70fY9Fftt+3OD/c/afv/4SwUmpyiDDh3pjAx02zOS7pe0R9JOSbfY3rnqsD2Sdgz+7JP0hTHXCUxUv0PnH7DIZbbFMbskLUbECUmyfVjSXknPDx2zV9IXIyIkPWH7UtvvjYjXxl3w4y8t648ffX70gcB5+K/v/q++/6KZrssAzkqbQN8k6eTQ9pKkD7Y4ZpOk/xfotvep38Fr69atZ1urJOni75vVjssuPqfvBdracdnF+sC293RdBnBW2gR61UJinMMxioiDkg5K0vz8/Nv2t3H15e/W1ZdffS7fCgBTrc0i4ZKkLUPbmyWdOodjAABrqE2gH5O0w/Z22xdJulnSkVXHHJF022Da5RpJ31mL9XMAQL2RSy4R0bN9p6THJM1IOhQRx23vH+w/IOmopBslLUp6U9Lta1cyAKBKmzV0RcRR9UN7+LUDQ1+HpDvGWxoA4GwwaAsAU4JAB4ApQaADwJQg0AFgSrh/PbODH2wvS3q5kx9+fjZKeqPrIjqwHt/3enzP0vp835ne8+URMVe1o7NAz8r2QkTMd13HpK3H970e37O0Pt/3tLxnllwAYEoQ6AAwJQj0s3ew6wI6sh7f93p8z9L6fN9T8Z5ZQweAKUGHDgBTgkAHgClBoJ8H25+yHbY3dl3LWrN9r+1/HzwE/G9sX9p1TWtp1IPRp43tLbb/yfYLto/bvqvrmibF9oztf7P9aNe1nC8C/RzZ3iLpw5Je6bqWCfmqpJ+MiCslvSTpdzuuZ820fDD6tOlJ+u2IeJ+kayTdsQ7e8xl3SXqh6yLGgUA/d38u6XdU8ai9aRQR/xARvcHmE+o/lWparTwYPSJOSzrzYPSpFRGvRcRTg6//R/2A29RtVWvP9mZJH5X0QNe1jAOBfg5sf0zSqxHxTNe1dOQ3JH2l6yLWUN1Dz9cF29sk/Yykf+m4lEn4rPqNWdlxHWPR6gEX65Htf5T0IxW7Pi3p9yR9ZLIVrb2m9xwRXx4c82n1/3n+0CRrm7BWDz2fRrYvlvSwpE9GxHe7rmct2b5J0usR8aTt6zouZywI9BoR8YtVr9v+KUnbJT1jW+ovPTxle1dE/OcESxy7uvd8hu1fk3STpOtjun+BYV0+9Nz2O9QP84ci4pGu65mAD0n6mO0bJb1T0rtsfykifrXjus4Zv1h0nmz/h6T5iMhyp7ZzYnu3pD+TdG1ELHddz1qyPav+hd/rJb2q/oPSb42I450Wtobc707+UtK3I+KTHZczcYMO/VMRcVPHpZwX1tDR1n2SLpH0VdtP2z4w6huyGlz8PfNg9Bck/fU0h/nAhyR9XNIvDP7/Pj3oXJEIHToATAk6dACYEgQ6AEwJAh0ApgSBDgBTgkAHgClBoAPAlCDQAWBK/B++B2F2cTIE2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def step_function(x):\n",
    "    return np.array(x > 0, dtype=np.int)\n",
    "\n",
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y = step_function(x)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.2 Sigmoid 函数\n",
    "\n",
    "由于阶跃函数的不连续性，实际中常用 sigmoid 函数作为激活函数。\n",
    "\n",
    "$$ \\sigma(x) = \\frac{1}{1+e^{-x}} $$\n",
    "\n",
    "它把可能在较大范围内变化的输入值挤压到 (0, 1) 输出范围内，因此有时也称为“挤压函数”。它的导数有性质：\n",
    "\n",
    "$$ \\sigma(x)' = \\sigma(x) \\cdot (1-\\sigma(x)) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.26894142, 0.73105858, 0.88079708])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([-1.0, 1.0, 2.0])\n",
    "sigmoid(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之所以 sigmoid 函数的实现能支持 NumPy 数组，原因就在于 NumPy 的广播功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf9klEQVR4nO3deXhV1d328e8vcwghCSRhSICEWUaFAIpVsVYFpXVsHbFaWx/b0vbppNaqfftqW2sn7VsrRbS2FcU6VKmz1lq1FiTIIEOAMAYCZCLznLPePxJtxEAOcJJ9hvtzXedK9tk7yX00uV2us/fa5pxDRERCX5TXAUREJDBU6CIiYUKFLiISJlToIiJhQoUuIhImYrz6wenp6S4nJ8erHy8iEpJWrVpV5pzL6GqfZ4Wek5NDfn6+Vz9eRCQkmdmuw+3TlIuISJhQoYuIhAkVuohImFChi4iEiW4L3cweNrMSM1t/mP1mZr81s0IzW2dmUwMfU0REuuPPCP0RYM4R9s8FRnc8bgAeOP5YIiJytLotdOfcW0DFEQ65APiza7ccSDWzwYEKKCIi/gnEeehZQFGn7T0dz+079EAzu4H2UTzDhg0LwI8WEQkePp+jrrmV6sZWahpbqGlspbaxlZqm9o/1za3UNrUybXgap43u8tqg4xKIQrcunutykXXn3CJgEUBeXp4WYheRoOWco7aplbLaZspqmyiraaK8rpmKjsfB+mYO1rdQWd9MVUMLlfUt1DS24POj2b46e2TQFvoeYGin7WygOADfV0SkRzjnqKxvYW9lA3sONlBc2cD+6kb2VTVyoKqRkppGDlQ30dDS1uXXJ8fHkJYUR1qfWNL6xJGbnkRKYiwpibH0S4ilX2IMyQmxJCfE0De+45EQQ1J8DElxMURHdTUOPn6BKPRlwAIzWwrMBKqcc5+YbhER6U3OOUpqmtheWseOsjp2ltexq7yO3RUNFFXUU9vU+rHj42KiGJySwMB+CUzKTuUzyfFk9osnve9/HwP6xpHWJ464mOA847vbQjezx4HZQLqZ7QF+BMQCOOcWAi8C5wGFQD1wXU+FFRHpSnltEwX7ayjYX8Pm/dVsLamlsKSWmsb/lnZcTBTD+vdheP8+zMztT3ZaItlpiWSl9mFIagL9k+Iw65mRc2/pttCdc1d0s98BXw9YIhGRIzhY18yaPZWsLapk/d4qNhRXs6+q8aP96X3jGJ2ZzIUnZjEqsy8jMpLITU9iSEoiUT001REsPFttUUSkO845tpfVkb+zgvd2HGTVrgp2ltcDYAYjM/oyM7c/E4akcMLgfowdlExGcrzHqb2jQheRoFJc2cDbW0t5d1s5724rp7SmCYD+SXFMG57GZdOHMWVoCpOzU+kbrwrrTP80RMRTrW0+Vu48yBsFB3hzcylbS2oBSO8bz6yRAzhl5ABm5PZnRHpSyM9x9zQVuoj0usaWNt7aUsrL6/fzj4ISqhpaiIuOYuaI/lw2fSinj8lgdGZfFfhRUqGLSK9obfPxTmEZz60p5rWNB6htaiUlMZazxmVyzoSBnDY6gyRNoRwX/dMTkR619UANf80v4tk1xZTWNJGSGMt5kwZx/uQhzBo5gNjo4DynOxSp0EUk4Bpb2nhp/T4eW7GblTsPEhttnDk2k4unZvPpcZlBe2FOqFOhi0jAlFQ38ujyXSxZsZvyumZy05O49bxxXDI1mwF9I/d0wt6iQheR41ZYUsMDb25n2dq9tPocZ40byHWn5nDKiAFhfzFPMFGhi8gxW7+3ivv/WcjLG/YTHxPFVTOHc+2sHHLSk7yOFpFU6CJy1LYcqOE3r23hpfX7SU6IYcGZo7h2Vo6mVTymQhcRvxVXNvDLVzbztzV7SYqL4Vtnjeb603LplxDrdTRBhS4ifqhramXhv7ax6K3tOOCG00Zw4xkjSUuK8zqadKJCF5HDcs7x93X7uOv5jZTUNPG5KUO4ac5YstP6eB1NuqBCF5EubSut5Y7n1vPvwnImZaWwcP40pg5L8zqWHIEKXUQ+pqXNx6K3tnPf61uJj43izgsmcOXM4T122zQJHBW6iHxkY3E1339qLRuKqzl/0mB+9LnxZCYneB1L/KRCFxHafI4H397Or17dTEpiLA9cNZW5kwZ7HUuOkgpdJMLtq2rg20+sYfn2CuZOHMRPL5qks1dClApdJIK9vvEA331yLS1tPu65dDKfn5atNchDmApdJAK1tvn45atbWPivbUwY0o/fXTmVXF2uH/JU6CIRprSmiQWPvc+KHRVcOXMYd8wbT0JstNexJABU6CIRZP3eKr7y53wO1jfz6y9M4eKp2V5HkgBSoYtEiBfW7eO7T66hf584nrpxFhOzUryOJAGmQhcJc845fv/mNn7xymamDU9j4dXTyEjWqojhSIUuEsZa23zc/twGHn9vNxeeOISfXzqZ+BjNl4crFbpImKpvbmXBY6t5o6CEr80eyffPHatTEsOcCl0kDFU1tHDdH99jTVEld144kfknD/c6kvQCFbpImCmrbWL+Q+9RWFLD/VfqEv5IokIXCSP7qhq4avEKiisbWPzF6ZwxJsPrSNKLVOgiYWJfVQOXL1pORW0zf7l+JtNz+nsdSXpZlD8HmdkcM9tsZoVmdksX+1PM7O9mttbMNpjZdYGPKiKH07nM/3z9DJV5hOq20M0sGrgfmAuMB64ws/GHHPZ1YKNzbgowG/iVmWm5NpFesL+qkSsWLae8tpk/XT+Dk3RXoYjlzwh9BlDonNvunGsGlgIXHHKMA5Kt/ZyovkAF0BrQpCLyCeW1TVy5eDllHSNz3SIusvlT6FlAUaftPR3PdfY74ASgGPgA+JZzznfoNzKzG8ws38zyS0tLjzGyiABUN7ZwzcPvUVzZwMPXTleZi1+F3tWVCO6Q7XOBNcAQ4ETgd2bW7xNf5Nwi51yecy4vI0Pvvoscq4bmNq5/ZCVbDtSw8OppzMjVnLn4V+h7gKGdtrNpH4l3dh3wjGtXCOwAxgUmooh01trm42tLVrFq10HuvewkZo/N9DqSBAl/Cn0lMNrMcjve6LwcWHbIMbuBswDMbCAwFtgeyKAi0r7Q1g//tp5/bi7lzgsncv5kXTQk/9XteejOuVYzWwC8AkQDDzvnNpjZjR37FwJ3Ao+Y2Qe0T9Hc7Jwr68HcIhHpt/8o5In8Ir7x6VFcNVOX88vH+XVhkXPuReDFQ55b2OnzYuCcwEYTkc7+ml/Eb17fwqXTsvnO2WO8jiNByK8Li0TEW//ZVs6tz3zAaaPT+dnFk7RqonRJhS4S5HaW1fHVJavISU/i/qumEhutP1vpmn4zRIJYVX0LX/rTSgx46It59EuI9TqSBDEtziUSpFrbfCx4/H2KKupZ8uWTGT4gyetIEuRU6CJB6hevbObtrWXcc8lkXTgkftGUi0gQ+vvaYv7w1nbmnzycL0wf2v0XiKBCFwk6G4uruempdUzPSeP2eYcubCpyeCp0kSBS1dDCjY+uol9iDPdfNZW4GP2Jiv80hy4SJJxzfP/JtRRXNvDE/5xCZnKC15EkxOg//yJBYvHbO3h14wF+cN4JTBuupXDl6KnQRYLAyp0V3P1yAXMnDuJLp+Z4HUdClApdxGMVdc1847HVDE1L5OeXTtZl/XLMNIcu4iHnHDc9tZaKumae+dosXQkqx0UjdBEPPfLuTl7fVMIPzhvHxKwUr+NIiFOhi3hk/d4qfvZiAZ85IZNrZ+V4HUfCgApdxAP1za188/HVpCXFcs+lUzRvLgGhOXQRD9z1wiZ2lNex5Msz6Z8U53UcCRMaoYv0stc3HuCxFbu54fQRzBqZ7nUcCSMqdJFeVFrTxM1Pr2P84H66jZwEnKZcRHqJc46bn15HbVMrSy8/kfiYaK8jSZjRCF2klyxdWcQbBSX8YO44Rg9M9jqOhCEVukgvKKqo567nN3LqqAFcc0qO13EkTKnQRXqYz+f43pNrMTPuuXQKUVE6RVF6hgpdpIf98d2drNhRwR2fHU9WaqLXcSSMqdBFetC20lruebn9atDPT8v2Oo6EORW6SA9p8zluemodCbHR/PSiSboaVHqcCl2khzzy7k5W7TrIjz47nsx+uvuQ9DwVukgP2FlWxy9eKeCscZlcdFKW13EkQqjQRQLM53Pc9PQ6YqOj+ImmWqQXqdBFAmzJil28t6OC2+eNZ1CKplqk96jQRQKouLKBu18q4LTR6TqrRXqdX4VuZnPMbLOZFZrZLYc5ZraZrTGzDWb2r8DGFAl+zjlue3Y9PofOahFPdLs4l5lFA/cDZwN7gJVmtsw5t7HTManA74E5zrndZpbZQ3lFgtaytcW8UVDC7fPGM7R/H6/jSATyZ4Q+Ayh0zm13zjUDS4ELDjnmSuAZ59xuAOdcSWBjigS3irpmfvz3jZw4NFW3kxPP+FPoWUBRp+09Hc91NgZIM7M3zWyVmV3T1TcysxvMLN/M8ktLS48tsUgQuuv5jVQ3tPDzSyYTrbVaxCP+FHpXv53ukO0YYBpwPnAucLuZfWL1fufcIudcnnMuLyMj46jDigSjd7aW8czqvdx4xkjGDtKyuOIdf25wsQcY2mk7Gyju4pgy51wdUGdmbwFTgC0BSSkSpBqa27j1bx+Qm57Egk+P8jqORDh/RugrgdFmlmtmccDlwLJDjnkOOM3MYsysDzAT2BTYqCLB57dvbGV3RT0/uWgiCbG6A5F4q9sRunOu1cwWAK8A0cDDzrkNZnZjx/6FzrlNZvYysA7wAYudc+t7MriI1wr2V/PgW9v5/LRs3exZgoI5d+h0eO/Iy8tz+fn5nvxskePl8zkuWfguu8rr+cd3ziAtKc7rSBIhzGyVcy6vq326UlTkGDz23m5W767ktvNPUJlL0FChixylkppGfv5yAbNGDtBKihJUVOgiR+nO5zfR1Orjrgsn6vJ+CSoqdJGj8K8tpfx9bTFfnz2KERl9vY4j8jEqdBE/Nba0cfuz6xmRkcSNs0d4HUfkE/y5sEhEgPv/Wcjuinoe+8pM4mN0zrkEH43QRfxQWFLLwn9t4+KTsnTOuQQtFbpIN5xz3P7sehJjo7n1/BO8jiNyWCp0kW48u2Yv/9lezs1zx5HeN97rOCKHpUIXOYKq+hbuen4TJw1L5Yrpw7yOI3JEelNU5AjueaWAg/XN/Pn6GURpnXMJchqhixzG6t0Heey93Vw7K5cJQ1K8jiPSLRW6SBda23z88G/ryUyO5zvnfOJeLSJBSYUu0oU//2cXG/dVc8e8CfSN18ykhAYVusgh9lc18qtXN3P6mAzOmzTI6zgiflOhixzizhc20uJz3HnBBC2+JSFFhS7SyVtbSnlh3T4WnDmK4QOSvI4jclRU6CIdGlvauOO59YxIT+J/ztDiWxJ69G6PSIcH3tzGzvJ6lnxZi29JaNIIXQTYXlrLA29u43NThnDqKC2+JaFJhS4RzznH7c+tJz42itvmafEtCV0qdIl4y9YW8+/Ccm6aM47M5ASv44gcMxW6RLSq+hbufH4jU4amcuUMLb4loU2FLhHtF68WUFHXzE8unEi0Ft+SEKdCl4j1/u6DLFmxmy/OymFilhbfktCnQpeI1NLm49ZnPmBgcgLfPWes13FEAkLnoUtEevidHRTsr2Hh1dO0+JaEDY3QJeIUVdRz7+tb+cwJAzl3wkCv44gEjApdIopzjh8t24AZ/FiLb0mYUaFLRHnhg328UVDCtz8zhqzURK/jiASUCl0iRlV9C/9n2UYmZvXjulNzvI4jEnB+FbqZzTGzzWZWaGa3HOG46WbWZmaXBi6iSGDc/XIBFXVN3H3xZGKiNZaR8NPtb7WZRQP3A3OB8cAVZjb+MMf9HHgl0CFFjtd7Oyp4/L3dXP+pXJ1zLmHLn2HKDKDQObfdOdcMLAUu6OK4bwBPAyUBzCdy3Jpa2/jBM+vISk3k22frhs8Svvwp9CygqNP2no7nPmJmWcBFwMIjfSMzu8HM8s0sv7S09GizihyT371RyLbSOn5y0UT6xOmccwlf/hR6V+d1uUO27wVuds61HekbOecWOefynHN5GRkZfkYUOXab9lXzwJvbuPikLGaPzfQ6jkiP8me4sgcY2mk7Gyg+5Jg8YGnHOb3pwHlm1uqcezYQIUWORZvPccvT60hJjOX2eZ9420ck7PhT6CuB0WaWC+wFLgeu7HyAcy73w8/N7BHgeZW5eO2P/97B2j1V/PaKk0hLivM6jkiP67bQnXOtZraA9rNXooGHnXMbzOzGjv1HnDcX8cLOsjp++epmzhqXyWcnD/Y6jkiv8OsdIufci8CLhzzXZZE75649/lgix87nc9z09Dpio6K466KJurxfIoaurpCw8+iKXby3o4Lb541ncIou75fIoUKXsFJUUc/dLxVw+pgMPp+X7XUckV6lQpew4fM5bn56HVFm/OziSZpqkYijQpewsWTFLt7dVs4PzhunlRQlIqnQJSzsLKvjpy+2T7VcOWOY13FEPKFCl5DX5nN878m1xEYb91wyWVMtErG0sIWEvIfe2U7+roP85rIpDEpJ8DqOiGc0QpeQtnl/Db98dQvnThjIhSdmdf8FImFMhS4hq7GljW8tXU2/hBh+cpHOahHRlIuErF+9upmC/TX88drppPeN9zqOiOc0QpeQ9O/CMh58ewfzTx7OmeO0LK4IqNAlBFXWN/Pdv65lREYSt553gtdxRIKGCl1CinOOm55aR3ldE/dddhKJcdFeRxIJGip0CSmPLt/FqxsPcNO545iUrZs9i3SmQpeQsWlfNXe+sIkzxmRw/adyu/8CkQijQpeQUN/cyjceX01KYiy/+sIUoqJ0iqLIoXTaogQ95xy3PbuebaW1/OVLM3WKoshhaIQuQe+JlUU88/5evvnp0XxqdLrXcUSClgpdgtqG4iruWLaBT41K55tnjfY6jkhQU6FL0KpubOFrS96nf5847rv8RKI1by5yRJpDl6Dk8zm+88Qa9h5sYOkNJzNA8+Yi3dIIXYLSff/YyuubSrh93njycvp7HUckJKjQJei8tvEA9/1jK5dOy+aaU4Z7HUckZKjQJagUltTy7SfWMDk7hbsunKglcUWOggpdgsbBumau/9NK4mOiWHj1NBJitU6LyNHQm6ISFJpbfdz46Cr2VTby+A0zGZKa6HUkkZCjQhfPOee447n1rNhRwb2Xnci04XoTVORYaMpFPPfg29tZurKIBWeO4sKTdF9QkWOlQhdPPbdmLz99sYDzJw3mO2eP8TqOSEhToYtn3t1WxveeXMuM3P5aQVEkAFTo4onN+2v4n7+sImdAEg/Oz9MZLSIB4Fehm9kcM9tsZoVmdksX+68ys3Udj3fNbErgo0q42FVex/yHVtAnLppHvjSDlD6xXkcSCQvdFrqZRQP3A3OB8cAVZjb+kMN2AGc45yYDdwKLAh1UwsP+qkaufmgFLW0+Hr1+Jlk6PVEkYPwZoc8ACp1z251zzcBS4ILOBzjn3nXOHezYXA5kBzamhIOKumaufmgFFbXNPHLdDEYPTPY6kkhY8afQs4CiTtt7Op47nOuBl7raYWY3mFm+meWXlpb6n1JCXmV9M/MfWsHuinoWf3E6U4ameh1JJOz4U+hdnXrgujzQ7EzaC/3mrvY75xY55/Kcc3kZGRn+p5SQVlnfPjLfeqCWP8yfxikjB3gdSSQs+XOl6B5gaKftbKD40IPMbDKwGJjrnCsPTDwJdVX1LVz90Aq27K/lD9dM48yxmV5HEglb/ozQVwKjzSzXzOKAy4FlnQ8ws2HAM8B859yWwMeUUFRe28RVDy1vL/P5KnORntbtCN0512pmC4BXgGjgYefcBjO7sWP/QuAOYADw+47lTludc3k9F1uC3f6qRq5avJy9lQ0sumYas1XmIj3OnOtyOrzH5eXlufz8fE9+tvSsXeV1XLV4BZX1LTx87XRm5GqxLZFAMbNVhxswa7VFCagP9lRx3SMrafP5eOwrM5mcnep1JJGIoUv/JWD+ubmEyxb9h/iYKJ688RSVuUgv0whdAmLpe7v54bPrGTcomT9eO53MfgleRxKJOCp0OS6tbT5+9lIBD72zg9PHZPD7q6bSN16/ViJe0F+eHLOq+hYWPP4+b28t49pZOdx2/gnERGsWT8QrKnQ5JgX7q/nqo++z52A9d188ictnDPM6kkjEU6HLUXtq1R5ue/YDkhNieewrJzM9R6cligQDFbr4raG5jR//fQNLVxZxyogB3HfFiWQm681PkWChQhe/rN9bxTeXrmZHWR1fmz2S75w9RvPlIkFGhS5H1OZzPPj2dn716mYGJMWz5PqZzBqV7nUsEemCCl0Oq7Ckhu8/tY7VuyuZM2EQP7t4EmlJcV7HEpHDUKHLJ7S0+Xjw7e3c+/pW+sRFc9/lJ/K5KUPoWHhNRIKUCl0+ZuXOCm7723o2H6hh7sRB/N8LJpKRHO91LBHxgwpdACipaeSelzfz1Ko9ZKUmsmj+NM6ZMMjrWCJyFFToEa6xpY3Fb2/ngTe30dzm46uzR/KNT4+iT5x+NURCjf5qI1Rrm49nVu/l3te2UFzVyLkTBnLL3BPITU/yOpqIHCMVeoTx+RzPf7CPe1/bwvayOiZnp/Dry07k5BG6cbNIqFOhR4iWNh/L1hTz+zcL2VZax9iByfxh/jTOGT9QZ6+IhAkVepirbWrlyfwiFr+9g72VDYwblMz/u+Ikzps0mOgoFblIOFGhh6mdZXX8Zfku/rqyiJqmVvKGp3HnhRM4c2ymRuQiYUqFHkaaW328vukAj63YzTuFZcREGfMmD+a6U3OZMjTV63gi0sNU6CHOOcf6vdU8/f4elq0tpqKumazURL579hi+MH0oA3UrOJGIoUIPUVsO1PD8un28sK6YbaV1xMVEcfb4gVw6NZvTx2RoflwkAqnQQ4TP51hdVMlrGw/w2sb9bCutI8pgZu4AvvSpXOZNGkJKn1ivY4qIh1ToQaystol3tpbxry2lvL21lLLaZmKijJkj+vPFWTnMmThIN5gQkY+o0INIRV0zK3dWsHx7Of/ZVk7B/hoA+ifFcfrodM4cl8nssZmkJGokLiKfpEL3SJvPseVADWuLKllTVMnKnRVsK60DID4miuk5/fn+uUP41Kh0JmWlEKU5cRHphgq9FzS2tLH1QC2b9lezYW8V64ur2VhcTUNLGwApibFMG57GJdOyyRvenylDU4iPifY4tYiEGhV6AFXVt7CjvI4dZbVsPVBLYUn7Y2d5HT7XfkxSXDTjh/TjsulDmTI0hSnZqeSmJ+liHxE5bir0o9DY0kZxZQN7KxvYe7CBPQcb2F1R/9Gjoq75o2NjoozhA/owemBfPjtlCOMGJTN2UDI5A5I0fSIiPSLiC905R3VjKxV1zZTXNlFa00RZx8cD1U0cqGnkQHUT+6saOFjf8rGvjY4yhqQmMKx/H86dMJDc9CRyBiSRm57E8AFJxMVEefSqRCQS+VXoZjYHuA+IBhY75+4+ZL917D8PqAeudc69H+CsXXLO0dTqo66plbqmNmqbWqltaqWmsYWaxvaP1Y2tVDW0UFXfQmVDMwfrW6isb/94sK6Z1g/nQzqJMkjvG09mv3iGpCQwbXgqg1MSGdQvgay0RLJSExmUkkBstEpbRIJDt4VuZtHA/cDZwB5gpZktc85t7HTYXGB0x2Mm8EDHx4B7c3MJdz6/kfrmto5HKy1tnyzkQyXERpGSGEtKYiypfeLITU9iap840pLiGJAUR/+kOAb0jSe9bxwZyfH07xNHjMpaREKIPyP0GUChc247gJktBS4AOhf6BcCfnXMOWG5mqWY22Dm3L9CB+yXGMm5QP/rERbc/4mPoGx9DUlw0SfExJCfE0Dc+lr4JMfRLiKFfYizJCTE6a0REwp4/hZ4FFHXa3sMnR99dHZMFfKzQzewG4AaAYcOGHW1WAKYOS2PqVWnH9LUiIuHMnzmFrk7JOHSOw59jcM4tcs7lOefyMjIy/MknIiJ+8qfQ9wBDO21nA8XHcIyIiPQgfwp9JTDazHLNLA64HFh2yDHLgGus3clAVU/Mn4uIyOF1O4funGs1swXAK7Sftviwc26Dmd3YsX8h8CLtpywW0n7a4nU9F1lERLri13nozrkXaS/tzs8t7PS5A74e2GgiInI0dKK1iEiYUKGLiIQJFbqISJiw9ulvD36wWSmwy5MffnzSgTKvQ3ggEl93JL5miMzXHUqvebhzrssLeTwr9FBlZvnOuTyvc/S2SHzdkfiaITJfd7i8Zk25iIiECRW6iEiYUKEfvUVeB/BIJL7uSHzNEJmvOyxes+bQRUTChEboIiJhQoUuIhImVOjHwcy+Z2bOzNK9ztLTzOwXZlZgZuvM7G9mlup1pp5kZnPMbLOZFZrZLV7n6WlmNtTM/mlmm8xsg5l9y+tMvcXMos1stZk973WW46VCP0ZmNpT2+6zu9jpLL3kNmOicmwxsAX7gcZ4e0+k+unOB8cAVZjbe21Q9rhX4rnPuBOBk4OsR8Jo/9C1gk9chAkGFfux+A9xEF3dmCkfOuVedc60dm8tpv4lJuProPrrOuWbgw/vohi3n3D7n3Psdn9fQXnBZ3qbqeWaWDZwPLPY6SyCo0I+BmX0O2OucW+t1Fo98CXjJ6xA96HD3yI0IZpYDnASs8DhKb7iX9oGZz+McAeHXeuiRyMxeBwZ1seuHwK3AOb2bqOcd6TU7557rOOaHtP/v+ZLezNbL/LpHbjgys77A08D/Oueqvc7Tk8xsHlDinFtlZrM9jhMQKvTDcM59pqvnzWwSkAusNTNon3p438xmOOf292LEgDvca/6QmX0RmAec5cL7AoaIvEeumcXSXuZLnHPPeJ2nF5wKfM7MzgMSgH5m9qhz7mqPcx0zXVh0nMxsJ5DnnAuVldqOiZnNAX4NnOGcK/U6T08ysxja3/g9C9hL+311r3TObfA0WA+y9tHJn4AK59z/ehyn13WM0L/nnJvncZTjojl08dfvgGTgNTNbY2YLu/uCUNXx5u+H99HdBPw1nMu8w6nAfODTHf9+13SMXCWEaIQuIhImNEIXEQkTKnQRkTChQhcRCRMqdBGRMKFCFxEJEyp0EZEwoUIXEQkT/x9tyCvXmGvBcQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y = sigmoid(x)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.3 Tanh 函数\n",
    "\n",
    "双曲正切函数有时也被用做神经网络的激活函数，它修正了 Sigmoid 函数不关于原点对称的问题，可是它在两边还是有饱和（也就是导数趋近于0）的问题。\n",
    "\n",
    "$$ tanh(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}} $$\n",
    "\n",
    "$$tanh^{'}(x) =1-tanh^2(x) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tanh(x):\n",
    "    return np.tanh(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAh6klEQVR4nO3deXRc9X338fd3RottecOWbMuLvGFsvIAxwmwNS8DGmMUhaVogITRp6pCG5yRN0qe0eVpyep62OeU0adPQUIcQyEJYnrK4jYO3hBACGMvgfcHGC5a1WF4lW9YyM9/nD12RQUi27BnpamY+r3OGucvvznwHS/rM/d3lZ+6OiIjkrkjYBYiISLgUBCIiOU5BICKS4xQEIiI5TkEgIpLj8sIu4FwUFxf7hAkTwi5DRCSjrFu37pC7l3RcnpFBMGHCBCoqKsIuQ0Qko5jZvs6Wq2tIRCTHKQhERHKcgkBEJMcpCEREcpyCQEQkx6UlCMzsMTM7aGabu1hvZvZdM9tlZhvNbE7SugVmtiNY90A66hERke5L1x7B48CC06y/GZgSPBYD3wcwsyjwcLB+OnCXmU1PU00iItINabmOwN1fMbMJp2myCPixt93z+g0zG2pmpcAEYJe77wYws6eCtlvTUZeIZL94wmmOxWmJJWiOJWiJJWiNJ2iNe/DcNh1LJIgnnFjCiceduDvxRNsjkTTtDgl3EsGzu+NAItG2zKFtmYPT/swH5tu13+a/vU379Pvr8aS2ycuTdBgq4I45Y5lYXJSG/3O/11sXlI0B9ifNVwbLOlt+eWcvYGaLaduboKysrGeqFJFelUg4RxtbqDvRzJETLRxpbOFoYyvHG1uob4pxvLGVE80xGppjnAwep1rjNLbEaWqJ0xSL0xrP/jFVzH4/PWf8eRkbBNbJMj/N8g8vdF8CLAEoLy/P/n95kSyQSDgHjp1i3+FG9h4+yXtHGjlw7BRVx05Rc7yJuoZmYonOf5375UcY3C+fQf3yGFiYR1FhHucNGEBRYZQBBVH65bc9CvMiFOYFz/kR8qMRCvMiFEQj5EUj5EWN/EiEaMTIjxrRSIeHGZH2ZzMiEYhGDKNt2jDMIGptz4ZhkbY/XmYWPP++Hfz+D/cHlgXt26fbJf+RN+vsT2LP660gqATGJc2PBaqAgi6Wi0iGaWqNs72mgY2Vx9hUeZx3ahvYefAEjS3x99sU5EUYM7Q/pUP6cdXkYkYNKaRkYCHFgwoZXlTIsKICzhuQz5AB+RTmRUP8NLmlt4JgKXB/cAzgcuC4u1ebWR0wxcwmAgeAO4G7e6kmEUlBU2ucir1HeWP3Yd7YfZgNlcfe76YZXlTAtNJB/PFl45gyYhATi4uYUDyAkYP6EYmE861XupaWIDCznwPXAcVmVgk8COQDuPsjwDJgIbALaAQ+G6yLmdn9wHIgCjzm7lvSUZOIpN/Rky2s3FbLqq21vLrrEI0tcaIR46KxQ/jcH0zkknFDuWjsUEqH9Autm0POXrrOGrrrDOsd+FIX65bRFhQi0ge1xBL8esdBnnurkl9tP0hr3Ckd0o9PzBnLR6eN4LKJwxhYmJE3MpaA/vVEpFNHT7bw5Jvv8cRreznY0EzxwAI+c+UEPjZ7DDPHDNY3/iyiIBCRD6hraObhX+/iqbXv0dSa4CNTivmnj8/i2gtKyIvqrjTZSEEgIgA0NLXyg1d28+ire2iOJfj4JWP4/EcmMXXUoLBLkx6mIBDJce7OLzfX8ODSLdQ1NHPLRaV8bd4FTCoZGHZp0ksUBCI5rPr4Kf72hS2s2lbLjNGDefQz5Vw8bmjYZUkvUxCI5KhVW2v56jPraYkn+JuF0/jc1RN1DCBHKQhEckwsnuChFTv4z9/sZsbowTx89xwmpPneNZJZFAQiOeR4YyuLf1LBmj1H+NTlZfztrdPpl69bOeQ6BYFIjqg+foo/eWwtew6d5Nt/dDEfnzM27JKkj1AQiOSAXQcb+MwP36S+Kcbjn72Mq84vDrsk6UMUBCJZbmdtA3/0n6+TF43w9BeuYMboIWGXJH2MgkAki713uJFPPbqG/GiEZ75wpQ4KS6cUBCJZqra+iU//cA0t8QRPL1YISNd00rBIFqpvauWeH67h8IlmHv/sXN0mQk5LewQiWSaRcL769AberTvJjz83l9m6UljOQHsEIlnmu7/ayapttfyfWy7kap0dJN2QliAwswVmtsPMdpnZA52s/0szWx88NptZ3MyGBev2mtmmYF1FOuoRyVUrt9byr6t28vE5Y/iTqyaEXY5kiJS7hswsCjwMzKNtkPq1ZrbU3be2t3H3h4CHgva3AX/h7keSXuZ6dz+Uai0iuWz/kUa++vR6Zo0Zwj/eMUsDx0i3pWOPYC6wy913u3sL8BSw6DTt7wJ+nob3FZFAPOF87ZkNAPzHp+bothFyVtIRBGOA/UnzlcGyDzGzAcAC4L+SFjuwwszWmdnirt7EzBabWYWZVdTV1aWhbJHs8dire3hz7xEevH0G44YNCLscyTDpCILO9j+9i7a3Ab/r0C10tbvPAW4GvmRm13S2obsvcfdydy8vKSlJrWKRLPJObQMPLd/B/Okj+cScTr+DiZxWOoKgEhiXND8WqOqi7Z106BZy96rg+SDwPG1dTSLSDa3xBF99Zj2D+uXxjx/XcQE5N+kIgrXAFDObaGYFtP2xX9qxkZkNAa4FXkxaVmRmg9qngfnA5jTUJJITfvS7PWw+UM8/3DGL4oGFYZcjGSrls4bcPWZm9wPLgSjwmLtvMbP7gvWPBE3vAFa4+8mkzUcCzwffYvKAJ939pVRrEskFtfVN/NuqndwwbQQLZo4KuxzJYGm5stjdlwHLOix7pMP848DjHZbtBi5ORw0iueaflm2jNeH83W3Twy5FMpyuLBbJQGt2H+aF9VXcd80kxg/XzeQkNQoCkQwTiyd4cOkWxgztzxevOz/sciQLKAhEMsyz6yrZXtPA/7nlQvoX6MIxSZ2CQCSDNLXG+bdVO7mkbKgOEEvaKAhEMshP39hHTX0Tf3nTVF0zIGmjIBDJECeaY3z/5Xe5+vzhXDVZt5eW9FEQiGSIH726h8MnW/j6/KlhlyJZRkEgkgGON7ay5Le7ufHCkVxSdl7Y5UiWURCIZIAfvbaHhqYYX5t/QdilSBZSEIj0cada4jzx2l5umDaCC0sHh12OZCEFgUgf90zFfo42tnLfdZPDLkWylIJApA+LxRP84Le7mVM2lPLxOjYgPUNBINKHLdtcQ+XRU9x37WRdNyA9RkEg0ke5O4+8/C6TSoq48cKRYZcjWUxBINJHvbrrEFur6/nCNZOIRLQ3ID0nLUFgZgvMbIeZ7TKzBzpZf52ZHTez9cHj77q7rUiueuK1fRQPLOBjl2gcYulZKQ9MY2ZR4GFgHm3jF681s6XuvrVD09+6+63nuK1ITqk82sivttfy59edT2Ge7jAqPSsdewRzgV3uvtvdW4CngEW9sK1I1npyzXsA3HV5WciVSC5IRxCMAfYnzVcGyzq60sw2mNkvzWzGWW4rkjOaY3GeXrufGy4cyZih/cMuR3JAOsYs7uwolneYfwsY7+4nzGwh8AIwpZvbtr2J2WJgMUBZmb4lSfZ6aXMNh0+2cM8V48MuRXJEOvYIKoFxSfNjgarkBu5e7+4ngullQL6ZFXdn26TXWOLu5e5eXlJSkoayRfqmn7y+jwnDB/AH5+tW09I70hEEa4EpZjbRzAqAO4GlyQ3MbJQFV8OY2dzgfQ93Z1uRXLK1qp6KfUf59BXjdcqo9JqUu4bcPWZm9wPLgSjwmLtvMbP7gvWPAH8IfNHMYsAp4E53d6DTbVOtSSRT/fzN9yjMi/CHl44NuxTJIek4RtDe3bOsw7JHkqa/B3yvu9uK5KKm1jgvrj/AgpmjGDqgIOxyJIfoymKRPmLl1lrqm2J88tJxZ24skkYKApE+4tl1lYwe0o+rJg8PuxTJMQoCkT6g+vgpfruzjk9cOlYHiaXXKQhE+oDn3jqAOzpILKFQEIiEzN15tmI/cycOY/zworDLkRykIBAJWcW+o+w93MgntTcgIVEQiITsubcqGVAQZeGs0rBLkRylIBAJUXMszrJNNdw0YxRFhWm5rEfkrCkIREL0yjuHOH6qldtnjw67FMlhCgKREL24/gDDigp0gzkJlYJAJCQnmmOs2lbLLbNKyY/qV1HCo58+kZCs3FpDU2uCReoWkpApCERCsnR9FWOG9mdO2XlhlyI5TkEgEoLDJ5p5Zechbrt4tG4pIaFTEIiEYNnmGuIJV7eQ9AkKApEQ/PeGKqaMGMi0UYPCLkUkPUFgZgvMbIeZ7TKzBzpZ/ykz2xg8XjOzi5PW7TWzTWa23swq0lGPSF92sKGJtXuPcMtFpQQjuIqEKuVLGc0sCjwMzKNtMPq1ZrbU3bcmNdsDXOvuR83sZmAJcHnS+uvd/VCqtYhkguWba3BHt5SQPiMdewRzgV3uvtvdW4CngEXJDdz9NXc/Gsy+AejuWpKzfrGpmvNHDOSCkeoWkr4hHUEwBtifNF8ZLOvKnwK/TJp3YIWZrTOzxV1tZGaLzazCzCrq6upSKlgkLHUNzby554j2BqRPScddrjrr5PROG5pdT1sQ/EHS4qvdvcrMRgArzWy7u7/yoRd0X0JblxLl5eWdvr5IX/fSlhoSDrcoCKQPScceQSWQPNr2WKCqYyMzuwh4FFjk7ofbl7t7VfB8EHietq4mkaz0y03VTCop4oKRA8MuReR96QiCtcAUM5toZgXAncDS5AZmVgY8B9zj7u8kLS8ys0Ht08B8YHMaahLpcw6daOaN3Ye5ZZbOFpK+JeWuIXePmdn9wHIgCjzm7lvM7L5g/SPA3wHDgf8IfgFi7l4OjASeD5blAU+6+0up1iTSFy0PuoV0fED6mrSMhOHuy4BlHZY9kjT9eeDznWy3G7i443KRbPTLTTVMLC7SRWTS5+jKYpFecKyxhdd3H2bBzFHqFpI+R0Eg0gtWbztIPOEsmDEq7FJEPkRBINILXtpSQ+mQflw0dkjYpYh8iIJApIc1tsR45Z06bpqhbiHpmxQEIj3sNzvqaI4lmD9jZNiliHRKQSDSw5ZvqeG8AfnMnTAs7FJEOqUgEOlBLbEEq7cf5MYLR5KnAeqlj9JPpkgPen33YRqaYiyYqbOFpO9SEIj0oJc211BUEOXq84vDLkWkSwoCkR6SSDgrt9Zy3dQR9MuPhl2OSJcUBCI95O39xzh0ollnC0mfpyAQ6SErttaQHzWunzYi7FJETktBINID3J0VW2q5YtJwBvfLD7sckdNSEIj0gHfrTrDn0Enm695CkgEUBCI9YPmWWgDmXajjA9L3KQhEesCKrbVcPG4oo4b0C7sUkTNKSxCY2QIz22Fmu8zsgU7Wm5l9N1i/0czmdHdbkUxTW9/Ehv3HmD9dewOSGVIOAjOLAg8DNwPTgbvMbHqHZjcDU4LHYuD7Z7GtSEZZubWtW+gmnTYqGSIdewRzgV3uvtvdW4CngEUd2iwCfuxt3gCGmllpN7cVySgrttYysbiIySUDwy5FpFvSEQRjgP1J85XBsu606c62AJjZYjOrMLOKurq6lIsW6Qn1Ta28/u4h5k0fqbEHJGOkIwg6+2n3brbpzrZtC92XuHu5u5eXlJScZYkivePlHXW0xl3dQpJR8tLwGpXAuKT5sUBVN9sUdGNbkYyxYksNxQMLmT3uvLBLEem2dOwRrAWmmNlEMysA7gSWdmizFPhMcPbQFcBxd6/u5rYiGaE5FuflHXXMmz6CaETdQpI5Ut4jcPeYmd0PLAeiwGPuvsXM7gvWPwIsAxYCu4BG4LOn2zbVmkTC8Pq7hznRHGP+dF1NLJklHV1DuPsy2v7YJy97JGnagS91d1uRTLRyay1FBVGunDw87FJEzoquLBZJg/axB66dWqKxByTjKAhE0mBD5TEONjSrW0gykoJAJA1WbK0lL2JcP1VjD0jmURCIpMGKLTVcMWk4QwZo7AHJPAoCkRTtOtjAu3UnNSSlZCwFgUiK2sce0PEByVQKApEUvbS5htkae0AymIJAJAUHjp1i04HjLJipvQHJXAoCkRQs31wDwE0am1gymIJAJAXLt9RwwciBTCwuCrsUkXOmIBA5R4dPNLN27xEWaG9AMpyCQOQcrdpWS8JhvoJAMpyCQOQcvbS5hrHn9WfG6MFhlyKSEgWByDmob2rld7sOc9OMURqSUjKegkDkHKzeVktLPMHCWeoWksynIBA5B7/YWMOowf24RENSShZIKQjMbJiZrTSzncHzh34rzGycmf3azLaZ2RYz+3LSum+a2QEzWx88FqZSj0hvaGhq5ZWddSyYOYqIhqSULJDqHsEDwGp3nwKsDuY7igFfc/cLgSuAL5nZ9KT133H32cFDI5VJn/er7QdpiSW45aLSsEsRSYtUg2AR8EQw/QTwsY4N3L3a3d8KphuAbcCYFN9XJDS/2FjNyMGFXFqmbiHJDqkGwUh3r4a2P/jAaUflMLMJwCXAmqTF95vZRjN7rLOupaRtF5tZhZlV1NXVpVi2yLk50Rzj5XfquHlmqbqFJGucMQjMbJWZbe7ksehs3sjMBgL/BXzF3euDxd8HJgOzgWrgX7ra3t2XuHu5u5eXlJSczVuLpE17t9DNusmcZJG8MzVw9xu7WmdmtWZW6u7VZlYKHOyiXT5tIfAzd38u6bVrk9r8APifsylepLct21hNyaBCyicMC7sUkbRJtWtoKXBvMH0v8GLHBtZ2tc0PgW3u/u0O65KPtt0BbE6xHpEec7I5xsvvHOTmmaOIqltIskiqQfAtYJ6Z7QTmBfOY2Wgzaz8D6GrgHuCjnZwm+s9mtsnMNgLXA3+RYj0iPWbVtlqaWhPcMktnC0l2OWPX0Om4+2Hghk6WVwELg+lXgU6/Prn7Pam8v0hvenF9FaOH9OMydQtJltGVxSLdcORkC6+8U8dts0frbCHJOgoCkW5YtqmaWMJZdLEugZHsoyAQ6YalG6qYMmIgF5YOCrsUkbRTEIicQdWxU7y55wiLZo/WLaclKykIRM7gvzdUAXDbxaNDrkSkZygIRM7gxfVVzB43lPHDNUC9ZCcFgchp7KxtYGt1Pbdrb0CymIJA5DSeXVdJXsRYNFtBINlLQSDShdZ4gufeOsBHp41g+MDCsMsR6TEKApEu/GZHHYdONPPJ8nFhlyLSoxQEIl14dt1+igcWcN1U3fZcspuCQKQTh080s3rbQe64ZAz5Uf2aSHbTT7hIJ15YX0Us4fzhpeoWkuynIBDpwN15tmI/F40dwtRRuqWEZD8FgUgHmw/Us72mgU9eOjbsUkR6RUpBYGbDzGylme0MnjsdfN7M9gYD0Kw3s4qz3V6kN/30jX30z49y+2zdaVRyQ6p7BA8Aq919CrA6mO/K9e4+293Lz3F7kR53vLGVFzcc4GOXjGZI//ywyxHpFakGwSLgiWD6CeBjvby9SFr9v7cqaWpN8OkrxoddikivSTUIRrp7NUDwPKKLdg6sMLN1Zrb4HLYX6XGJhPPTN/Yxp2woM0YPCbsckV5zxjGLzWwVMKqTVd84i/e52t2rzGwEsNLMtrv7K2exPUGALAYoKys7m01FuuV37x5iz6GTfPmPZ4ddikivOmMQuPuNXa0zs1ozK3X3ajMrBQ528RpVwfNBM3semAu8AnRr+2DbJcASgPLycj9T3SJn6yev72NYUQE3z+rse49I9kq1a2gpcG8wfS/wYscGZlZkZoPap4H5wObubi/SG6qOnWLVtlr++LJxFOZFwy5HpFelGgTfAuaZ2U5gXjCPmY02s2VBm5HAq2a2AXgT+IW7v3S67UV62+Ov7cXMuHuuuh0l95yxa+h03P0wcEMny6uAhcH0buDis9lepDcdP9XKk2ve45ZZpYwbNiDsckR6na4slpz3szX7ONEcY/E1k8IuRSQUCgLJaU2tcX70u718ZEoxM8folFHJTQoCyWkvvH2AuoZm7rt2ctiliIRGQSA5K55wlryym5ljBnPV5OFhlyMSGgWB5KwVW2rYfegk9107GTMLuxyR0CgIJCfFE863V77DpJIiFszQBWSS2xQEkpOWbjjAzoMn+Oq8C8jTUJSS4/QbIDmnNZ7gOyt3Mr10MAtnloZdjkjoFASSc56p2M97Rxr5+k0XEIno2ICIgkBySlNrnH9fvYs5ZUO5fqruei4CCgLJMY+/tpea+ib+8qZpOlNIJKAgkJxRW9/Ev6/eyUenjeBKXTcg8j4FgeSMf1y2jdaE8+Bt08MuRaRPURBITliz+zAvrq/ivmsmMX54UdjliPQpCgLJerF4ggeXbmHM0P588brzwy5HpM9REEjWe/y1vWyvaeBvb51O/wKNPibSUUpBYGbDzGylme0Mns/rpM1UM1uf9Kg3s68E675pZgeS1i1MpR6RjnYdPMFDy3dww7QR3DRjZNjliPRJqe4RPACsdvcpwOpg/gPcfYe7z3b32cClQCPwfFKT77Svd/dlHbcXOVet8QRffWY9Awqi/NMnZul0UZEupBoEi4AngukngI+dof0NwLvuvi/F9xU5o//49btsrDzOP9wxixGD+oVdjkiflWoQjHT3aoDg+UyXat4J/LzDsvvNbKOZPdZZ11I7M1tsZhVmVlFXV5da1ZL1NlUe599/tZNFs0ezcJbuJyRyOmcMAjNbZWabO3ksOps3MrMC4Hbg2aTF3wcmA7OBauBfutre3Ze4e7m7l5eUlJzNW0uOOdbYwp8/uY7igYX8/e0zwy5HpM/LO1MDd7+xq3VmVmtmpe5ebWalwMHTvNTNwFvuXpv02u9Pm9kPgP/pXtkinYsnnP/187epOd7E01+4kiED8sMuSaTPS7VraClwbzB9L/DiadreRYduoSA82t0BbE6xHslxDy3fwW93HuLvF81kTlmXPY0ikiTVIPgWMM/MdgLzgnnMbLSZvX8GkJkNCNY/12H7fzazTWa2Ebge+IsU65Ec9t8bqnjkN+9y9+Vl3DW3LOxyRDLGGbuGTsfdD9N2JlDH5VXAwqT5RuBDd/ly93tSeX+Rdr/dWcfXntlA+fjz+OZtM8IuRySj6MpiyXjr9h1l8Y/XMamkiB/eexkFefqxFjkb+o2RjLa1qp7P/uhNRg4u5Cd/erkODoucAwWBZKx1+45y96NvUFSYx08/fzklgwrDLkkkIykIJCOt3lbLpx59g6H983l68ZWMPW9A2CWJZKyUDhaLhOGpN9/jGy9sZsbowTz2J5dRPFB7AiKpUBBIxmhqjfPgi1t4umI/H5lSzCOfvpSiQv0Ii6RKv0WSEfYcOskXf7qO7TUN3H/9+XzlxinkRdWzKZIOCgLp02LxBI+/tpd/WfEO/fIj/Oizl3H91DPd21BEzoaCQPqsLVXH+evnNrGx8jg3TBvB/71jJqVD+oddlkjWURBIn7P/SCP/umonz71dyfCiAr539yXcMqtUA8uI9BAFgfQZ+4808sNX9/DkmvfA4M8+Mok/v24yQwcUhF2aSFZTEEio3J2KfUd57NU9LN9SQ8SMT8wZy1fmTVE3kEgvURBIKPYfaeSFtw/w3NsH2HPoJIP75bH4msnce9V4BYBIL1MQSK9IJJwtVfWs2lbL6u21bD5QD8DlE4dx37WTuO3i0Qwo0I+jSBj0myc9oqk1zrbqet5+7xhv7D7Mmj1HOH6qFTOYU3Yef7VgGrdeVMq4Ybo1hEjYFASSknjCqTp2infrTrCz9gQ7ahvYXlPP9uoGYgkHYNyw/syfPpIrJg3n2qkluiWESB+TUhCY2SeBbwIXAnPdvaKLdguAfwOiwKPu3j6S2TDgaWACsBf4I3c/mkpNkj5NrXGONrZwqKGFuhNN1DU0U3O8merjpzhw7BQHjp5i/9FGWuP+/jbFAwuZNmoQf3bNJC4eO4SLxg5l9FD1+Yv0ZanuEWwGPg78Z1cNzCwKPEzbUJWVwFozW+ruW4EHgNXu/i0zeyCY/6sUa8o67k7C2759J9yJJ5xYwonFE8QTTmsw3Rp3WuMJWuMJWmJtj+Zguqk1TnNrgqZYnFMtcRpb4pxqjXOiOcbJ4FHfFKP+VCv1p1o5dqqVxpZ4p/UUDyxkzNB+TB01iJtmjmL8sAFMKC7igpGDGFakUz1FMk2qQ1VuA850oc9cYJe77w7aPgUsArYGz9cF7Z4AXqYHg+C7q3eydEPV+/Pu3mk772KmfdLdP9Cm/WUcxz1pPmjnnrSuff79dW3PiUTb+oQ78eAPfyL4w5/ovMyU9cuPMLAwj6LCPIoK8hjcP4+yYQMY3D+fIf3zGVZUwPCiAs4rKmDEoEJKgkdhXrRnChKRUPTGMYIxwP6k+Urg8mB6pLtXA7h7tZl1eRMZM1sMLAYoKzu3gclHDCpk6shBHV64i/f74Ht/aLlZ520s+I9h77ex5PmgQcQ+uDxiRsTapqMRC+bbps2MqBnRCEQibdN50Qh5ESMaMfKjRl4kQl7UKIhGKMiLkBeNUBCNUJjf9twvP0q//Lbn/sEjEtGVuiLSjSAws1XAqE5WfcPdX+zGe3T21+asv+O6+xJgCUB5efk5fUe+c24Zd849txAREclWZwwCd78xxfeoBMYlzY8F2vtnas2sNNgbKAUOpvheIiJylnrjhu5rgSlmNtHMCoA7gaXBuqXAvcH0vUB39jBERCSNUgoCM7vDzCqBK4FfmNnyYPloM1sG4O4x4H5gObANeMbdtwQv8S1gnpntpO2som+lUo+IiJw96+rMmb6svLzcKyo6vWRBRES6YGbr3L2843KN9ScikuMUBCIiOU5BICKS4xQEIiI5LiMPFptZHbAv7DrOQTFwKOwielkufmbIzc+di58ZMutzj3f3ko4LMzIIMpWZVXR2xD6b5eJnhtz83Ln4mSE7Pre6hkREcpyCQEQkxykIeteSsAsIQS5+ZsjNz52Lnxmy4HPrGIGISI7THoGISI5TEIiI5DgFQUjM7Otm5mZWHHYtPc3MHjKz7Wa20cyeN7OhYdfUU8xsgZntMLNdwTjcWc/MxpnZr81sm5ltMbMvh11TbzGzqJm9bWb/E3YtqVAQhMDMxtF22+33wq6ll6wEZrr7RcA7wF+HXE+PMLMo8DBwMzAduMvMpodbVa+IAV9z9wuBK4Av5cjnBvgybbfXz2gKgnB8B/jfnMOQnZnI3VcE41IAvEHbKHXZaC6wy913u3sL8BSwKOSaepy7V7v7W8F0A21/GMeEW1XPM7OxwC3Ao2HXkioFQS8zs9uBA+6+IexaQvI54JdhF9FDxgD7k+YryYE/iMnMbAJwCbAm5FJ6w7/S9oUuEXIdKTvjmMVy9sxsFTCqk1XfAP4GmN+7FfW8031md38xaPMN2roRftabtfUi62RZTuz1AZjZQOC/gK+4e33Y9fQkM7sVOOju68zsupDLSZmCoAe4+42dLTezWcBEYIOZQVsXyVtmNtfda3qxxLTr6jO3M7N7gVuBGzx7L16pBMYlzY8FqkKqpVeZWT5tIfAzd38u7Hp6wdXA7Wa2EOgHDDazn7r7p0Ou65zogrIQmdleoNzdM+XOhefEzBYA3waudfe6sOvpKWaWR9vB8BuAA8Ba4O6kMbqzkrV9q3kCOOLuXwm5nF4X7BF83d1vDbmUc6ZjBNIbvgcMAlaa2XozeyTsgnpCcED8fmA5bQdMn8n2EAhcDdwDfDT4910ffFOWDKE9AhGRHKc9AhGRHKcgEBHJcQoCEZEcpyAQEclxCgIRkRynIBARyXEKAhGRHPf/AYnmb+KNODcwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y = tanh(x)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.4 ReLU 函数\n",
    "\n",
    "在神经网络发展的历史上，sigmoid 函数很早就开始被使用了，而最近则主要使用 ReLU（Rectified Linear Unit）函数。ReLU函数可以表示为下面的式：\n",
    "\n",
    "$$ h(x) = \\begin{cases}\n",
    "x&x > 0 \\\\\n",
    "0&x <= 0\n",
    "\\end{cases} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU函数的实现也很简单，可以写成如下形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(x):\n",
    "    return np.maximum(0, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWt0lEQVR4nO3deXiU9bnG8fsxgMgmKhGRxaAissgSIlBRW5cq7qeLlkXPabWlh8Widak92tq9vVq3VsAeTrW1JYC41dZqFRdqXWsWQDCA7DsJsq/ZnvNHEho1mElm3nnfmfl+rovLQIaZZyDe8+PJzNzm7gIARNcRYQ8AAPh0BDUARBxBDQARR1ADQMQR1AAQcS2CuNJOnTp5Tk5OEFcNAGmpsLBwq7tnN/S5QII6JydHBQUFQVw1AKQlM1tzuM+x+gCAiIvpRG1mqyXtllQlqdLd84IcCgDwb01ZfZzn7lsDmwQA0CBWHwAQcbEGtUt60cwKzWxcQxcws3FmVmBmBWVlZYmbEAAyXKxBPcLdcyVdImmimZ378Qu4+3R3z3P3vOzsBp9hAgBohpiC2t031v63VNLTkoYGORQA4N8aDWoza2tm7es+lnSRpEVBDwYAqeRfq7bp4ddXKYi3jo7lWR+dJT1tZnWXn+nuf0/4JACQosp2H9SkmUVqe2QLjR7aXW1aJfa1hI1em7uvlDQwobcKAGmiqto1eXaxdu6v0KPXD014SEsBvYQcADLFAy8t05srPtQvvzxAfbp0COQ2eB41ADTTvKWlevCV5bp6SDddk9c9sNshqAGgGTbu2K+bH5uv009orx9d1T/Q2yKoAaCJyiurNXFmkSqqXNPG5uqoVlmB3h47agBool88v0TFa3do6phcnZzdLvDb40QNAE3w/Hub9Mgbq/TVs3J02YAuSblNghoAYrRq617d9sRCDezeUf9zaZ+k3S5BDQAxOFBRpfEzCpV1hGnqmMFq1SJ58cmOGgBi8IO/LNaSzbv1yFfz1O2YNkm9bU7UANCIJwvXa/a76zThc6fo/NM7J/32CWoA+BRLNu/SnX9+T8N6Hqtvf/60UGYgqAHgMPYcrNSE/CK1b91SD44ZrBZZ4UQmQQ0ADXB33fHkQq3eule/GTVYx7dvHdosBDUANOBPb6/Rsws36ZaLeuszpxwX6iwENQB8zIJ1O/TjZ9/Xeb2zNf6zp4Q9DkENAPXt2FeuCflFOr59a913zSAdcYSFPRLPowaAOtXVrlvmLFDp7gN6/L/P0jFtW4U9kiRO1ABwyP++tlIvLynVnZf20aDuHcMe5xCCGgAkvb3yQ93z4lJddkYX/ddZOWGP8xEENYCMV7r7gG6cVayTjm2jX3zpDNWWeUcGO2oAGa2q2jV51nztPlChP90wVO1btwx7pE8gqAFktPvnLtNbKz/Ur748QKefEEw5bbxYfQDIWK8uLdWUV5frmrxuujrActp4EdQAMtKGJJbTxougBpBxyiurNTG/SJVVroeuHaLWLYMtp40XO2oAGefnz5do/rodmjY2Vz07tQ17nEZxogaQUZ57b5N+/8ZqffWsHF16RnLKaeNFUAPIGKu27tXtTyzUoCSX08aLoAaQEerKaVtkmaaOzU1qOW282FEDyAh3P1NTTvv7r52prh2PCnucJkmdhxQAaKYnCtfrsYJ1mnjeKTqv9/Fhj9NkMQe1mWWZWbGZPRvkQACQSEs279Jdf35Pw08+VjdfGE45bbyacqKeLKkkqEEAINHql9P+ZnR45bTximlqM+sm6TJJvwt2HABIjPrltA+ODrecNl6xPrw8IOl2SdXBjQIAiVNXTnvbxadr+MnhltPGq9GgNrPLJZW6e2EjlxtnZgVmVlBWVpawAQGgqebXltNecPrx+ua5J4c9TtxiOVGPkHSlma2WNFvS+WY24+MXcvfp7p7n7nnZ2dkJHhMAYrNjX7km1pbT3nvNwEiU08ar0aB29++6ezd3z5E0StIr7n5t4JMBQBNVV7u+PWeBynYf1LSxuerYJhrltPFKzW+BAkADfvvaCr2ypFR3Xd5HAyNUThuvJr0y0d3nSZoXyCQAEIe3V36oe15YqssHdNF1w08Ke5yE4kQNIOXVldPmdGqrX3xpQOTKaePFe30ASGmVVdWHymln3DBM7Y5Mv1hLv3sEIKPc/1JNOe09Vw9U7xPahz1OIFh9AEhZry4p1dRXV+gred315SHdwh4nMAQ1gJS0Ycd+3Txnvvp06aAfXtUv7HECRVADSDn1y2mnjc2NfDltvNhRA0g5P3uuppz2oRQpp40XJ2oAKeVvCzfpD2+u1tdG5OiSFCmnjRdBDSBlrCzbo+88uVCDe3TUdy9JnXLaeBHUAFLCgYoqTcgvUsss09QxqVVOGy921ABSwvefWaSlW3br9189UyemWDltvDLnIQlAynq8YJ3mFKzXpPNO1edSsJw2XgQ1gEhbsnmXvvfMIp11ynG6KUXLaeNFUAOIrN0HKjR+RpE6tG6pX48arKw0KAFoDnbUACLJ3XXHU+9p7bZ9mvn1Ycpuf2TYI4WGEzWASHr0zdX628JNuvWi3hqW4uW08SKoAURO8drt+ulzJWlTThsvghpApGzfW65JM4vTqpw2XuyoAURGTTntfJXtPqjH//szaVNOGy9O1AAi46F/rNCrS8vSrpw2XgQ1gEh4a8WHuvfFpbpi4IlpV04bL4IaQOjql9P+/ItnpF05bbzYUQMIVWVVtb41q1h7DlYo/+vpWU4bL/5EAITq/peW6e2V23RvGpfTxovVB4DQ1JXTjjqzu76UxuW08SKoAYSirpy2b5cO+sGV6V1OGy+CGkDS1ZXTVmVIOW282FEDSLr65bQ5GVBOGy9O1ACSqq6c9voRPTOmnDZeBDWApKkrp83t0VF3XHJ62OOkDIIaQFLsL/93Oe2UDCunjRc7agBJkcnltPFq9CHNzFqb2b/MbIGZLTazHyZjMADpY8676/R44XrdmKHltPGK5UR9UNL57r7HzFpKet3Mnnf3twOeDUAaKNlUU0474tTjNDlDy2nj1WhQu7tL2lP705a1PzzIoQCkh90HKjQhv0hHH9VSD3wlc8tp4xXTNt/MssxsvqRSSXPd/Z0GLjPOzArMrKCsrCzBYwJINe6u7zy5UGu37dOUMbkZXU4br5iC2t2r3H2QpG6ShppZ/wYuM93d89w9Lzs7O8FjAkg1f3hztZ57b7Nuu7i3hvY8NuxxUlqTnh/j7jskzZM0MohhAKSH4rXb9bPnSnRhn+M17hzKaeMVy7M+ss2sY+3HR0m6UNKSgOcCkKLqymk7d2ite68eRDltAsTyrI8ukh41syzVBPscd3822LEApKLqatfNteW0T4z/jI5u0zLskdJCLM/6WChpcBJmAZDiHvrHCs1bWqYfX9VPA7p1DHuctMFrOAEkxJsrth4qp72WctqEIqgBxK101wF9a9Z8ymkDwnt9AIhLZVW1bqScNlD8iQKIy31zl+mdVdt03zWU0waF1QeAZntlyRZNm7dCo4d21xdzKacNCkENoFnWb9+nmx9boL5dOujuKyinDRJBDaDJDlZWaWJ+kaqrKadNBnbUAJrsZ38r0YL1O/XbaymnTQZO1ACa5K8LNurRt9bohrN7amR/ymmTgaAGELMVZXt0B+W0SUdQA4jJ/vIqTZhRpFYtjtCUMblqmUV8JAs7agAx+d4zi7SsdLce/dpQymmTjIdEAI2a8+46PVG4Xjee30vnnkYxSLIR1AA+1fsba8ppzz61kyZf0CvscTISQQ3gsHYfqNDEmUXq2KalHhg1iHLakLCjBtCg+uW0s8cNV6d2lNOGhRM1gAb9/o2actrbL+6tM3Mopw0TQQ3gE4rWbtfPny/RhX06a9y5lNOGjaAG8BHb95ZrUn5RbTntQEoAIoAdNYBD6sppt+4p15Pjz6KcNiI4UQM4ZNq85Zq3tEzfu6Kvzuh2dNjjoBZBDUBSTTntfXOX6cqBJ+raYT3CHgf1ENQADpXT9qScNpLYUQMZrrKqWpNmFWvvwUrN/MYwtaWcNnL4GwEy3L1zl+lfteW0p3WmnDaKWH0AGezlki16aN4KjR7ag3LaCCOogQy1bts+fXvOAvU7sYPuvqJv2OPgUxDUQAY6WFmlSTOLVO2U06YCdtRABvrpoXLaITrpOMppo44TNZBh/rpgo/741hp9/eyeGtn/hLDHQQwIaiCD1JXTDjnpGH2HctqU0WhQm1l3M3vVzErMbLGZTU7GYAASa195pcbPKNSRLbM0ZcxgymlTSCw76kpJt7h7kZm1l1RoZnPd/f2AZwOQIO6uu/68SB+U7tGjXxuqLkdTTptKGn1IdfdN7l5U+/FuSSWSugY9GIDEeezddXqqaAPltCmqSf/2MbMcSYMlvdPA58aZWYGZFZSVlSVoPADxWrxxp77/l8WU06awmIPazNpJelLSTe6+6+Ofd/fp7p7n7nnZ2TxiA1Gw60CFJuQX6RjKaVNaTM+jNrOWqgnpfHd/KtiRACSCu+v2xxdq/fb9lNOmuFie9WGSHpZU4u73BT8SgER45I3V+vvizfrOSMppU10sq48Rkq6TdL6Zza/9cWnAcwGIQ+Ga7fr5cyX6fN/O+sY5lNOmukZXH+7+uiQWW0CK2La3XJNmFqlLx9a6h3LatMB7fQBppLraddNj8/VhXTntUZTTpgNemgSkkamvLtdry8r0fcpp0wpBDaSJN5dv1f0vLdNVg07UWMpp0wpBDaSBLbsO6Fuzi3Vydjv97AuU06YbdtRAiqusqtaNs4q192CVZn0jl3LaNMTfKJDi7nmxppz2/q8MVC/KadMSqw8ghb1cskW//UdNOe0XBlNOm64IaiBFrdu2Tzc/Nl/9u1JOm+4IaiAFHays0sSZRXJJ08YMoZw2zbGjBlLQT54t0cL1OzX9uiHqcVybsMdBwDhRAynmLws26k9vr9E3zumpi/pRTpsJCGoghSwvrSmnzTvpGN0+knLaTEFQAyliX3mlJuQXqnXLLD1IOW1GYUcNpID65bR/vJ5y2kzDQzKQAurKaSdf0Evn9KLqLtMQ1EDE1ZXTntOrk248n3LaTERQAxFWV057bJtWeuArlNNmKnbUQETVL6d9bNxwHUc5bcbiRA1E1MOvr9LfF2/WHSNPVx7ltBmNoAYiqHDNNv3i+SW6qG9nff2cnmGPg5AR1EDE1JTTFuvEjkfpV5TTQuyogUg5VE67t1xPUU6LWpyogQiZUltO+4Mr+ql/V8ppUYOgBiLijdpy2i8M7qrRQ7uHPQ4ihKAGImDLrgOaPLtYp2a300+/0J+9ND6CHTUQsoqqak2aWaR95VWaPS5XbVrxvyU+iq8IIGT3vLBU767erl+PGqRTj6ecFp/E6gMI0dz3t+h/X1upscN66KpBXcMeBxFFUAMhWbdtn26ZU1NO+73LKafF4RHUQAgOVFRpQj7ltIgNO2ogBD/52/t6bwPltIhNoydqM3vEzErNbFEyBgLS3TPzN2jG22s17tyTKadFTGJZffxB0siA5wAywvLS3fruU+/pzJxjdNvFvcMeBymi0aB299ckbUvCLEBa21deqfEzinRUyyw9ODqXclrEjB01kATurrueXqTlZXv0p+uH6YSjW4c9ElJIwh7SzWycmRWYWUFZWVmirhZIC7PfXaenimvKac/u1SnscZBiEhbU7j7d3fPcPS87m5ZkoM6iDTt1d2057bcop0UzsCQDArTrQIUmzvx3Oe0RlNOiGWJ5et4sSW9J6m1m683shuDHAlKfu+u2xxdow/b9mjp2MOW0aLZGv5no7qOTMQiQbh5+fZVeWLxFd13WR0NOopwWzcfqAwhAXTntxf0664azKadFfAhqIME+3HNQE/OL1fWYo/TLL1NOi/jxPGoggapqy2m37SvX0xMop0VicKIGEmjKK8v1zw+26odX9lO/EymnRWIQ1ECCvP7BVj3w8jJ9cXBXjTqTclokDkENJMDmnTXltL2Ob6efUE6LBCOogThVVFXrxllF2l9RpWljKadF4vEVBcSJcloEjRM1EIe6ctprh1NOi+AQ1EAz1ZXTntH1aMppESiCGmiGunJaSZo2NldHtqCcFsFhRw00Q1057f/9Z566H0s5LYLFiRpoorpy2m+ee7I+37dz2OMgAxDUQBPUL6e9lXJaJAlBDcSorpy2TassTRlDOS2Shx01EAN315215bQzbhimzh0op0XycCQAYjDrX+v0dPEG3XzhaRpxKuW0SC6CGmjEog079YO/Lta5p2Vr0nmnhj0OMhBBDXyKnfsrNCG/SMe1pZwW4WFHDRxGXTntxh379dg3h+vYtq3CHgkZihM1cBgPv75KL76/RXdccjrltAgVQQ00oK6cdmS/EyinRegIauBjPlJOe/UASgAQOnbUQD31y2mfGn+WOrSmnBbh40QN1PPgKx8cKqft35VyWkQDQQ3U+ucHZfr1yx/oi7mU0yJaCGpA0qad+3XT7Pk15bT/QTktooWgRsarqKrWpJnFteW0QyinReTwFYmM98u/L1Hhmu36zejBOvX4dmGPA3wCJ2pktBcWb9b//XOVrht+kq4ceGLY4wANIqiRsdZ+uE+3Pr5AA7odrbsu7xP2OMBhEdTISAcqqjQ+v1AmaeoYymkRbTEFtZmNNLOlZrbczO4IeiggaD969n0t3rhL910ziHJaRF6jQW1mWZKmSrpEUl9Jo82sb9CDAUH5c/EGzXxnrb752ZN1IeW0SAGxPOtjqKTl7r5SksxstqSrJL2f6GGuePB1HaioSvTVAh+xZts+Dc05VrddRDktUkMsQd1V0rp6P18vadjHL2Rm4ySNk6QePXo0a5hTstuqvKq6Wb8XiFVuj2N0y0WnqQXltEgRsQR1Qy/R8k/8gvt0SdMlKS8v7xOfj8UDowY357cBQFqL5UixXlL9Nz7oJmljMOMAAD4ulqB+V1IvM+tpZq0kjZL0l2DHAgDUaXT14e6VZjZJ0guSsiQ94u6LA58MACApxvf6cPfnJD0X8CwAgAbwbW8AiDiCGgAijqAGgIgjqAEg4sy9Wa9N+fQrNSuTtCbhVxy8TpK2hj1EkmXifZYy835zn6PtJHfPbugTgQR1qjKzAnfPC3uOZMrE+yxl5v3mPqcuVh8AEHEENQBEHEH9UdPDHiAEmXifpcy839znFMWOGgAijhM1AEQcQQ0AEUdQN8DMbjUzN7NOYc+SDGb2KzNbYmYLzexpM+sY9kxBycSiZjPrbmavmlmJmS02s8lhz5QsZpZlZsVm9mzYs8SDoP4YM+su6fOS1oY9SxLNldTf3QdIWibpuyHPE4gMLmqulHSLu/eRNFzSxAy535I0WVJJ2EPEi6D+pPsl3a4G6sbSlbu/6O6VtT99WzUtPunoUFGzu5dLqitqTmvuvsndi2o/3q2a4Ooa7lTBM7Nuki6T9LuwZ4kXQV2PmV0paYO7Lwh7lhBdL+n5sIcISENFzWkfWPWZWY6kwZLeCXmUZHhANYeulG/Mjqk4IJ2Y2UuSTmjgU3dK+h9JFyV3ouT4tPvt7s/UXuZO1fwzOT+ZsyVRTEXN6crM2kl6UtJN7r4r7HmCZGaXSyp190Iz+1zI48Qt44La3S9s6NfN7AxJPSUtMDOp5p//RWY21N03J3HEQBzuftcxs/+SdLmkCzx9n1yfsUXNZtZSNSGd7+5PhT1PEoyQdKWZXSqptaQOZjbD3a8Nea5m4QUvh2FmqyXluXuqvPNWs5nZSEn3Sfqsu5eFPU9QzKyFar5ZeoGkDaopbh6T7h2gVnPyeFTSNne/KeRxkq72RH2ru18e8ijNxo4akjRFUntJc81svpn9NuyBglD7DdO6ouYSSXPSPaRrjZB0naTza/9+59eeNJEiOFEDQMRxogaAiCOoASDiCGoAiDiCGgAijqAGgIgjqAEg4ghqAIi4/wf8nTrg8pIQwAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y = relu(x)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 神经网络的实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们来实现一个简单 3 层神经网络从输入到输出的（前向）处理。先看一下从输入层到第 1 层的第 1 个神经元的信号传递过程，图中增加了表示偏置的神经元“1”。\n",
    "![img](images/chapter09/layer1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中，$ w_{12}^{(1)} $ 表示前一层的第 2 个神经元 $ x2 $ 到后一层的第 1 个神经元 $ a_1^{(1)}$ 的权重。因此：\n",
    "\n",
    "$$ a_1^{(1)} = w_{11}^{(1)} x_1 + w_{12}^{(1)} x_2 + b_1^{(1)} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果使用矩阵的乘法运算，则可以将第 1 层的加权和表示成：\n",
    "$$ A^{(1)} = XW^{(1)} + B^{(1)} $$\n",
    "\n",
    "其中，$A^{(1)}$、$X$、$B^{(1)}$、$W^{(1)}$如下所示：\n",
    "\n",
    "$$ A^{(1)} = \\bigl( \\begin{matrix} a_1^{(1)} & a_2^{(1)} & a_3^{(1)}\\end{matrix} \\bigr) $$\n",
    "\n",
    "$$ X = \\bigl( \\begin{matrix}x_1 & x_2 \\end{matrix} \\bigr)$$\n",
    "\n",
    "$$ B^{(1)} = \\bigl( \\begin{matrix}b_1^{(1)} & b_2^{(1)}&  b_3^{(1)}\\end{matrix} \\bigr) $$\n",
    "\n",
    "$$ W^{(1)} =  \\bigl( \\begin{matrix}\n",
    "w_{11}^{(1)}&w_{21}^{(1)}&w_{31}^{(1)}\\\\\n",
    "w_{12}^{(1)}&w_{22}^{(1)}&w_{32}^{(1)}\n",
    "\\end{matrix}\\bigr) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们用 NumPy 多维数组来实现，这里将输入信号、权重、偏置设置成任意值。$X$ 是元素个数为 2 的一维数组，$W_1$是2 × 3 的数组。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2,)\n",
      "(2, 3)\n",
      "(3,)\n"
     ]
    }
   ],
   "source": [
    "X = np.array([1.0, 0.5])\n",
    "W1 = np.array([[0.1, 0.3, 0.5], [0.2, 0.4, 0.6]])\n",
    "B1 = np.array([0.1, 0.2, 0.3])\n",
    "print(X.shape)  # (2,)\n",
    "print(W1.shape) # (2, 3)\n",
    "print(B1.shape) # (3,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来实现第 1 层中激活函数的计算过程：$Z^{(1)} = \\sigma(A^{(1)})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.3 0.7 1.1]\n",
      "[0.57444252 0.66818777 0.75026011]\n"
     ]
    }
   ],
   "source": [
    "A1 = np.dot(X, W1) + B1\n",
    "print(A1)\n",
    "Z1 = sigmoid(A1)\n",
    "print(Z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面，我们来实现第 1 层到第 2 层的信号传递，其中第 1 层的输出（$Z^{(1)}$）变成了第 2 层的输入。\n",
    "![img](images/chapter09/layer2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3,)\n",
      "(3, 2)\n",
      "(2,)\n",
      "[0.62624937 0.7710107 ]\n"
     ]
    }
   ],
   "source": [
    "W2 = np.array([[0.1, 0.4], [0.2, 0.5], [0.3, 0.6]])\n",
    "B2 = np.array([0.1, 0.2])\n",
    "print(Z1.shape) # (3,)\n",
    "print(W2.shape) # (3, 2)\n",
    "print(B2.shape) # (2,)\n",
    "A2 = np.dot(Z1, W2) + B2\n",
    "Z2 = sigmoid(A2)\n",
    "print(Z2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后是第 2 层到输出层的信号传递。输出层的实现也和之前的实现基本相同。不过，最后的激活函数和之前的隐藏层有所不同。这里定义了 identity_function() 函数（也称为“恒等函数”），并将其作为输出层的激活函数。\n",
    "\n",
    "![img](images/chapter09/layer3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.31682708 0.69627909]\n"
     ]
    }
   ],
   "source": [
    "def identity_function(x):\n",
    "    return x\n",
    "\n",
    "W3 = np.array([[0.1, 0.3], [0.2, 0.4]])\n",
    "B3 = np.array([0.1, 0.2])\n",
    "A3 = np.dot(Z2, W3) + B3\n",
    "Y = identity_function(A3)\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 前向传播(forward propagation)代码小结\n",
    "\n",
    "现在我们把之前的代码实现全部整理一下。这里，我们按照神经网络的实现惯例，只把权重记为大写字母 $W_1$，$W_2$，$W_3$，其他的（偏置或中间结果等）都用小写字母表示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.31682708 0.69627909]\n"
     ]
    }
   ],
   "source": [
    "def init_network():\n",
    "    network = {}\n",
    "    network['W1'] = np.array([[0.1, 0.3, 0.5], [0.2, 0.4, 0.6]])\n",
    "    network['b1'] = np.array([0.1, 0.2, 0.3])\n",
    "    network['W2'] = np.array([[0.1, 0.4], [0.2, 0.5], [0.3, 0.6]])\n",
    "    network['b2'] = np.array([0.1, 0.2])\n",
    "    network['W3'] = np.array([[0.1, 0.3], [0.2, 0.4]])\n",
    "    network['b3'] = np.array([0.1, 0.2])\n",
    "    return network\n",
    "\n",
    "def forward(network, x):\n",
    "    W1, W2, W3 = network['W1'], network['W2'], network['W3']\n",
    "    b1, b2, b3 = network['b1'], network['b2'], network['b3']\n",
    "    a1 = np.dot(x, W1) + b1\n",
    "    z1 = sigmoid(a1)\n",
    "    a2 = np.dot(z1, W2) + b2\n",
    "    z2 = sigmoid(a2)\n",
    "    a3 = np.dot(z2, W3) + b3\n",
    "    y = identity_function(a3)\n",
    "    return y\n",
    "\n",
    "network = init_network()\n",
    "x = np.array([1.0, 0.5])\n",
    "y = forward(network, x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里定义了 init_network() 和 forward() 函数。init_network() 函数会进行权重和偏置的初始化，并将它们保存在字典变量 network 中。这个字典变量 network 中保存了每一层所需的参数（权重和偏置）。forward() 函数中则封装了将输入信号转换为输出信号的处理过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. 输出层的设计\n",
    "\n",
    "神经网络可以用在分类问题和回归问题上，需要根据具体任务改变输出层的激活函数。一般而言，回归问题用恒等函数，二分类问题使用 sigmoid 函数，多分类问题用 softmax 函数。\n",
    "\n",
    "分类问题中使用的 softmax 函数可以用下式表示，其中假设输出层共有 n 个神经元，计算第 k 个神经元的输出值 $y_k$。神经网络把输出值最大的神经元所对应的类别作为识别结果。\n",
    "\n",
    "$$ y_k = \\frac{exp(a_k)}{\\sum_{i=1}^n exp(a_i)} $$\n",
    "\n",
    "$$ \\sum_{i=1}^n y_i = 1 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "softmax 函数的输出是 0.0 到 1.0 之间的实数。并且，softmax 函数的输出值的总和为 1。即便使用了softmax 函数，各个元素之间的大小关系也不会改变，这是因为指数函数 $y = exp(x)$ 是单调递增函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 softmax 函数的实现\n",
    "\n",
    "现在我们来实现 softmax 函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.34985881 18.17414537 54.59815003]\n"
     ]
    }
   ],
   "source": [
    "a = np.array([0.3, 2.9, 4.0])\n",
    "exp_a = np.exp(a)               # 指数函数\n",
    "print(exp_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "74.1221542101633\n"
     ]
    }
   ],
   "source": [
    "sum_exp_a = np.sum(exp_a)       # 指数函数的和\n",
    "print(sum_exp_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.01821127 0.24519181 0.73659691]\n"
     ]
    }
   ],
   "source": [
    "y = exp_a / sum_exp_a\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将其写成函数形式如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(a):\n",
    "    exp_a = np.exp(a)\n",
    "    return exp_a / np.sum(exp_a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的 softmax 函数的实现虽然正确，但在计算机的运算上有一定的缺陷。这个缺陷就是指数函数运算数值溢出问题。例如，$ e^{100}$ 等于2.69e+43，$e^{1000}$ 的结果会返回一个表示无穷大的inf。如果在这些超大值之间进行除法运算，结果会出现“不确定”的情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Devon\\AppData\\Local\\Temp/ipykernel_7284/2006519253.py:1: RuntimeWarning: overflow encountered in exp\n",
      "  np.exp(1000)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "inf"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Devon\\AppData\\Local\\Temp/ipykernel_7284/3634570022.py:2: RuntimeWarning: overflow encountered in exp\n",
      "  exp_a = np.exp(a)\n",
      "C:\\Users\\Devon\\AppData\\Local\\Temp/ipykernel_7284/3634570022.py:3: RuntimeWarning: invalid value encountered in true_divide\n",
      "  return exp_a / np.sum(exp_a)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([nan, nan, nan])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([1010, 1000, 990])\n",
    "softmax(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了防止溢出，softmax 函数的实现可以下式这样进行修改，\n",
    "\n",
    "$$ \n",
    "\\begin{aligned}\n",
    "y_k &= \\frac{exp(a_k)}{\\sum_{i=1}^n exp(a_i)} \\\\\n",
    "&= \\frac{exp(a_k - C')}{\\sum_{i=1}^n exp(a_i - C')}\n",
    "\\end{aligned}$$\n",
    "\n",
    "我们可以令 $ C'$ 为输入信号中的最大值，通过对所有输入信号减去输入信号中的最大值来改进 softmax 函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1010\n",
      "[  0 -10 -20]\n"
     ]
    }
   ],
   "source": [
    "a = np.array([1010, 1000, 990])\n",
    "print(np.max(a))\n",
    "a = a - np.max(a)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9.99954600e-01, 4.53978686e-05, 2.06106005e-09])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(a) / np.sum(np.exp(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "综上，我们可以像下面这样实现 softmax 函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(a):\n",
    "    exp_a = np.exp(a - np.max(a)) # 溢出对策\n",
    "    return exp_a / np.sum(exp_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9.99954600e-01, 4.53978686e-05, 2.06106005e-09])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([1010, 1000, 990])\n",
    "softmax(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 输出层的神经元数量\n",
    "\n",
    "输出层的神经元数量需要根据待解决的问题来决定。对于分类问题，输出层的神经元数量一般设定为类别的数量。比如，对于手写数字图像识别问题，预测是图中的数字 0 到 9 中的哪一个的问题（10 类别分类问题），可以将输出层的神经元设定为 10 个。\n",
    "\n",
    "![img](images/chapter09/hand_writing_number.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. 手写数字识别\n",
    "\n",
    "现在我们来试着解决实际问题——进行手写数字图像的分类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 [MNIST 数据集](http://yann.lecun.com/exdb/mnist/)\n",
    "\n",
    "这里使用的数据集是 MNIST 手写数字图像集。MNIST 是机器学习领域最有名的数据集之一，被应用于从简单的实验到发表的论文研究等各种场合。实际上，在阅读图像识别或机器学习的论文时，MNIST 数据集经常作为实验用的数据出现。\n",
    "\n",
    "MNIST 数据集是由 0 到 9 的数字图像构成的。训练集包含 6 万张图像，测试集包含 1 万张图像，这些图像可以用于学习和推理。MNIST 数据集的一般使用方法是，先用训练图像进行学习，再用学习到的模型度量能在多大程度上对测试图像进行正确的分类。\n",
    "![img](images/chapter09/mnist.png)\n",
    "\n",
    "MNIST 的图像数据是 28 像素× 28 像素的 8bit 灰度图像（单通道），每个像素的取值在 0 到 255 之间。每个图像数据都相应地标有“7”、“2”、“1”等标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 3s 0us/step\n",
      "Dataset loading completed\n"
     ]
    }
   ],
   "source": [
    "from demo_code.download import load_mnist\n",
    "(train_images, train_labels), (test_images, test_labels) = load_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train_images 和 train_labels 组成了训练集（training set），模型将从这些数据中进行学习。然后在测试集（test set，即 test_images 和 test_labels）上对模型进行测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "image = train_images[0]\n",
    "label = train_labels[0]\n",
    "print(label) # 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAN80lEQVR4nO3df6hcdXrH8c+ncf3DrBpTMYasNhuRWBWbLRqLSl2RrD9QNOqWDVgsBrN/GHChhEr6xyolEuqP0qAsuYu6sWyzLqgYZVkVo6ZFCF5j1JjU1YrdjV6SSozG+KtJnv5xT+Su3vnOzcyZOZP7vF9wmZnzzJnzcLife87Md879OiIEYPL7k6YbANAfhB1IgrADSRB2IAnCDiRxRD83ZpuP/oEeiwiPt7yrI7vtS22/aftt27d281oAesudjrPbniLpd5IWSNou6SVJiyJia2EdjuxAj/XiyD5f0tsR8U5EfCnpV5Ku6uL1APRQN2GfJekPYx5vr5b9EdtLbA/bHu5iWwC61M0HdOOdKnzjND0ihiQNSZzGA03q5si+XdJJYx5/R9L73bUDoFe6CftLkk61/V3bR0r6kaR19bQFoG4dn8ZHxD7bSyU9JWmKpAci4o3aOgNQq46H3jraGO/ZgZ7ryZdqABw+CDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUii4ymbcXiYMmVKsX7sscf2dPtLly5tWTvqqKOK686dO7dYv/nmm4v1u+66q2Vt0aJFxXU///zzYn3lypXF+u23316sN6GrsNt+V9IeSfsl7YuIs+toCkD96jiyXxQRH9TwOgB6iPfsQBLdhj0kPW37ZdtLxnuC7SW2h20Pd7ktAF3o9jT+/Ih43/YJkp6x/V8RsWHsEyJiSNKQJNmOLrcHoENdHdkj4v3qdqekxyTNr6MpAPXrOOy2p9o++uB9ST+QtKWuxgDUq5vT+BmSHrN98HX+PSJ+W0tXk8zJJ59crB955JHF+nnnnVesX3DBBS1r06ZNK6577bXXFutN2r59e7G+atWqYn3hwoUta3v27Cmu++qrrxbrL7zwQrE+iDoOe0S8I+kvauwFQA8x9AYkQdiBJAg7kARhB5Ig7EASjujfl9om6zfo5s2bV6yvX7++WO/1ZaaD6sCBA8X6jTfeWKx/8sknHW97ZGSkWP/www+L9TfffLPjbfdaRHi85RzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlrMH369GJ948aNxfqcOXPqbKdW7XrfvXt3sX7RRRe1rH355ZfFdbN+/6BbjLMDyRF2IAnCDiRB2IEkCDuQBGEHkiDsQBJM2VyDXbt2FevLli0r1q+44opi/ZVXXinW2/1L5ZLNmzcX6wsWLCjW9+7dW6yfccYZLWu33HJLcV3UiyM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTB9ewD4JhjjinW200vvHr16pa1xYsXF9e9/vrri/W1a9cW6xg8HV/PbvsB2zttbxmzbLrtZ2y/Vd0eV2ezAOo3kdP4X0i69GvLbpX0bEScKunZ6jGAAdY27BGxQdLXvw96laQ11f01kq6uty0Adev0u/EzImJEkiJixPYJrZ5oe4mkJR1uB0BNen4hTEQMSRqS+IAOaFKnQ287bM+UpOp2Z30tAeiFTsO+TtIN1f0bJD1eTzsAeqXtabzttZK+L+l429sl/VTSSkm/tr1Y0u8l/bCXTU52H3/8cVfrf/TRRx2ve9NNNxXrDz/8cLHebo51DI62YY+IRS1KF9fcC4Ae4uuyQBKEHUiCsANJEHYgCcIOJMElrpPA1KlTW9aeeOKJ4roXXnhhsX7ZZZcV608//XSxjv5jymYgOcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9knulFNOKdY3bdpUrO/evbtYf+6554r14eHhlrX77ruvuG4/fzcnE8bZgeQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtmTW7hwYbH+4IMPFutHH310x9tevnx5sf7QQw8V6yMjIx1vezJjnB1IjrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHUVnnnlmsX7PPfcU6xdf3Plkv6tXry7WV6xYUay/9957HW/7cNbxOLvtB2zvtL1lzLLbbL9ne3P1c3mdzQKo30RO438h6dJxlv9LRMyrfn5Tb1sA6tY27BGxQdKuPvQCoIe6+YBuqe3XqtP841o9yfYS28O2W/8zMgA912nYfybpFEnzJI1IurvVEyNiKCLOjoizO9wWgBp0FPaI2BER+yPigKSfS5pfb1sA6tZR2G3PHPNwoaQtrZ4LYDC0HWe3vVbS9yUdL2mHpJ9Wj+dJCknvSvpxRLS9uJhx9sln2rRpxfqVV17ZstbuWnl73OHir6xfv75YX7BgQbE+WbUaZz9iAisuGmfx/V13BKCv+LoskARhB5Ig7EAShB1IgrADSXCJKxrzxRdfFOtHHFEeLNq3b1+xfskll7SsPf/888V1D2f8K2kgOcIOJEHYgSQIO5AEYQeSIOxAEoQdSKLtVW/I7ayzzirWr7vuumL9nHPOaVlrN47eztatW4v1DRs2dPX6kw1HdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2SW7u3LnF+tKlS4v1a665plg/8cQTD7mnidq/f3+xPjJS/u/lBw4cqLOdwx5HdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2w0C7sexFi8abaHdUu3H02bNnd9JSLYaHh4v1FStWFOvr1q2rs51Jr+2R3fZJtp+zvc32G7ZvqZZPt/2M7beq2+N63y6ATk3kNH6fpL+PiD+X9FeSbrZ9uqRbJT0bEadKerZ6DGBAtQ17RIxExKbq/h5J2yTNknSVpDXV09ZIurpHPQKowSG9Z7c9W9L3JG2UNCMiRqTRPwi2T2ixzhJJS7rsE0CXJhx229+W9Iikn0TEx/a4c8d9Q0QMSRqqXoOJHYGGTGjozfa3NBr0X0bEo9XiHbZnVvWZknb2pkUAdWh7ZPfoIfx+Sdsi4p4xpXWSbpC0srp9vCcdTgIzZswo1k8//fRi/d577y3WTzvttEPuqS4bN24s1u+8886WtccfL//KcIlqvSZyGn++pL+V9LrtzdWy5RoN+a9tL5b0e0k/7EmHAGrRNuwR8Z+SWr1Bv7jedgD0Cl+XBZIg7EAShB1IgrADSRB2IAkucZ2g6dOnt6ytXr26uO68efOK9Tlz5nTSUi1efPHFYv3uu+8u1p966qli/bPPPjvkntAbHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IIk04+znnntusb5s2bJiff78+S1rs2bN6qinunz66acta6tWrSque8cddxTre/fu7agnDB6O7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQRJpx9oULF3ZV78bWrVuL9SeffLJY37dvX7FeuuZ89+7dxXWRB0d2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUjCEVF+gn2SpIcknSjpgKShiPhX27dJuknS/1ZPXR4Rv2nzWuWNAehaRIw76/JEwj5T0syI2GT7aEkvS7pa0t9I+iQi7ppoE4Qd6L1WYZ/I/Owjkkaq+3tsb5PU7L9mAXDIDuk9u+3Zkr4naWO1aKnt12w/YPu4FusssT1se7i7VgF0o+1p/FdPtL8t6QVJKyLiUdszJH0gKST9k0ZP9W9s8xqcxgM91vF7dkmy/S1JT0p6KiLuGac+W9KTEXFmm9ch7ECPtQp729N425Z0v6RtY4NefXB30EJJW7ptEkDvTOTT+Ask/Yek1zU69CZJyyUtkjRPo6fx70r6cfVhXum1OLIDPdbVaXxdCDvQex2fxgOYHAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ9HvK5g8k/c+Yx8dXywbRoPY2qH1J9NapOnv7s1aFvl7P/o2N28MRcXZjDRQMam+D2pdEb53qV2+cxgNJEHYgiabDPtTw9ksGtbdB7Uuit071pbdG37MD6J+mj+wA+oSwA0k0Enbbl9p+0/bbtm9toodWbL9r+3Xbm5uen66aQ2+n7S1jlk23/Yztt6rbcefYa6i322y/V+27zbYvb6i3k2w/Z3ub7Tds31Itb3TfFfrqy37r+3t221Mk/U7SAknbJb0kaVFEbO1rIy3YflfS2RHR+BcwbP+1pE8kPXRwai3b/yxpV0SsrP5QHhcR/zAgvd2mQ5zGu0e9tZpm/O/U4L6rc/rzTjRxZJ8v6e2IeCcivpT0K0lXNdDHwIuIDZJ2fW3xVZLWVPfXaPSXpe9a9DYQImIkIjZV9/dIOjjNeKP7rtBXXzQR9lmS/jDm8XYN1nzvIelp2y/bXtJ0M+OYcXCarer2hIb7+bq203j309emGR+YfdfJ9OfdaiLs401NM0jjf+dHxF9KukzSzdXpKibmZ5JO0egcgCOS7m6ymWqa8Uck/SQiPm6yl7HG6asv+62JsG+XdNKYx9+R9H4DfYwrIt6vbndKekyjbzsGyY6DM+hWtzsb7ucrEbEjIvZHxAFJP1eD+66aZvwRSb+MiEerxY3vu/H66td+ayLsL0k61fZ3bR8p6UeS1jXQxzfYnlp9cCLbUyX9QIM3FfU6STdU92+Q9HiDvfyRQZnGu9U042p43zU+/XlE9P1H0uUa/UT+vyX9YxM9tOhrjqRXq583mu5N0lqNntb9n0bPiBZL+lNJz0p6q7qdPkC9/ZtGp/Z+TaPBmtlQbxdo9K3ha5I2Vz+XN73vCn31Zb/xdVkgCb5BByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/D+f1mbt6t55/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(image.shape)\n",
    "plt.imshow(image, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 神经网络的推理\n",
    "\n",
    "神经网络的输入层有 784 个神经元（图像分辨率 28 × 28 = 784），输出层有 10 个神经元（数字 0 到9，共 10 个类别）。此外，这个神经网络有 2 个隐藏层，第 1 个隐藏层有 50 个神经元，第 2 个隐藏层有 100 个神经元。这个 50 和 100 可以设置为任何值。\n",
    "\n",
    "init_network() 会读入保存在 pickle 文件 sample_weight.pkl 中的事先学习好的权重参数，这个文件中以字典变量的形式保存了权重和偏置参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data():\n",
    "    (train_images, train_labels), (test_images, test_labels) = load_mnist()\n",
    "    return test_images, test_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_network():\n",
    "    with open(\"demo_code/mnist_weight.pkl\", 'rb') as f:\n",
    "        network = pickle.load(f)\n",
    "    return network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(network, x):\n",
    "    W1, W2, W3 = network['W1'], network['W2'], network['W3']\n",
    "    b1, b2, b3 = network['b1'], network['b2'], network['b3']\n",
    "    a1 = np.dot(x, W1) + b1\n",
    "    z1 = sigmoid(a1)\n",
    "    a2 = np.dot(z1, W2) + b2\n",
    "    z2 = sigmoid(a2)\n",
    "    a3 = np.dot(z2, W3) + b3\n",
    "    y = softmax(a3)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset loading completed\n",
      "Accuracy: 0.9352\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "network = init_network()\n",
    "\n",
    "test_images, test_labels = get_data()\n",
    "test_images = test_images.astype(np.float32) / 255.0      # Normalization\n",
    "\n",
    "accuracy_cnt = 0\n",
    "\n",
    "for i in range(len(test_images)):\n",
    "    y = predict(network, test_images[i].flatten())\n",
    "    p = np.argmax(y)                  # argmax获取数组中最大值的索引\n",
    "    if p == test_labels[i]:\n",
    "        accuracy_cnt += 1\n",
    "        \n",
    "print(\"Accuracy: \" + str(float(accuracy_cnt) / len(test_images)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个例子中，我们对图像进行了预处理，将图像的各个像素值除以 255，使得数值在 0.0~1.0的范围内。\n",
    "\n",
    "执行上面的代码后，会显示“Accuracy: 0.9352”。这表示有 93.52% 的手写数字图像被正确分类了。目前我们的目标是运行学习到的神经网络，所以暂不讨论识别精度本身。不过以后我们会花精力在神经网络的结构和学习方法上，思考如何进一步提高这个精度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 批处理\n",
    "\n",
    "批处理对神经网络的运算大有利处，可以大幅缩短每张图像的处理时间。因为多数处理数值计算的库都对大型数组运算的处理进行了优化。并且，在神经网络的运算中，当数据传送成为瓶颈时，批处理可以减轻数据总线的负荷（严格地讲，相对于数据读入，可以将更多的时间用在计算上）。也就是说，批处理一次性计算大型数组要比分开逐步计算各个小型数组速度更快。\n",
    "\n",
    "现在我们来考虑打包输入多张图像的情形。例如，我们想用 predict() 函数一次性打包处理 100 张图像。可以将 x 的形状改为 100 × 784，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset loading completed\n",
      "Accuracy: 0.9352\n"
     ]
    }
   ],
   "source": [
    "network = init_network()\n",
    "\n",
    "test_images, test_labels = get_data()\n",
    "test_images = test_images.astype(np.float32) / 255.0      # Normalization\n",
    "\n",
    "batch_size = 100\n",
    "accuracy_cnt = 0\n",
    "        \n",
    "for i in range(0, len(test_images), batch_size):\n",
    "    test_images_batch = test_images[i:i+batch_size].reshape(batch_size, 784)\n",
    "    y_batch = predict(network, test_images_batch)\n",
    "    p = np.argmax(y_batch, axis=1)          # 沿指定轴找到最大值元素的索引\n",
    "    accuracy_cnt += np.sum(p == test_labels[i:i+batch_size])\n",
    "    \n",
    "print(\"Accuracy: \" + str(float(accuracy_cnt) / len(test_images)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}