3.6.softmax回归的从零开始实现.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T02:56:47.229072Z",
     "start_time": "2019-05-14T02:56:46.496019Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import d2lzh as d2l\n",
    "from mxnet import autograd, nd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 获取和读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:00:50.202779Z",
     "start_time": "2019-05-14T03:00:49.695164Z"
    }
   },
   "outputs": [],
   "source": [
    "# 使用Fashion-MNIST数据集\n",
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化模型参数"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "输入28x28=784，输出10，因此softmax的权重和偏置矩阵分别为784x10, 1x10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T04:55:48.557559Z",
     "start_time": "2019-05-14T04:55:48.552572Z"
    }
   },
   "outputs": [],
   "source": [
    "# 输入784，输出10\n",
    "num_inputs = 784\n",
    "num_outputs = 10\n",
    "# 随机初始化W，正态分布(0, 0.01)，shape 784x10，偏置b为1x10的全0向量\n",
    "W = nd.random.normal(scale=0.01, shape=(num_inputs, num_outputs))\n",
    "b = nd.zeros(num_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T04:55:49.716656Z",
     "start_time": "2019-05-14T04:55:49.712669Z"
    }
   },
   "outputs": [],
   "source": [
    "# 为参数附上梯度\n",
    "W.attach_grad()\n",
    "b.attach_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实现softmax运算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:11:33.382351Z",
     "start_time": "2019-05-14T03:11:33.366380Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\n",
       " [[5. 7. 9.]]\n",
       " <NDArray 1x3 @cpu(0)>, \n",
       " [[ 6.]\n",
       "  [15.]]\n",
       " <NDArray 2x1 @cpu(0)>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对列（axis=0）或行（axis=1）的元素求和\n",
    "X = nd.array([[1, 2, 3], [4, 5, 6]])\n",
    "X.sum(axis=0, keepdims=True), X.sum(axis=1, keepdims=True)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Softmax函数：输入特征矩阵X，行数是样本数，列数是输出个数，输出矩阵中的任意一行元素代表了一个样本在各个输出类别上的预测概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:16:24.163525Z",
     "start_time": "2019-05-14T03:16:24.158540Z"
    }
   },
   "outputs": [],
   "source": [
    "def softmax(X):\n",
    "    # 对X矩阵每一个元素求exp\n",
    "    X_exp = X.exp()\n",
    "    # partition 是按行对每一行元素求和\n",
    "    partition = X_exp.sum(axis=1, keepdims=True)\n",
    "    # 对每一个元素，除以他所在行的总和，最终得到的矩阵每行元素和为1且非负\n",
    "    return X_exp / partition  # 这里应用了广播机制"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:16:24.395485Z",
     "start_time": "2019-05-14T03:16:24.381522Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\n",
       " [[0.01606132 0.4672664  0.16146936 0.33617657 0.01902632]\n",
       "  [0.27215305 0.16692853 0.04439886 0.39427412 0.12224543]]\n",
       " <NDArray 2x5 @cpu(0)>, \n",
       " [1. 1.]\n",
       " <NDArray 2 @cpu(0)>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = nd.random.normal(shape=(2, 5))\n",
    "X_prob = softmax(X)\n",
    "X_prob, X_prob.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:25:55.206463Z",
     "start_time": "2019-05-14T03:25:55.202470Z"
    }
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    # X.reshape((-1, num_inputs))，表示把X变成长度为num_inputs的向量，\n",
    "    # -1表示让程序自己计算，如X是28x28，num_inputs是784，-1自动计算为1\n",
    "    # 通过XW+b,得到一个样本在每个类别上的预测值1x10，再通过softmax运算，变成概率\n",
    "    return softmax(nd.dot(X.reshape((-1, num_inputs)), W) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义损失函数"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "为了得到标签的预测概率，我们可以使用pick函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:40:20.419206Z",
     "start_time": "2019-05-14T03:40:20.406238Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[0.1 0.5]\n",
       "<NDArray 2 @cpu(0)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = nd.array([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])\n",
    "y = nd.array([0, 2], dtype='int32')\n",
    "nd.pick(y_hat, y)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "交叉熵损失函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:41:02.080482Z",
     "start_time": "2019-05-14T03:41:02.077462Z"
    }
   },
   "outputs": [],
   "source": [
    "def cross_entropy(y_hat, y):\n",
    "    return -nd.pick(y_hat, y).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 计算分类准确率"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "给定一个类别的预测概率分布y_hat，我们把预测概率最大的类别作为输出类别。如果它与真实类别y一致，说明这次预测是正确的。分类准确率即正确预测数量与总预测数量之比\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:51:40.143050Z",
     "start_time": "2019-05-14T03:51:40.140051Z"
    }
   },
   "outputs": [],
   "source": [
    "def accuracy(y_hat, y):\n",
    "    # y_hat.argmax(axis=1)即预测值按行求最大值，返回索引，y是真实类别标签转成浮点\n",
    "    # ==判断返回的是0或1的NDArray，求均值即是正确预测数量与总预测数量之比，再转标量\n",
    "    return (y_hat.argmax(axis=1) == y.astype('float32')).mean().asscalar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T03:51:40.494732Z",
     "start_time": "2019-05-14T03:51:40.483760Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(y_hat, y)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "评价模型net在数据集data_iter上的准确率："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T04:02:12.800305Z",
     "start_time": "2019-05-14T04:02:12.796287Z"
    }
   },
   "outputs": [],
   "source": [
    "# 本函数已保存在d2lzh包中方便以后使用。该函数将被逐步改进：它的完整实现将在“图像增广”一节中\n",
    "# 输入数据集data_iter和模型net\n",
    "def evaluate_accuracy(data_iter, net):\n",
    "    # acc_sum为正确预测数，n为总预测数，这两个值初始化为0，随着迭代每一批小数据集，\n",
    "    # 这两个值会不断变化，算出每一批次的精度\n",
    "    acc_sum, n = 0.0, 0\n",
    "    # 对data_iter的每一批小数据集\n",
    "    for X, y in data_iter:\n",
    "        # y为真实类别，转为浮点32\n",
    "        y = y.astype('float32')\n",
    "        # X传入模型返回softmax输出的概率，按行求最大值并返回对应索引\n",
    "        # 用索引与真实类别比较，返回的NDArray再求和即正确预测的数量\n",
    "        acc_sum += (net(X).argmax(axis=1) == y).sum().asscalar()\n",
    "        # n 即总预测数量，y：(样本数, 1)，y.size:样本数x1=样本数\n",
    "        n += y.size\n",
    "    return acc_sum / n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T04:37:26.479089Z",
     "start_time": "2019-05-14T04:37:25.778992Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0856"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 随机初始化，未训练的结果，接近类别个数10的倒数0.1\n",
    "evaluate_accuracy(test_iter, net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练模型"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "跟线性回归的从零开始实现基本相似，使用小批量随机梯度下降来优化模型的损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T04:56:21.214282Z",
     "start_time": "2019-05-14T04:55:54.687030Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.7880, train acc 0.748, test acc 0.806\n",
      "epoch 2, loss 0.5726, train acc 0.811, test acc 0.821\n",
      "epoch 3, loss 0.5285, train acc 0.825, test acc 0.831\n",
      "epoch 4, loss 0.5051, train acc 0.831, test acc 0.836\n",
      "epoch 5, loss 0.4893, train acc 0.835, test acc 0.837\n"
     ]
    }
   ],
   "source": [
    "num_epochs, lr = 5, 0.1\n",
    "\n",
    "# 本函数已保存在d2lzh包中方便以后使用\n",
    "def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size,\n",
    "              params=None, lr=None, trainer=None):\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0\n",
    "        for X, y in train_iter:\n",
    "            with autograd.record():\n",
    "                y_hat = net(X)\n",
    "                #这步也可以不sum,因为backward默认会对l.sum().backward(),\n",
    "                # 但下面要改成train_l_sum += l.sum().asscalar()\n",
    "                l = loss(y_hat, y).sum() \n",
    "            l.backward()\n",
    "            if trainer is None:\n",
    "                d2l.sgd(params, lr, batch_size)\n",
    "            else:\n",
    "                trainer.step(batch_size)  # “softmax回归的简洁实现”一节将用到\n",
    "            y = y.astype('float32')\n",
    "            train_l_sum += l.asscalar()\n",
    "            train_acc_sum += (y_hat.argmax(axis=1) == y).sum().asscalar()\n",
    "            n += y.size\n",
    "        test_acc = evaluate_accuracy(test_iter, net)\n",
    "        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'\n",
    "              % (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))\n",
    "\n",
    "train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, batch_size,\n",
    "          [W, b], lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从test里去一个批次数据出来\n",
    "for X, y in test_iter:\n",
    "    break\n",
    "\n",
    "true_labels = d2l.get_fashion_mnist_labels(y.asnumpy())\n",
    "pred_labels = d2l.get_fashion_mnist_labels(net(X).argmax(axis=1).asnumpy())\n",
    "titles = [true + '\\n' + pred for true, pred in zip(true_labels, pred_labels)]\n",
    "# 前10个打印出来\n",
    "d2l.show_fashion_mnist(X[0:9], titles[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习\n",
    "\n",
    "* 在本节中，我们直接按照softmax运算的数学定义来实现softmax函数。这可能会造成什么问题？（提示：试一试计算$\\exp(50)$的大小。）\n",
    "* 本节中的`cross_entropy`函数是按照[“softmax回归”](softmax-regression.ipynb)一节中的交叉熵损失函数的数学定义实现的。这样的实现方式可能有什么问题？（提示：思考一下对数函数的定义域。）\n",
    "* 你能想到哪些办法来解决上面的两个问题？"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "raw",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T05:06:40.589094Z",
     "start_time": "2019-05-14T05:06:40.584086Z"
    }
   },
   "source": [
    "1. c如果数值很大，exp(c)会上溢出 \n",
    "2.如果c为负数，且|c|很大，此时分母是一个极小的正数，有可能四舍五入为0，导致下溢出，计算log(0)会得到-∞\n",
    "为每一个x减去M=max(xi)，可以解决这两个问题\n",
    "参考：https://blog.csdn.net/csuzhaoqinghui/article/details/79742685"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gluon",
   "language": "python",
   "name": "gluon"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}