PyTorch-day02-AutoGrad.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5c50ea66",
   "metadata": {},
   "source": [
    "## 自动求导\n",
    "PyTorch 中， 所有神经网络的核心是 autograd 包。autograd包为张量上的所有操作提供了自动求导机制。\n",
    "\n",
    "### 本节内容：\n",
    "- autograd的求导机制\n",
    "\n",
    "- 梯度的反向传播\n",
    "\n",
    "### Autograd简介\n",
    "torch.Tensor 是这个包的核心类。如果设置它的属性 .requires_grad 为 True，那么它将会追踪对于该张量的所有操作。当完成计算后可以通过调用 .backward()，来自动计算所有的梯度。这个张量的所有梯度将会自动累加到.grad属性。\n",
    "\n",
    "> 注意：在 y.backward() 时，如果 y 是标量，则不需要为 backward() 传入任何参数；否则，需要传入一个与 y 同形的Tensor。\n",
    "\n",
    "requires_grad参数说明：\n",
    "- requires_grad=True时，表示参数需要参与训练，并在训练过程中进行梯度计算\n",
    "- requires_grad=False时，表示参数不需要参与训练，可以将代码块包装在 with torch.no_grad(): 中"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ab83b0fe",
   "metadata": {},
   "source": [
    "autograd 要点：\n",
    "- Tensor 和 Function 互相连接生成了一个无环图 (acyclic graph)，它编码了完整的计算历史。每个张量都有一个.grad_fn属性，该属性引用了创建 Tensor 自身的Function\n",
    "- 张量是用户手动创建的，即这个张量的grad_fn是 None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "df839921",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import torch\n",
    "# 手动创建Tensor ，则grad fun 为None\n",
    "x = torch.randn(3,3,requires_grad=True)\n",
    "print(x.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "85590d27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# 计算导数\n",
    "# 设置requires_grad=True用来追踪其计算历史\n",
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "print(x)\n",
    "print(x.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "227ac27a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], grad_fn=<PowBackward0>)\n",
      "<PowBackward0 object at 0x00000219CFA7B9A0>\n"
     ]
    }
   ],
   "source": [
    "#y是计算的结果，所以它有grad_fn属性。\n",
    "y = x**2\n",
    "print(y)\n",
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2f750712",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<MulBackward0>) tensor(3., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "#.requires_grad_(...) 原地改变了现有张量的requires_grad标志。\n",
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0b5b0459",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "# 如果没有指定的话，默认输入的这个标志是 False。\n",
    "a = torch.randn(2, 2) # 缺失情况下默认 requires_grad = False\n",
    "b = ((a * 3) / (a - 1))\n",
    "print(b.requires_grad)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "91e8acc6",
   "metadata": {},
   "source": [
    "### 梯度求解"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e430146a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 求解out\n",
    "# 因为 out 是一个标量，因此out.backward()和 out.backward(torch.tensor(1.)) 等价\n",
    "out.backward()\n",
    "#d(out)/dx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b4242da7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMAAAABtCAYAAAD3Xm43AAAZ+UlEQVR4nO2df2xUV5agv163ntWleaxHVQLluclSgaG8blGQcZWYUDQbV3ubcqJgoE0xAxVrMInGmBU4LdlkJ4ZWB486GKkxUadwb/xDCwY1BepQaBPsGShHNIWGfbWKXVYsVwZ43mH8WkFVaq/frKf9hFX7h23YJNguOy7sst/3V8H7dd71Pe+ce++553wnmUwmMTBYCPT42bSvDesbjbRW2IlfPsy29+Ho+dOUrEzPI79jKIDBgmFUJXSyljMPrBSu1IgN2/H9Fx/O5el7pKEABkuafzffAhgYzCeGAhgsaQwFMFjSfHe+BZh3HkYJtt9iYNiCc6c3rQOupUqiK8i1fxxgyOJk9w4n5qz5lugJi88CjKq0HyvHf0eb/tyH7dT8PIJ1WyWVOy10vrGLli/SL+JSIvFJDbWfWSl5s5Ld5k727W4hNjo399a7Wig/1o76Le63yBRAJ/rhYd4d9bLbIU57duxjP6GeAbQsYLmbXXsF/L8Kkki/oEuEGMHGEN2qBllgLtyFz+TH//HctLCwYRe+0XepOC2jz/Iei0oB9C4/xy9IHDnoScnM2l49yZnGSlzjurLsexa4HaU7BeNhkAo2St47Q+ubLsaaeBnZORDu6mZumljEffgY1ivH8UdmpwKLRwGGZfzH2hArqiiRUrtE+1LmUqOfto52WqprePeqDCbQflfP8Z+WU7gvgAIwGqPt4BO3Sn8QpuFQG7G0vcwiYVRjIHIJf2Mb7R0t1FS/S7ALBB7xTxeOU3NwF1vfC48pw2CI+rIagv3AqI56s4HDF1Jo4eUeqg5aaHunAXl45iIuEgXQiZ6tpw0fB7ZbU7uiq4HygzKut4/i2+qh/C0XepcOL4j0tFlwF1nQ7sVQNOBemMAdsEgaoUY/TY1+2rqGGEnvS2U4OtEPyqnocnHkHR+ereVUbdKJDkPeyt/z3z93UvLiCImIQhzQIiECgxYsfwzh/7AJf1Mb8lBqLWzdUYUvO8DxpuiMXaHFoQD9V2hoUnDt8+I0pXKBQqCuDWWDm81fsxb2olc4/OsStHAIaasbpwjKZzdQJTt5KyTcFZXs3mhJx1ssLu4FOH5WwfnyZqSvuKN23Fv+irp3JGJXVew7XVjRiUbaYZWNvDw3lRW7cebM4FmCHW+ZHfXsCQL3ZibmIlAAjdCHDURNXrxbU/R9HkaR+8G+xYl5/L8SkRCy5MH347WI2X1EOwTsG+2IaCjRGILDjlVI20ssOhKfyyjYcb34uIWRO2WkrT6KVwuID/u4oVrZ7LACCrHbYHXYH/89Zoq07QBeU4yGD9tnNL7IfAX44hJnOnSkPR5cKX39gRyJXBNYxPHRryZzrkml5FAl7uXAYJwBLLzwvAjDUcId4NywnunnlQwmEFfkImBh2UQT3zlHs1pC1UH3WCcfVIlhY9VzwD2ZG6qEy547+wcKTkrKJLju51Jv6pdl+EKYjhxsQsFOjcee+mWCk6rGKqqPHab28zz0/iHs7zRStXH8+7N8MyWlQdoutfDHwYsEsVL1g9l+m5YmwsYqWg9VU/tWLX35Oopm56i/6slCY34xVYXvEvywjUivn5ipmP2rv52JtW31YW+sx381jC/fRSp3y2wFGAxx6aIO6zw4ZxgvLuT7OH3Z99Rj+oMBxJ0naVlrJtZ6hRalEHua4tEXLwK2stNcKnv60UT/EPZDrfhWjhB6p4Ery5ys/7YmduVmPOvqiV4OEnrThSeFb1ZGu0Dq9QAhwO5xYp2z5fUEoVPlHPjFDdThMMHWOK6y7dgFAJVQo5/mTxUY7qTtl36CPcaiwcyJcvGNA9Sel9H6r9HWYcVbNu4aPQjhb2wm1A/69TYaGoNEU25iic2v2oEQ564rKV2RwfsBVAL7tlHfY6Pq4nl8q+fuzsrlw9SrTpyqzH3Hfo7ssCMuoPiVzEcj/H4NIZOL7GgUobSKqi0pTmBMR3+AvaX1xPJruHTWy3ST4pnrAj24RXsPIP0I56q5vbW19DRnAHi6izQf6IMaiCLCJIqoaxqYZn/82SLiOnQGVzpuvWo9P5Ig1htE7vdinaZvZKwCJD4LEwX44TpsKf5RHQ5H2uSJRCJpu7d6uZxt70Wh9DS33/7m4E6/U8/LBwPok3z1pjs+V6SzfSHVNrax7mXgQozOz1S8q6a2LBmqABrdd8IAuO15KV/1jQYc1VDC17jWG2eZ9UeUFNkWpKsjmq2I9CGtsDx1ZkP4UwmrCdQ1FpaNAlkzOz5XPK2DzkcodJ7dAxfake9E0XZIU09fJzORkVvJEwUFyYKCPclzd2d9k2T3mYrku+33kyOPhpJ95w8lX/rbG8mhuZRziRP/uDpZ0dKdHHqUTMZDJ5Kv/aQ52ffoGTxYuZjcU1CQLCh4N3nr/059ambOAvXHuAXAemyz9f+1WwSaZPoSIGSJ2Eq9bO+ooblr7sRc2qQ3FHpKpFzWA3CLWP/Up2akAiTudqMCOFYx67VDk5P9TS3UvTbuEQsiAhCOpjZ9ZjAd6Q6FngLBis0BkKD77tQKl5FjgIG7Y/4/q61fC7SaCRqx9mZCohv3ihihOwoTXV+9Xk/bXQl6gsTNdoTvaqiKg6Ot4wNIPUHs6imC5reoKTRWiJ/KeCh0c3+M4o06oevdxLtAKALUEPVn+5DoIZgwY/9eNtoDBUdtC945mc6WkNYCEQjH7sMUEUYZaAFUlPFYD1vuLKMyR1WC1aU0m/ZTd7AET2kNu56TUZBwPteN/xMbvjd9eGwKod5c8pbLRLU4mg5aVwD/b5rxv9fOwL/N3VstLqYIhc5fSfiDILaySnweG0pnjNw8M3KPRnx4tvu6vokl1zb245465i1MQgZagDjK+GRDrnl2CqDdbObETQvesryvzqpITlx//gp2l4CYpSL3grTFgbeiHF/F+DkbvFRuiNLwfgDDWZqEiVDoY08JhXb8Ge6ikwg5oN6JgeTCUVpJ+Z5KQCfaVENDuwz5biTMSKNhImuO0LjPnlJszwQWsxWIQURhYJRJPYXMswAPn7gqknl2wSNKTxBd2oxzIvhqNEr4Cth3eHGaBUQToPUhRwSc66wIgK7N3ddpsTNlKPQqEHIEQKMvIj8JMx/W0H9/jeZeBx6HRt9dK6//bGzvRbT1Fn0zlEG0THwcY6hfTn5e5inAYGJcAZxIK2Z3C4tkhxELlnH9UT9qJrC2kiO7bYx0HmeTo5xAR5QQedjWiNAfoPp9Of2Dt0XClKHQWojjWxyUX75G9HeQl5+HOKoQqG1AFoqpq8vjX29D3k43tiwY6JdhtYUZ2/ociTEnSEZ5OPlpmecCDQ099umyZzkAlnbUUX+3ltqDcVwrBlBzimk95cFmAs0kYs0XidyJ494yQPijNvoH43jf8Br7AVJkylBoTUT8vg0xEiH+souB2wHaFI349v14zQKoCrIqYF8jATEi13WspeuxaDqIM3CCTNlYgBigT7GzMuMUIKFOOEDWWVsAsiTcb7fgfsohcWMV5zfO8r4G40wRCi06qbpwftIr9f4YsmnzWDr0/m5uqFZKVnRz+GcKdb/0pL5jzCxhBcKAoqrA00MiMs4FGhmd8MXnZ3+i1hXA3xggCsi/9eNvCk85y2AwMwbuygj5LuxmIMdCXs4I1y5FcFW4Z7ZdMiv78c+RKRJnZZwFiD+Q5/X54gYvlRu8UFE3r3IsVqxll7g9YTly3By9/jQ7PTOi/zzAorEAj3FIs18FNlgCWLCmEJyauQpAdtqiGg0WA9kpeckZpgAJlLvzLYNBxtGvTprvNcMUwMBgFuhMmsXPUACDJU3GKcCFL8ayX5n/T5jsac41WNqk0lcyTAHM5PyH/wjATwrXzDqNnsFSILW+kmEKYGAwtxgKYLCkMRTAYEljKIDBksZQAIMljaEABksaQwEMljTTh0MnQtS/1Uz3ozjKFwl0QFxlQyqsovWgc56i8g0M5obpFcDspuasG0ZlGjYeoI0S6s4eTb0ckYHBAiZ1F+heDyGATXasRgyCwSIhZQVI9I2lI7T9hf1bZGPLEEZ1tEEjDUpa0TW0WRS2nmtSVACdvt4wIOBcm67s8gsFHfmXL1NYtIm9F43UV2lBl6kvKqRwy14C0ySvTTcpKkAf3Z8AbCZvTTrFWQgILFthRUAkz7xsvoUBQOs8TqHDwa7W2HyLMjdkLUP6vgA5eY9zM80XqW2K7+8jPAw4HNinEVi73UBF4ww2rm+opPGnrgWVc8dWdv7JxuwFgJZQ0ICRL+Po2DJ/5i3Lhu/C7QVRgColBdDuRokBkmP9tP6/uKmK85vmQDKDx0ilLdwuGq8RNt/CLDJSUoC+aDsAm9fZ0iqMweQIOQvJRi4evjkGeCgTaGwj/DjbU4yeTwHc2I3+b7DI+JoFUAkcOUB9D0jfdXL1DRvck7mhAkVunCl8hGY8BlhXSWP1whoDZCILqwxq5vBUF8hcVMPJ3baxQhIf+ImZXBw96Empwt9iGAPEzu5l3/sK7vc6qCta+Kr5rMqgzhmjMdpe30fDv7ipv1qHO2f+RPmaAkhsrz1K/y8C+D/oh2g32X9xjPPHPNjmUchni87Qlwo6OmoiDhlgm55VGdQ5Y3QI9V90GFaJDwILRwFAWF1CTVPJfMiyQBBw/vRTOt8cIVtc+J0fgLU+zt9cCJOKKSI4qbneyYGRbOa7iTMuOe4zIUtAzDEmHNOKIM4o3X+6MPYDGCxpDAVYCnzRxt4tDja9EzLKPH0NQwGWAPofVJRh0NU48SmKRSxFDAV4CrGze9nk2ETt9cXxvRQ21vBpZyedv/ZiXQgzRKMx2vZswrGlltDg/IpiKMA3+Po06OJAEBfGoBP45jToPGLMAn2DDJwGzTSMadAFjjENmn6MaVADg/nHUACDJY2hAAZLmgxTgATxu/8LgP/2P/qMAtUGU5BaX8kwBYCyH4z/kP7TvMphsPBJpa9knAIYGMwlGaYAwrzPGxtkIAKLpUieiGXFfMtgkHGskhZhkbxHk5U+NjAAGGIkhTCLzF0J7lIZGGX2eUo1hXDHNbrjy3jBXYJnreFbzTkPowTbbzEwbMG504tz+bN8uIbaO/1ZGWcBcle5vv1N9Cj+t9vQHG9QuddJ/FdbF03k54LhYTs1P49g3VZJ5U4LnW/souWL+RHFac2d9FjGKcATNPRZekHazQAtd/qII4Bow/uX22l/u5no3Aq4pIl97CfUM4CWBSx3s2uvgP9XQRLPSgBdI5X83hmnAObnJ5J+zD6UVty4n5amOkpWjv1bMAlAmOi9uZDQAMD26knONFbiGvcsl33PArejdD8rQzsYZ2BClpXSpKdl3hjAtAwJULlPYhCY/N0mZzBGe0uIPylyY+kNEXkwngY9EaL+fJC+mxqeX7fgXQ2xCwc48YdyTh90ZkCClIWD9qXMpQ8VYq860a+H6P6DDKZitN/Vc/x6H6E/eGhp9WIdjdF26ARDZaep3DiHLTyYYOyvasPyp5OflnEWALM0nvgpwdBsCiyoQQ7vbUYor6Nymwdv9S6kOwpITr5zsxnLy8VYBvuI/bMGxAj/RoYVuUbnnwF6VwPlB2Vcbx/Ft9VD+Vsu9C4dXhDpabPgLrKg3YuhaMC9MIE7YJHmuIWHh8bDHyxTrh1lngUwS9hMEB4GRU3ApDO8T0Mj1HSCcI6X/flfDUaXNrrw/M0BuH0Cv1TM6RdF6O/mhirhzP4Hyvf9PdqXFmxbRCySBeVsDGupDW10GdmfXWRo1wXqts5ElsWKQqCuDWXDUTZ/zTrbi17hcImZ8Ht+pK2ncYqgfHIDVXLyfN9xyo9EUbJtuFdCrnUZ1/5hhEIXJLLMjNyM8MJ/baR8XWqbCBLqRHETK9IUf5bMswDkkrth7Fffw5luWVSIXtWRNjmxTrRjT5gAdrbvcmIWs+n7rB1hnQt7znhaeJOTfx+7gOPVYoSHUbI3HqGqzI11UCaouThysByPPUF7h/zsBngLmYdR5H6wb3E+/jQlIiFkyYPvx2sRs/uIdgjYN9oR0VCiMQSHlf99LY6nxIl2T8Hqq6P8VRvSF9foWeHjaMVunJYoTeG+lMWIfzmuAJusvDDFVHkGKoAZ6/jXO6HGUxrpP8GCtA5GLJYxl2ZUJdASIO/gEXxrBUAjroJlTS4iOtHb7bDBzrbKy+wW+4iZivG4RNDixBEo+c9OBFSUXhCeMxtuEkCORK4JLBN+hyZzrkml5FAl7uWMD04tvPC8CMNRwh3g3LCZHT8/Sd7wLVhdgnst4z68E88WK4zeR7kNeRZLikLoxL8cc4CENdYpfYTMc4EA6xo30A6fKwzgmkEyWAnv39XT/04tB+IuJFVFfLWV00UTVVfMbN7hJXg+QMuIxsWrYD1kx2wSkT9vhw1HeUEAolHaTZupWwkMRglHBIpLX2BkGISlXj5WcFLVWEX1scPUfp6H3j+E/Z1GqjaOd8PlmykpDdJ2qYU/Dl4kiJWqH5gRRBXljorwYt7YJMfnMmqOE+tyoLeHEFa8+RY0nRS2Ug4wML4Ittk2de/ISAUQ19ix0U7snoI6DNaZdDrJTU2r++nHhlUGTCWc/LUN870WrjQpFL6YCygoXWAtsmMGlHsyLC/GZgbt+i1CJjd1o+coPeumo8I+B2+Y2Qj5Pk5ffnquUv3BAOLOk7SsNRNrvUKLUoh9JaArxCICxbutgI4SkxG27MIKxO5cQV21HSlWTW3/MU6/Ms1YS1fp7wWwYV8ztV3OQBcIWJmH0wRwi9gcVhlMfHqC8oMnuKHqhH/bRHyLj+35AmgKsV4RV34uoKHEYoibbOQC4vN52IgRuK5zZKfR+acmQehUOQd+cQN1OEywNY6rbDt2AXigIJvWj3fYAZSIwPp1eYiAtMqJqN+g7baTyqIUJhr6Y9wCMDmxrZr61O8kk8nkt36vZ45O+L1NHL4MrmMdnN42R7Mv/QEOn1RxOlRkxcH+au+0RQENZoZy+TD1qhOnKnPfsZ8jO+yIc5ysK3H1MFvfDcO209w+5pqyrlqGKgBo12spfLs9pZcEcDgcaZcpEomk7d764HiRvEk6y3QVYtJdQWbhtO+Tj6P77zqp3zrNFyyZqXx5JXmooCBZUHAq+T8fzbcw6WXg0r5kQUFBsuAXt5IjTzk+8o8nki8VFCQLXr+YvD+L44uKR93JUz8sSBYUVCevxac/PTPHAADL3RQXAVyjZ7b1o0c1lJsB/I1+2jpiaAs0caxotiIiYFtheaqlm6gQI05UiJnh8XSS6ArS1ujHf1km8SyefS9CaJixmnapVJ5Jv0qmj6H26mRBQUHyxx/2zeLqkWT3mYrku+33kyOPhpJ95w8lX/rbG8mhOZdy6RL/uDpZ0dKdHHqUTMZDJ5Kv/aQ52Zdma93X8lqyoKAgWf1xCp//ZCZbAEAs3IXXBIlLYWZsBLRbBJpk+hIgZInYSr1s76ihuSsdki5FYgQbQ3SrGmSBuXAXPpMf/8fpXC9X6O5UweSl5OXUJkYyWgEQnJSUSZC4yI2eGV5rcrK/qYW618YXSoSxKuzhqDLlZQapYqPkvTO0vjlRAncZ2TkQ7upOX5GO3k7aekHa48GV4tpQRi6E/f/YtvqwN9bT8nGYN9ZNPxv0BI1YezMh0Y17RYzQHYWJrq9er6ftrgQ9QeJmO8J3NVTFwdHW8RKkeoLY1VMEzW9RU2gEwD2VUY2ByCWa+2MUb9QJXe8m3gVCEaCGqD/bh0QPwYQZ+/ey0R4oOGrHQtBnh478cRMqLo5uS309JrMtAMDK7VSVSXA5SChV6zqqEqwupdm0n7qDJXhKa9j1nIyChPO5bvyf2PC96cNjUwj15pK3XCaqxdF00LoC+H/TjP+9dgb+La1vlsHoRD8op6LLxZF3fHi2llO1SSc6DHn5Kwl/EMRWVonPY0PpjJGbZ0bu0YgPzyyy6ysMhrh0UUfa46N4BntEMt4CgIC9rBL32VrOdSh49kwfGaTdbObETQvesryvWgzJievPX8HuEhCzVORekLY48FaU46sYP2eDl8oNURreD2A4S5NwL8DxswrOY5u/lrTAjtvxZ7iLTiLkgHonBpILR2kl5XsqAZ1oUw0N7TLku5EwI42Giaw5QuM++5TWPXbZT8jk4uhfOmfgBSwGCwCQ4+FAtZ3YL8/QnsI2SaUniC5txrl6vKlGo4SvgH2HF6dZQDQBWh9yRMC5zooA6Nq3+DotMRKfyyjYcb34OCAauVNG2uqjeBUIOWORt30RGcFhHwtNH9bQf3+N5l4HHodG310rr/+skt0bLURbbzFlIPRgiHNnVVyHjlAywx2Ci0MBAGvpUWrWhTjVIk8bIm2R7DBiwTK+SKh+1ExgbSVHdtsY6TzOJkc5gY4oIfKwrRGhP0D1+7JRYTFFxBW5CFhYNhERfecczWoJVQfdmLUQx7c4KL98jejvIC8/D3FUIVDbgCwUU1eXx7/ehrydbmxZMNAvw2oLkwdC60TPNtC+uoqqHTPfH7sIXKBxsqx4f1ZDe+lx2jxXKc+f/FRpRx31d2upPRjHtWIANaeY1lMebCbQTCLWfJHInTjuLQOEP2qjfzCO9w2vEe+fIsLGKloPVVP7Vi19+TqKZueov2osL5AmIn7fhhiJEH/ZxcDtAG2KRnz7frxmAVQFWRWwr5GAGJHrOtbS9Vgmi4Puv0LDWSs1H/lmVwAwvcsSz56B4KHkS6+fS95P64JLd/JUQUHyUIqLLQapMxI+kSz44XgYg3IxuaegNHmu/WKy4q1ryW+09qP7yYt//VLy0G8HZv28ReMCTSBtO0lr6RByT3ocFq0rgL8xQBSQf+vH3xQ26hTMIQN3ZYR8F3YzkGMhL2eEa5ciuCrc39jZpfVGie9opG7bbFKDjJGx0aAGBnPBorMABgYzwVAAgyXN/wNKbOLI5a/nhQAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "f9bcd186",
   "metadata": {},
   "source": [
    "数学上，若有向量函数$y = f(x)$,\n",
    "，那么y关于x的梯度就是一个雅可比矩阵： \n",
    " ![image.png](attachment:image.png)\n",
    "而 torch.autograd 这个包就是用来计算一些雅可比矩阵的乘积的\n",
    "\n",
    "> 注意：grad在反向传播过程中是累加的(accumulated)，这意味着每一次运行反向传播，梯度都会累加之前的梯度，所以一般在反向传播之前需把梯度清零"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "13a3f213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4., 4.],\n",
      "        [4., 4.]])\n"
     ]
    }
   ],
   "source": [
    "# 再来反向传播⼀一次，注意grad是累加的\n",
    "out2 = x.sum()\n",
    "out2.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5b3d4e44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "# 梯度清零 再求导\n",
    "out3 = x.sum()\n",
    "x.grad.data.zero_()\n",
    "out3.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e5f5d9e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.5809, -1.0104,  1.2924], requires_grad=True)\n",
      "tensor(3.4806)\n",
      "tensor(6.9612)\n",
      "tensor(13.9224)\n",
      "tensor(27.8449)\n",
      "tensor(55.6897)\n",
      "tensor(111.3795)\n",
      "tensor(222.7590)\n",
      "tensor(445.5180)\n",
      "tensor(891.0359)\n",
      "tensor([ -594.8848, -1034.6632,  1323.3912], grad_fn=<MulBackward0>)\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "print(x)\n",
    "\n",
    "y = x * 2\n",
    "i = 0\n",
    "# norm y每个元素进行平方，然后对它们求和，最后取平方根 ,L2范式\n",
    "while y.data.norm() < 1000:\n",
    "    print(y.data.norm())\n",
    "    y = y * 2\n",
    "    i = i + 1\n",
    "print(y)\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7313732c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.0240e+02, 1.0240e+03, 1.0240e-01])\n"
     ]
    }
   ],
   "source": [
    "# y 不再是标量。torch.autograd 不能直接计算完整的雅可比矩阵，\n",
    "#但是如果我们只想要雅可比向量积，只需将这个向量作为参数传给 backward\n",
    "v = torch.tensor([0.1,1.0,0.0001],dtype=torch.float)\n",
    "y.backward(v)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "189790cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# 也可以通过将代码块包装在 with torch.no_grad(): 中，\n",
    "#来阻止 autograd 跟踪设置了.requires_grad=True的张量的历史记录。\n",
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "\n",
    "with torch.no_grad():\n",
    "    print((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8a8302cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.])\n",
      "True\n",
      "False\n",
      "tensor([2.], grad_fn=<MulBackward0>)\n",
      "tensor([1.], requires_grad=True)\n",
      "tensor([2.])\n"
     ]
    }
   ],
   "source": [
    "#如果我们想要修改 tensor 的数值，但是又不希望被 autograd 记录(即不会影响反向传播)， \n",
    "#那么我们可以对 tensor.data 进行操作\n",
    "x = torch.ones(1,requires_grad=True)\n",
    "\n",
    "print(x.data) # 还是一个tensor\n",
    "print(x.requires_grad)\n",
    "print(x.data.requires_grad) # 但是已经是独立于计算图之外\n",
    "\n",
    "y = 2 * x\n",
    "print(y)\n",
    "x.data *= 100 # 只改变了值，不会记录在计算图，所以不会影响梯度传播\n",
    "print(x) # 更改data的值也会影响tensor的值 \n",
    "y.backward()\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "668bbdcb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}