chapters/chapter01_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e928cc3e",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#The-Supervised-Learning-Paradigm\" data-toc-modified-id=\"The-Supervised-Learning-Paradigm-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>The Supervised Learning Paradigm</a></span></li><li><span><a href=\"#Observation-and-Target-Encoding\" data-toc-modified-id=\"Observation-and-Target-Encoding-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Observation and Target Encoding</a></span><ul class=\"toc-item\"><li><span><a href=\"#One-Hot-Representation\" data-toc-modified-id=\"One-Hot-Representation-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>One Hot Representation</a></span></li><li><span><a href=\"#TF-Representation\" data-toc-modified-id=\"TF-Representation-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>TF Representation</a></span></li><li><span><a href=\"#TF-IDF-Representation\" data-toc-modified-id=\"TF-IDF-Representation-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>TF-IDF Representation</a></span></li></ul></li><li><span><a href=\"#Computational-Graphs\" data-toc-modified-id=\"Computational-Graphs-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Computational Graphs</a></span></li><li><span><a href=\"#PyTorch-Basics\" data-toc-modified-id=\"PyTorch-Basics-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>PyTorch Basics</a></span><ul class=\"toc-item\"><li><span><a href=\"#Creating-Tensors\" data-toc-modified-id=\"Creating-Tensors-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Creating Tensors</a></span></li><li><span><a href=\"#Tensor-Types-&amp;-Size\" data-toc-modified-id=\"Tensor-Types-&amp;-Size-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Tensor Types &amp; Size</a></span></li><li><span><a href=\"#Tensor-Operations\" data-toc-modified-id=\"Tensor-Operations-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Tensor Operations</a></span></li><li><span><a href=\"#Indexing,-Slicing-&amp;-Joining\" data-toc-modified-id=\"Indexing,-Slicing-&amp;-Joining-5.4\"><span class=\"toc-item-num\">5.4&nbsp;&nbsp;</span>Indexing, Slicing &amp; Joining</a></span></li><li><span><a href=\"#Tensors-&amp;-Computational-Graphs\" data-toc-modified-id=\"Tensors-&amp;-Computational-Graphs-5.5\"><span class=\"toc-item-num\">5.5&nbsp;&nbsp;</span>Tensors &amp; Computational Graphs</a></span></li><li><span><a href=\"#CUDA-Tensors\" data-toc-modified-id=\"CUDA-Tensors-5.6\"><span class=\"toc-item-num\">5.6&nbsp;&nbsp;</span>CUDA Tensors</a></span></li></ul></li><li><span><a href=\"#Exercises\" data-toc-modified-id=\"Exercises-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Exercises</a></span></li><li><span><a href=\"#Summary\" data-toc-modified-id=\"Summary-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Summary</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f85fefa",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9284a919",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- **NLP** refers to a set of techniques involving application of statistical methods with or without insights from linguistics, to understand text for the sake of solving real world tasks.\n",
    "\n",
    "\n",
    "- The **understanding** of text is mainly derived by transforming texts of useable computational **representations**, which are discrete or continous combinational structures such as vectors or tensors, graphs and trees.\n",
    "\n",
    "\n",
    "- The learning of representation for a task from data(text) is the subject of **Machine Learning**.\n",
    "\n",
    "\n",
    "- **Deep Learning** enables one to efficiently learn representations from data using an abstraction called the **computational graph** and numerical optimisation techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ffb3f33",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## The Supervised Learning Paradigm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd3be0c3",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src=\"../images/figure1_1.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0f862d",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Supervision in machine Learning or **supervised learning**, refers to cases where the ground truth for the *targets* is available for the *observations*. For example, in document classification, the _target is a categorical label_ and the _observation is a document_.\n",
    "\n",
    "\n",
    "**Supervised Learning Paradigm concepts:**\n",
    "\n",
    "- *Observations*: Sometimes refers to as _inputs_ and denoted as $ x $. These are items about which we want to predict something.\n",
    "- *Targets*: Sometimes known as _ground truth_ and referred as $ y $. These are labels corresponding to an obvsevation and these are usually being predicted.\n",
    "- *Model*: A mathematical functions that takes an observation $ x $ and predicts the value of its target label $ y $.\n",
    "- *Parameters*: Also known as _weights_ parameterize the model. These are denoted by $ \\hat w $. \n",
    "- *Predictions*: These are the values of the targets gussed by the model given the observations and also known as _estimates_. These are denoted as $ \\hat y $.\n",
    "- *Loss Function*: Loss function is used to compare how far off a prediction is from its target for observations in the training data. Given a target and its prediction, the loss function assigns a scalar real value called the _loss_. Lower the value of loss, the better the model is at predicting the target. Loss is denoted by $ L $."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e0ad2ae",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Consider a dataset $ D = \\{X_i, y_i\\} ^ {n}_{i=1} $ with $ n $ examples. Given this dataset, we want to learn a function $ f $ parameterized by weights $ w $. We make an assumption about the structure of $ f $, and given that structure, the learned values of the weights $ w $ will fully characterize the model. For a given input $ X $, the model predicts $ \\hat y $ as the target:\n",
    "\n",
    "$$ \\hat y = f(X, w) $$\n",
    "\n",
    "In supervised learning, for training examples, we know the true target $ y $ for an observation. The loss for this instance will then be $ L(y, \\hat y) $. Supervised learning then becomes a process of finding the optimal parameters/weights $ w $ that will minimize the cumulative loss for all the $ n $ examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8967db",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Observation and Target Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b4fa96",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In order to use Machine Learning Algorithms for Text observations, we need a method to represent these observations numerically. A simple way to represent text is a **numerical vector**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358f888f",
   "metadata": {
    "hidden": true
   },
   "source": [
    "![Figure 1.2](../images/figure1.2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16478ef1",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### One Hot Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "316c2cf7",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The _one-hot representation_, as the name suggestes, starts with a zero vector and sets as 1 the corresponding entry in the vector if the word is present in the sentence or document.\n",
    "\n",
    "Examples:\n",
    "- _Time flies like an arrow._\n",
    "- _Fruit flies like a banana._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4627c2c",
   "metadata": {
    "hidden": true
   },
   "source": [
    "![Figure 1.3](../images/figure1.3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763d71a9",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### TF Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5e5a600",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The _TF representation_ of a phrase, sentence or document is simply the sum of the one-hot representations of its constituent wordas. TF of a word $ w $ is denoted by $ TF(w) $."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "41426dda",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD4CAYAAADM6gxlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAS/klEQVR4nO3de5BcZZ3G8efJRY1ouBgVJlwCBBFcJEoIeAkmIFCCgVgriZes65ZbkVolBAvccqE0fxgLlwJLyvKSXbZQVqzKhVWCcluuIYiQxASSCJSQLCQD1maRyCXCJPntH+ed0Bmme85M5szpl3w/VV1zzume7mfOdD/zzunT5zgiBADIx7C6AwAA+ofiBoDMUNwAkBmKGwAyQ3EDQGZGVP0AXVueZLeVGo3qmFx3hD2yrXNZ3RH2SM7rP/d1n7uRY45ws+sYcQNAZihuAMgMxQ0AmaG4ASAzFDcAZIbiBoDMUNwAkBmKGwAyQ3EDQGYobgDIDMUNAJmhuAEgMxQ3AGSG4gaAzFDcAJAZihsAMkNxA0BmKG4AyAzFDQCZobgBIDMUNwBkhuIGgMxQ3ACQGYobADJDcQNAZihuAMgMxQ0AmaG4ASAzFDcAZIbiBoDMUNwAkBmKGwAyQ3EDQGYobgDIDMUNAJmhuAEgMxQ3AGSG4gaAzFDcAJAZihsAMkNxA0BmKG4AyAzFDQCZobgBIDMUNwBkhuIGgMxQ3ACQGYobADJDcQNAZihuAMhMtsV92Xeu0ilnf0bTZ51fd5QByT3/mWdM0bq19+rR9ffp65d8pe44/cb6r0/u674d8mdb3NPPOl0/vurbdccYsJzzDxs2TFd/f74+OW2Wjjt+qmbOnK5jjjmq7lj9wvqvT87rXmqP/AMqbtunD3aQ/po44TjtO/rtdccYsJzzTzrxA3riiY3asOEpdXV1aeHCX+mcaWfWHatfWP/1yXndS+2Rf6Aj7msGNQWy0jH2QD29qXPX/KbNz6ij48AaE+1dWP8Y0ewK2zc2u0rSO1rdqe3ZkmZL0g+v/Lb+8QufHXBAAMDumha3pMmSZkl6scdyS5rU6k4jYoGkBZLUteXJ2JOAaD+dm5/VIQd37Jo/eOxB6ux8tsZEexfWP1ptKnlA0ssRcU+Py92SHhuaeGhHD61YrfHjD9e4cYdo5MiRmjHjXC296ba6Y+01WP9oWtwR8YmIuKvJdadUF6mcS751uT7/5Yu08alNOm36LC1Zemvdkfol5/w7duzQhXMv029+fb3WPny3Fi9eqvXrH687Vr+w/uuT87qX2iO/I6rdksGmknqN6phcd4Q9sq1zWd0R9kjO6z/3dZ+7kWOOcLPrst2PGwD2VhQ3AGSmVHHbHmX76KrDAAD61mdx254mabWkW9L8hBb7eAMAKlZmxD1PxX7bz0tSRKyWdHhliQAALZUp7q6I2NpjGXuKAEBNWn1ysts625+TNNz2UZLmSLq/2lgAgGbKjLgvkPQ+Sa9Iul7SVklzK8wEAGihzxF3RLws6dJ0AQDUrMxeJbfb3q9hfn/beX1GFQDeQMpsKhkTEc93z0TEnyW9q7JEAICWyhT3TtuHds/YPkzsVQIAtSmzV8mlku6zfY+KY3FPVjpJAgBg6JV5c/IW2x+UdHJaNDcitlQbCwDQTJkRtyS9WdJz6fbH2lZE3FtdLABAM30Wt+3vSpopaZ2knWlxSKK4AaAGZUbc0yUdHRGvVJwFAFBCmb1KnpQ0suogAIByyoy4X5a02vYdKj72LkmKiDmVpQIANFWmuG9MFwBAGyizO+BPbY+SdGhEPDYEmQAALXAGHADIzEDPgHNEZYkAAC0N9Aw4O3u9JQCgcpwBBwAyM9Az4FxYZSgAQHNlRtxnR8RuZ8CxfZ6kRZWlAgA0VWbE/Y2SywAAQ6DpiNv2JySdJWms7asbrhotaXvVwQAAvWu1qaRT0gpJ50ha2bD8BUkXVRkKANBc0+KOiDWS1ti+PiK6hjATAKCFMm9OTrI9T9Jh6faWFBHBh3AAoAZlivsaFZtGVkraUW0cAEBfyhT31oi4ufIkAIBSyhT3XbavkHSDdj8e96rKUgEAmipT3CelrxMbloWkUwc/DgCgL2WOxz11KIIAAMopczzud9u+xvbNaf5Y21+qPhoAoDdlPvJ+raRbJXWk+cclza0oDwCgD2WKe0xELFQ6BndEbBe7BQJAbcoU90u236HiDUnZPlnFoV0BADUos1fJ11Sc5f1I28slvVPSpytNBQBoqsxeJatsf0zS0So+7v4Yxy4BgPo03VRi+0TbB0q7tmufIGm+pCttHzBE+QAAPbTaxv0TSa9Kku1TJF0u6Wcqtm8vqD4aAKA3rTaVDI+I59L0TEkLImKJpCW2V1eeDADQq1Yj7uG2u4v9NEl3NlxX5k1NAEAFWhXwLyTdY3uLpG2SlkmS7fFid0AAqE2rM+DMt32HpIMk3RYRka4aJumCoQgHAHg9v9bH1RjxprHVPgBa2ta5rO4IQC1GdUyuO8Ie2f7qZje7rswnJwEAbYTiBoDMUNwAkBmKGwAyQ3EDQGYobgDIDMUNAJmhuAEgMxQ3AGSG4gaAzFDcAJAZihsAMkNxA0BmKG4AyAzFDQCZobgBIDMUNwBkhuIGgMxQ3ACQGYobADJDcQNAZihuAMgMxQ0AmaG4ASAzFDcAZIbiBoDMUNwAkBmKGwAyQ3EDQGYobgDIDMUNAJmhuAEgMxQ3AGSG4gaAzFDcAJAZihsAMkNxA0BmKG4AyAzFDQCZobgBIDMUNwBkhuIGgMxQ3ACQGYobADJDcQNAZihuAMgMxQ0AmaG4ASAzFDcAZCbr4j7zjClat/ZePbr+Pn39kq/UHadfcs4uSZd95yqdcvZnNH3W+XVHGZCc8+ecXco/v1T/6zfb4h42bJiu/v58fXLaLB13/FTNnDldxxxzVN2xSsk5e7fpZ52uH1/17bpjDFjO+XPOLuWfvx1ev9kW96QTP6AnntioDRueUldXlxYu/JXOmXZm3bFKyTl7t4kTjtO+o99ed4wByzl/ztml/PO3w+u3ZXHbHm37yF6Wv7+6SOV0jD1QT2/q3DW/afMz6ug4sMZE5eWcHdjbtcPrt2lx254h6VFJS2yvs31iw9XXtrpT27Ntr7C9YufOlwYnKQBAUusR979IOiEiJkj6B0nX2f5Uus6t7jQiFkTExIiYOGzYPoOTtIfOzc/qkIM7ds0fPPYgdXY+W8ljDbacswN7u3Z4/bYq7uER8YwkRcSDkqZKusz2HEkxFOFaeWjFao0ff7jGjTtEI0eO1IwZ52rpTbfVHauUnLMDe7t2eP22Ku4XGrdvpxKfIulcSe+rOFefduzYoQvnXqbf/Pp6rX34bi1evFTr1z9ed6xScs7e7ZJvXa7Pf/kibXxqk06bPktLlt5ad6R+yTl/ztml/PO3w+vXEb0Pnm0fL+mliPhjj+UjJc2IiJ+XeYARbxpb++h8b7atc1ndEYBajOqYXHeEPbL91c1NN0mPaHZFRKxpsrxLUqnSBgAMvmz34waAvRXFDQCZKVXctkfZPrrqMACAvvVZ3LanSVot6ZY0P8H2jRXnAgA0UWbEPU/SJEnPS1JErJZ0eGWJAAAtlSnurojY2mMZu/gBQE2a7g7YYJ3tz0kabvsoSXMk3V9tLABAM2VG3Beo+KTkK5Kul7RV0twKMwEAWuhzxB0RL0u6NF0AADUrs1fJ7bb3a5jf33ZeBxcAgDeQMptKxkTE890zEfFnSe+qLBEAoKUyxb3T9qHdM7YPE3uVAEBtyuxVcqmk+2zfo+IECpMlza40FQCgqTJvTt5i+4OSTk6L5kbElmpjAQCaKTPilqQ3S3ou3f5Y24qIe6uLBQBops/itv1dSTMlrZO0My0OSRQ3ANSgzIh7uqSjI+KVirMAAEoos1fJk5JGVh0EAFBOmRH3y5JW275DxcfeJUkRMaeyVACApsoU943pAgBoA2V2B/yp7VGSDo2Ix4YgEwCgBc6AAwCZGegZcI6oLBEAoKWBngFnZ6+3BABUjjPgAEBmBnoGnAurDAUAaK7MiPvsiNjtDDi2z5O0qLJUAICmyoy4v1FyGQBgCDQdcdv+hKSzJI21fXXDVaMlba86GACgd602lXRKWiHpHEkrG5a/IOmiKkMBAJprWtwRsUbSGtvXR0TXEGYCALRQ5s3JSbbnSTos3d6SIiL4EA4A1KBMcV+jYtPISkk7qo0DAOhLmeLeGhE3V54EAFBKmeK+y/YVkm7Q7sfjXlVZKgBAU2WK+6T0dWLDspB06uDHAQD0pczxuKcORRAAQDlljsf9btvX2L45zR9r+0vVRwMA9KbMR96vlXSrpI40/7ikuRXlAQD0oUxxj4mIhUrH4I6I7WK3QACoTZnifsn2O1S8ISnbJ6s4tCsAoAZl9ir5moqzvB9pe7mkd0r6dKWpAABNldmrZJXtj0k6WsXH3R/j2CUAUJ+mm0psn2j7QGnXdu0TJM2XdKXtA4YoHwCgh1bbuH8i6VVJsn2KpMsl/UzF9u0F1UcDAPSm1aaS4RHxXJqeKWlBRCyRtMT26sqTAQB61WrEPdx2d7GfJunOhuvKvKkJAKhAqwL+haR7bG+RtE3SMkmyPV7sDggAtWl1Bpz5tu+QdJCk2yIi0lXDJF0wFOEAAK/n1/o4T7ZnR0S2b5aSv1455885u0T+PVHmk5PtbnbdAfYQ+euVc/6cs0vkH7A3QnEDwF6F4gaAzLwRijvbbWQJ+euVc/6cs0vkH7Ds35wEgL3NG2HEDQB7FYobADJDce+lbI+zvbbuHFWxPcf2H2xvtv2DtOx821+oO1sZDfl/3o/v+Y3t/dLln6rMV5btF9PXDtuL0/QXu38n7aZx3TVmbjds4x5itodHxI5m80OYY5ykmyLib4b6sYeC7UclfTxdJkbEV2uO1C/d+SNiU8OyEekQy3197zi1ye/W9osR8bYey76oNv2dtNO6ayWrEbftX9peaXud7dlp2Yu259teY/sB2+9u04xX2l4j6UO9zH/N9tp0mZu+5xLbc9L092zfmaZP7c8orA8jbP88jewW236r7W/afihlWWDb6XHvtv1d2w/aftz25LR8nO1ltlely4fT8inpexbbfjQ9Tvd99foYg8X2jyUdIelmSfs3LJ9n++I0faTtW9Lvapnt96bl56Vca2zfO5i5BpLf9lbb17k4+9R1PUertm+yPSVNb7Q9RsUhmI+0vdr2FTX8CK/T7D8822fb/q3tMbbPSNOrbC+y/bbe7qtijetuUXfmtN5/afv2tJ6/ml63v0+9c0C6Xa/Pq0EXEdlcJB2Qvo6StFZS97kwp6Xl/yrpsjbNOKPhNrvmVZyg4hFJ+0h6m6R1kj4g6WRJi9Jtlkl6UNJISd+S9OVByDku5fhImv8PSRd350/LrmtYt3dLujJNnyXpv9P0WyW9JU0fJWlFmp6i4mBkB6sYIPxW0kcb11HPxxjk38NGSWMkfVHSD9KyeZIuTtN3SDoqTZ8k6c40/YiksWl6vxqfR93550laKWlUWr7r50nzN0ma0uN7xklaW+froCHfiw3Pt7WNP4OkT6Xn9v4p972S9km3+WdJ36whb2POnpn/KOntKk7fuFXS+em670ma2+p5NdiX3A7POsf2p9L0ISqK4lUVT16peIKfXkewBr1l3CFpScNtGuc/Kum/IuIlSbJ9g6TJkn4k6QTboyW9ImmVpInpujmDlPXpiFiepv8z3e8G219XUcgHqPhDsjTd5ob0daWKJ7VU/DH5ge0J6ed6T8P9PxjpX30Xx3AfJ+k+SVNbPEbl0kjuw5IWNQz235y+Lpd0re2Feu3nrduNEbGt7hCD7FQVz+czIuIvtj8p6VhJy9Pv5E0q/ti3k7si4gVJL9jeqtees49Ien8fz6tBlU1xp38HPy7pQxHxsu27Jb1FUlekP28qiqO2n6lFxr/G7tuxe86/TkR02d6g4i/9/ZIeljRV0nhJfxikyD3f4AhJP1Sx/fFp2/NU5O/2SvrauJ4vkvQnScerGFn/tZfb7/oe22/p4zGGwjBJz0fEhJ5XRMT5tk+SdLaklbZPiIj/G+J8Pb3UML1du2/iHOp1N1ieULE56D2SVqg4n+3tEfHZWlO11vh83tkwv1PF66Hp82qw5bSNe19Jf06F+F4VmxLazUAyLpM0PW1f3kev/fvYfd3FKv6FXCbpfEm/b/hDtacOtf2hNP05FaNhSdqSRg+fLnEf+0p6JiJ2Svo7ScP7uH130fTnMQZVRPxFxX8W50mSC8en6SMj4ncR8U1J/6viv6Z2slHSBNvDbB8iaVIvt3lBxb/07ex/JP2tpJ/Zfp+kByR9xMXx/mV7H9vvaXUHFRnwumv1vBpsORX3LSpGbH9Q8QbCAzXn6U2/M0bEKknXqtiG/TtJ/x4Rv09XL1NxPPTfRsSfVIxml/V2PwP0mKSvpLz7q9g8828qts3fKumhEvfxQ0l/7+KN1vdq99Hh60TE8wN4jCp8XtKXUu51ks5Ny6+w/Uh6U+p+SWtqytfMckkbJK2XdLWKTWi7Sf8hLE9vsrbFm5O9iYhHVfweFkkareK/y1/YfljFZpJq3thrnWnXupM0kHXX7Hk1qNgdEAAyk9OIGwAgihsAskNxA0BmKG4AyAzFDQCZobgBIDMUNwBk5v8BhLnFlDIZu+kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import seaborn as sns\n",
    "\n",
    "corpus = [\n",
    "    \"Time flies flies like an arrow.\",\n",
    "    \"Fruit flies like a banana.\"\n",
    "]\n",
    "one_hot_vectorizer = CountVectorizer(binary=True)\n",
    "one_hot = one_hot_vectorizer.fit_transform(corpus).toarray()\n",
    "sns.heatmap(\n",
    "    one_hot, annot=True,\n",
    "    cbar=False, xticklabels=sorted(list(one_hot_vectorizer.vocabulary_.keys())),\n",
    "    yticklabels=['Sentence 1', 'Sentence 2']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1336fcdd",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### TF-IDF Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8145728c",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The _IDF(Inverse Document Frequency)_ penalizes common tokens and reward rare tokens in the vector represemtation. The _IDF(w)_ of a token w is defined with respect to a corpus as:\n",
    "\n",
    "$$ IDF(w) = \\log \\frac {N}{n_w} $$\n",
    "where $ n_w $ is the number of documents containing the word $ w $ and $ N $ is the total number of documents.\n",
    "\n",
    "The _TF-IDF_ score is simply the product $ TF(w) * IDF(w) $.\n",
    "\n",
    "Heuristic representations like TF-IDF are rarely used in Deep Learning as the goal is to learn such representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8bd2c78a",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD4CAYAAADM6gxlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZVklEQVR4nO3de3xU9Z3/8ddnkgABlChXQRAB0aL1goioWMG2WrWotFVb3bX+an+oqyJatOvqdt3+aq1V19aqrbbuj2qrj4Ja79eiIKCogLDclIp44aagBUVuSeazf8whJCEzOYacOfPV9/PxmEfOLTPvnMy8c+bMyTnm7oiISDgyaQcQEZHPRsUtIhIYFbeISGBU3CIigVFxi4gEpjzpB9j09K06bCVFu4y6Lu0IO2X1yAFpR9gpPZ5/M+0ILRb6uu9w2WlpR9gplcdfZPnmaYtbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCouEVEAqPiFhEJjIpbRCQwKm4RkcCUpx0grhmL3uGXD75ANuuMPmIQP/j6kAbzH355Mb96aDpdqzoC8N2jD+RbR+6fRtQmhZ6/seOPG8F//ddPKctk+O//fx+/vOG2tCM1UDFkKB3Ovxgry7D5ycfZNPHeBvPbnXQy7UaNhmwtvmkTG359I7XvvgNA2d796Dh2PNahPWSddRefB9Vb0/gx8tL6L55SfO0GUdy12SzXTZrC7y48le5VHTnrxr9wzAH96L/H7g2WO27wPlx52oh0QhYQev7GMpkMt/z6Wr5x4vdYvnwVM196gkcfe4bFi/+edrScTIaOF45j/ZU/Irt2DVW/uYOtM2fUFQPAluf/xubHHwGgzbAj6XDehXx81RWQKWOXK67mkxuupfatpdguu0JtTVo/SZO0/ounVF+7LdpVYmZfb+0ghSx45316d61izy6dqCgv4/jBA5ky/61iRtgpoedvbOhhh7B06dssW/Yu1dXVTJz4MCePOj7tWHXK9/0StStXkF29Cmpq2DLlOdocMbzBMr5x4/aRdpXgucGKQ4dQs2wptW8tzS33yceQzRYreixa/8VTqq/dlm5x3wX0ac0ghXyw7lN6RG9DALpXdWT+O6t3WG7yvKXMWbqSvbpWMf5bR9Njt12KFbGg0PM31rNXD95bvrJufPmKVQw97JAUEzWU6dyF7JoP6saza9dQvt+Xdliu3ahTqfzW6VBRwforxgFQtmdvcNj12hvIdKpiy9Tn2DTpvmJFj0Xrv3hK9bWbt7jN7JF8s4DOhe7UzMYAYwB+M/a7nHviUS0OGNcxB/TlhMEDaVNRxv0zFvDvf/obv794dOKP21pCzx+izY8+xOZHH6LtyK/R/syz2XDjdVhZGRUHfJl1F5+Hb9lMp1/cTM3f36B67py0437ufF7Wfxqv3UK7So4G7gBuauK2odCduvud7j7E3Ye0Rml3q+rA6nXbH/L9dRvo1qljg2WqOlTSpqIMgNFHDGLxex9QKkLP39jKFavpvWfPuvE9e+3BypU7boWkJfvhWjJdu9WNZ7p0Jbt2bd7lt0yZTJsjc2/la9esoXr+PPzj9bBlC1tfnUn5gIGJZ/4stP6Lp1Rfu4WKeyaw0d2nNrpNAd5IPFk9+/fpzrtr1rHiw/VU19Ty9JwlHPPlvRsss2b9p3XDU+cvY+/uuxUzYkGh52/s1VlzGTBgb/r27U1FRQWnn34Kjz72TNqx6tS88TplvfYk070HlJfTdsSxbJ05o8EymZ696obbDD2C2hXLAaie/QrlfftB27aQKaPiwIOoefftYsZvltZ/8ZTqazfvrhJ3P6HAvK8kE6dp5WUZ/vU7x3DB7Y+QzWY5ZdggBuzRmdsfn8mgPt0Y8eV+3Dd1HlMWLKM8Y+zavh0//aevFTNiQaHnb6y2tpZLxl3NE4/fS1kmw4Q//oVFi5akHWu7bC0bbvsVnX5+I2QybH7mCWrfeZv2Z/+AmiWvs3Xmi1Se/C0qBh8KNTVkN2xgw43XAeAbNrDpwYlU/eYOcGfrKy9T/crMlH+ghrT+i6dUX7vm7ok+wKanb032AaSgXUZdl3aEnbJ65IC0I+yUHs+/mXaEFgt93Xe47LS0I+yUyuMvsnzz9J+TIiKBUXGLiAQmVnGbWaWZ7Zt0GBERaV6zxW1mo4C5wFPR+MEFjvEWEZGExdnivgYYCqwDcPe5wN75FxcRkSTFKe5qd1/faJqOFBERSUmcc5UsNLMzgTIz2wcYC7yYbCwREcknzhb3xcD+wBbgXmA9MC7BTCIiUkCzW9zuvhG4KrqJiEjK4hxV8qyZVdUb383Mnk40lYiI5BVnV0kXd1+3bcTd/wF0y7+4iIgkKU5xZ82s7qIJZrYXOqpERCQ1cY4quQqYbmZTyV1E4WiiiySIiEjxxflw8ikzGwwMiyaNc/f8Z0UXEZFExb3mZFvgo2j5QWaGu7+QXCwREcmn2eI2s+uBM4CFwLbLLTug4hYRSUGcLe5TgX3dfUvCWUREJIY4R5W8BVQkHUREROKJs8W9EZhrZpPJ/ds7AO4+NrFUIiKSV5zifiS6iYhICYhzOOAfzawS6OPubxQhk4iIFKAr4IiIBKalV8Dpl1giEREpqKVXwMk2uaSIiCROV8AREQlMS6+Ac0mSoUREJL84W9wnuXuDK+CY2WnApMRSiYhIXnG2uK+MOU1ERIog7xa3mZ0AnAj0MrNb6s3aFahJOpiIiDSt0K6SlcAs4GRgdr3pnwCXJhlKRETyy1vc7j4PmGdm97p7dREziYhIAXE+nBxqZtcAe0XLG+Durn/CERFJQZzivovcrpHZQG2ycUREpDlxinu9uz+ZeBIREYklTnE/b2Y3AA/S8HzccxJLJSIiecUp7sOjr0PqTXPg2NaPIyIizYlzPu6RxQgiIiLxxDkfd3czu8vMnozGB5nZuclHExGRpsT5l/cJwNNAz2h8CTAuoTwiItKMOMXdxd0nEp2D291r0GGBIiKpiVPcn5pZZ3IfSGJmw8id2lVERFIQ56iSy8hd5b2/mc0AugLfSTSViIjkFeeokjlmdgywL7l/d39D5y4REUlP3l0lZnaYmfWAuv3ahwLXAjeZ2e5FyiciIo0U2sd9B7AVwMy+AvwCuJvc/u07k48mIiJNKbSrpMzdP4qGzwDudPcHgAfMbG7iyUREpEmFtrjLzGxbsX8VeK7evDgfaoqISAIKFfB9wFQzWwtsAqYBmNkAdDigiEhqCl0B51ozmwzsATzj7h7NygAXFyOciIjsyLb3cTLK2/RK9gGkoE0rp6UdYadsuWF82hF2StvLb0w7QouFvu6rbn457Qg7pWbrCss3L85/ToqISAlRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoFRcYuIBEbFLSISGBW3iEhgVNwiIoEJuriPP24ECxe8wOuLpnPF5RemHeczKfXs02fO4pvf/SEnnP4D/nDPxB3mP/T4sxx90hl8+/sX8u3vX8j9jzxVN++m2+7ilLPOY9SZY/j5zb/F3YsZHYCygYfQfvxvaH/5bVSMGJ1/uQOG0fH6B8n06p+b0L4j7cb8Jx1++mfanPLDIqXdkdZ/uuu/OWm/fsuL/oitJJPJcMuvr+UbJ36P5ctXMfOlJ3j0sWdYvPjvaUdrVqlnr62t5Wc33cbvf/VzenTrwhk/vISRww+n/957NVjuG8cew1U/+pcG016bv4jX5i/iwbtvB+DsC8bz6mvzGTr4wKLlxzK0PfX/sukP/4mv/5DKi35JzaJX8Q+WN1yuTTvaHHUSte8u2T6tupqtz9xHpnsfMj36FC9zPVr/6a7/5pTC6zfYLe6hhx3C0qVvs2zZu1RXVzNx4sOcPOr4tGPFUurZ5y9eQp89e9K71x5UVFRwwleP4blpM2N9r5mxdetWqmtq2FpdTXVNLZ13r0o2cCOZ3gPIfrgK/+h9qK2hZt50ygcN3WG5NsefydapD0H11u0Tq7eQfft1qKkuXuBGtP7TXf/NKYXXb8HiNrNdzax/E9OL+Oe7aT179eC95SvrxpevWEXPnj1STBRfqWf/YM1aenTrWjfevVsXPljz4Q7LPTt1OqPPvoBLr/oZq95fA8DBB3yJwwYfyMiTz2LkyWdx1OGD6d+3uFtO1qkzvm57Xl//IdZp9wbLZHr2I9OpM7Wvzy5qtji0/ktbKbx+8xa3mZ0OvA48YGYLzeywerMnFLpTMxtjZrPMbFY2+2nrJJWSMmL44Txz/wT+evdvOeKwwVz1s5sAeHf5St56+z0m//UennvoT7wyex6z5y5IOW0jZrT95jlseXxC2klaTOv/i63QFve/AYe6+8HA/wHuMbNtnzJYoTt19zvdfYi7D8lkOrRO0kZWrlhN7z171o3v2WsPVq5cnchjtbZSz96taxdWf7Cmbvz9D9bSrWvnBstUddqVNm3aAPDtUcez6I3c/r2/TX2Rg/bfj/btK2nfvpLhw4Ywb+Hi4oUn2sKr2p7XOnXG13+0fYG2lWR69KFyzP+j/Y9/R6bPQNqdc+X2D8hSpvVf2krh9VuouMvcfRWAu78CjASuNrOxQPE/pm7k1VlzGTBgb/r27U1FRQWnn34Kjz72TNqxYin17AfsN5B3l69k+crVVFdX8+TkqYwcPqzBMmvWbn8hPj99Jv326g3AHt27MmvufGpqaqmuqWHW3Pl184olu/xNMp33wHbrBmXllB80nNrFr25fYPNGPv3pOWy8/nw2Xn8+2XeXsHnCdWRXLC1qzny0/ktbKbx+Cx1V8omZ9Xf3pQDuvsrMRgAPAfsnH62w2tpaLhl3NU88fi9lmQwT/vgXFi1a0vw3loBSz15eXsa/XXoB5112NbW1tYz+5nEM6LcXt/7+bvbfbyAjjx7GnyY9zJTpMykrL6PTLrvws6t/BMBxI4fzypx5jD77Asxg+OFDGNGodBKXzbLl4T9Qee5PIJOh+tXJZN9/jzZf/y61y5c2LJEmtP/x77B2lbnS2f/w3NERjY+ISJDWf7rrvzml8Pq1fMd4mtlBwKfu/maj6RXA6e7+5zgPUN6mV+pb519km1ZOSzvCTtlyw/i0I+yUtpffmHaEFgt93Vfd/HLaEXZKzdYVeXdJ593idvd5eaZXA7FKW0REWl+wx3GLiHxRqbhFRAITq7jNrNLM9k06jIiINK/Z4jazUcBc4Klo/GAzeyThXCIikkecLe5rgKHAOgB3nwvsnVgiEREpKE5xV7v7+kbTdIifiEhK4pzWdaGZnQmUmdk+wFjgxWRjiYhIPnG2uC8m95+SW4B7gfXAuAQziYhIAc1ucbv7RuCq6CYiIimLc1TJs2ZWVW98NzN7OtFUIiKSV5xdJV3cfd22EXf/B9AtsUQiIlJQnOLOmlndJTTMbC90VImISGriHFVyFTDdzKaSu4DC0cCYRFOJiEhecT6cfMrMBgPbTuo7zt3XJhtLRETyibPFDdAW+ChafpCZ4e4vJBdLRETyaba4zex64AxgIZCNJjug4hYRSUGcLe5TgX3dfUvCWUREJIY4R5W8BVQkHUREROKJs8W9EZhrZpPJ/ds7AO4+NrFUIiKSV5zifiS6iYhICYhzOOAfzawS6OPubxQhk4iIFKAr4IiIBKalV8Dpl1giEREpqKVXwMk2uaSIiCROV8AREQlMS6+Ac0mSoUREJL84W9wnuXuDK+CY2WnApMRSiYhIXnG2uK+MOU1ERIog7xa3mZ0AnAj0MrNb6s3aFahJOpiIiDSt0K6SlcAs4GRgdr3pnwCXJhlKRETyy1vc7j4PmGdm97p7dREziYhIAXE+nBxqZtcAe0XLG+Durn/CERFJQZzivovcrpHZQG2ycUREpDlxinu9uz+ZeBIREYklTnE/b2Y3AA/S8HzccxJLJSIiecUp7sOjr0PqTXPg2NaPIyIizYlzPu6RxQgiIiLxxDkfd3czu8vMnozGB5nZuclHExGRpsT5l/cJwNNAz2h8CTAuoTwiItKMOMXdxd0nEp2D291r0GGBIiKpiVPcn5pZZ3IfSGJmw8id2lVERFIQ56iSy8hd5b2/mc0AugLfSTSViIjkFeeokjlmdgywL7l/d39D5y4REUlP3l0lZnaYmfWAuv3ahwLXAjeZ2e5FyiciIo0U2sd9B7AVwMy+AvwCuJvc/u07k48mIiJNKbSrpMzdP4qGzwDudPcHgAfMbG7iyUREpEmFtrjLzGxbsX8VeK7evDgfaoqISAIKFfB9wFQzWwtsAqYBmNkAdDigiEhqCl0B51ozmwzsATzj7h7NygAXFyOciIjsyLb3cZjMbIy7B/thqfKnK+T8IWcH5d8Zcf5zstSNSTvATlL+dIWcP+TsoPwt9nkobhGRLxQVt4hIYD4PxR3sPrKI8qcr5PwhZwflb7HgP5wUEfmi+TxscYuIfKGouEVEAqPi/oIys75mtiDtHEkxs7FmttjMVpjZrdG0883s7LSzxVEv/58/w/c8YWZV0e1fkswXl5ltiL72NLP7o+Fztv1OSk39dVc/c6nRPu4iM7Myd6/NN17EHH2Bx9z9gGI/djGY2evA16LbEHe/KOVIn8m2/O6+vN608ugUy819b19K5HdrZhvcvWOjaedQor+TUlp3hQS1xW1mD5nZbDNbaGZjomkbzOxaM5tnZjPNrHuJZrzJzOYBRzQxfpmZLYhu46LvudzMxkbDN5vZc9HwsZ9lK6wZ5Wb252jL7n4za29mPzGzV6Msd5qZRY87xcyuN7NXzGyJmR0dTe9rZtPMbE50OzKaPiL6nvvN7PXocbbdV5OP0VrM7HdAP+BJYLd6068xs/HRcH8zeyr6XU0zs/2i6adFueaZ2Qutmasl+c1svZndY7mrT93TeGvVzB4zsxHR8Ntm1oXcKZj7m9lcM7shhR9hB/ne4ZnZSWb2kpl1MbPjouE5ZjbJzDo2dV8Jq7/uJm3LHK33h8zs2Wg9XxS9bl+Lemf3aLkmn1etzt2DuQG7R18rgQXAtmthjoqm/xK4ukQznl5vmbpxcheomA90ADoCC4FDgGHApGiZacArQAXwH8B5rZCzb5TjqGj8v4Hx2/JH0+6pt26nADdFwycCf4uG2wPtouF9gFnR8AhyJyPbk9wGwkvA8PrrqPFjtPLv4W2gC3AOcGs07RpgfDQ8GdgnGj4ceC4ang/0ioarUnwebct/DTAbqIym1/080fhjwIhG39MXWJDm66Bevg31nm8L6v8MwOjoub1blPsFoEO0zI+Bn6SQt37OxpnfBHYhd/nG9cD50bybgXGFnletfQvt9KxjzWx0NNybXFFsJffkhdwT/OtpBKunqYy1wAP1lqk/Phz4q7t/CmBmDwJHA78FDjWzXYEtwBxgSDRvbCtlfc/dZ0TDf4rud5mZXUGukHcn94fk0WiZB6Ovs8k9qSH3x+RWMzs4+rkG1rv/Vzx6q2+5c7j3BaYDIws8RuKiLbkjgUn1NvbbRl9nABPMbCLbf960PeLum9IO0cqOJfd8Ps7dPzazbwKDgBnR76QNuT/2peR5d/8E+MTM1rP9OTsfOLCZ51WrCqa4o7eDXwOOcPeNZjYFaAdUe/TnjVxxpPYzFci42Rvux248vgN3rzazZeT+0r8I/A8wEhgALG6lyI0/4HDgdnL7H98zs2vI5d9mS/S1/nq+FHgfOIjclvXmJpav+x4za9fMYxRDBljn7gc3nuHu55vZ4cBJwGwzO9TdPyxyvsY+rTdcQ8NdnMVed61lKbndQQOBWeSuZ/usu38v1VSF1X8+Z+uNZ8m9HvI+r1pbSPu4OwH/iApxP3K7EkpNSzJOA06N9i93YPvbx23zxpN7CzkNOB94rd4fqp3Vx8yOiIbPJLc1DLA22nr4Toz76ASscvcs8M9AWTPLbyuaz/IYrcrdPyb3zuI0AMs5KBru7+4vu/tPgDXk3jWVkreBg80sY2a9gaFNLPMJubf0pewd4NvA3Wa2PzATOMpy5/vHzDqY2cBCd5CQFq+7Qs+r1hZScT9FbottMbkPEGamnKcpnzmju88BJpDbh/0y8Ad3fy2aPY3c+dBfcvf3yW3NTmvqflroDeDCKO9u5HbP/J7cvvmngVdj3MftwPct90HrfjTcOtyBu69rwWMk4Szg3Cj3QuCUaPoNZjY/+lDqRWBeSvnymQEsAxYBt5DbhdZA9A5hRvQha0l8ONkUd3+d3O9hErAruXeX95nZ/5DbTZLMB3uFM9WtO6Al6y7f86pV6XBAEZHAhLTFLSIiqLhFRIKj4hYRCYyKW0QkMCpuEZHAqLhFRAKj4hYRCcz/AuBXMKwX2OlBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vectorizer = TfidfVectorizer(binary=True)\n",
    "one_hot = tfidf_vectorizer.fit_transform(corpus).toarray()\n",
    "sns.heatmap(\n",
    "    one_hot, annot=True,\n",
    "    cbar=False, xticklabels=sorted(list(tfidf_vectorizer.vocabulary_.keys())),\n",
    "    yticklabels=['Sentence 1', 'Sentence 2']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3d4ab67",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Computational Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6da15ba",
   "metadata": {
    "hidden": true
   },
   "source": [
    "_Computational Graph_ is an abstraction that models mathematical expressions. Libraries like Theano, TensorFlow & PyTorch do additional book keeping to implement Automatic Differentiation needed to obtain gradients of parameters during training in the supervised learning paradigm.\n",
    "\n",
    "[Seppo Linnainmaa](https://en.wikipedia.org/wiki/Seppo_Linnainmaa) first introduced the idea of automatic differentiation on computationl graphs as part of his 1970 master thesis. Variants of that became the foundation for modern deep learning frameworks like Theano, TensorFlow & PyTorch.\n",
    "\n",
    "_Inference_ is simply expression evaluation. Consider following expressions\n",
    "\n",
    "$$ y = wx + b $$\n",
    "\n",
    "This can be writeen as two subexpressions, $ z = wx $ and $ y = z + b $. Then this can represented using a DAG(directed acyclic graph) in which nodes are the mathematical operations like multiplication & addition.\n",
    "\n",
    "![Figure 1.4](../images/figure1.4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3cccab",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## PyTorch Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb739dd",
   "metadata": {
    "hidden": true
   },
   "source": [
    "PyTorch implements a tape based [automatic differentiation method](https://justindomke.wordpress.com/2009/03/24/a-simple-explanation-of-reverse-mode-automatic-differentiation/) that allows us to define and execute computational graphs dynamically.\n",
    "\n",
    "_Tensor_ is a mathematical object holding some multidimensional data. A tensor of order zero is just a number or a _scalar_. A tensor of order one(1st ordet tensor) is an array of numbers, or a _vector_. Similarly, a 2nd order tensor is an array of vectors or a _matrix_.Hence tensor can be generalized as an _n-dimensioal_ array of scalars."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e9dd04",
   "metadata": {
    "hidden": true
   },
   "source": [
    "![Figure 1.5](../images/figure1.5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ba46ba",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Creating Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "da2d4db7",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def describe(x):\n",
    "    print(f\"Type: {x.type()}\")\n",
    "    print(f\"Shape/Size: {x.shape}\")\n",
    "    print(f\"Values: \\n{x}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "84dfecfa",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[4.1911e+21, 1.2983e-11, 1.3399e-08],\n",
      "        [2.1122e+20, 1.3107e-08, 5.1179e+22]])\n"
     ]
    }
   ],
   "source": [
    "# Creating a tensor with torc.Tensor\n",
    "import torch\n",
    "describe(torch.Tensor(2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "db628449",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0.8576, 0.7484, 0.3410],\n",
      "        [0.1979, 0.1658, 0.2336]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[-0.6241, -0.5137,  0.3788],\n",
      "        [ 0.0450, -1.1730,  1.0697]])\n"
     ]
    }
   ],
   "source": [
    "# Creating a randomly initialized tensor\n",
    "\n",
    "# Uniform random\n",
    "describe(torch.rand(2, 3))\n",
    "\n",
    "# random normal\n",
    "describe(torch.randn(2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "57d8b4a5",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]])\n"
     ]
    }
   ],
   "source": [
    "# Creating a filled tensor\n",
    "describe(torch.zeros(2, 3))\n",
    "x = torch.ones(2, 3)\n",
    "describe(x)\n",
    "\n",
    "# Method with underscore refers to an in-place operation\n",
    "x.fill_(5)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e20d133a",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 2., 3.],\n",
      "        [4., 5., 6.]])\n"
     ]
    }
   ],
   "source": [
    "# Creating and initializing a tensor from lists\n",
    "x = torch.Tensor(\n",
    "    [\n",
    "        [1, 2, 3],\n",
    "        [4, 5, 6]\n",
    "    ]\n",
    ")\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4c179f78",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.DoubleTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0.6170, 0.6025, 0.9243],\n",
      "        [0.4747, 0.6294, 0.7599]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "# Creating and initializing a tensor from NumPy\n",
    "import numpy as np\n",
    "npy = np.random.rand(2, 3)\n",
    "describe(torch.from_numpy(npy))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db8f20c",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tensor Types & Size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f957f6de",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 2., 3.],\n",
      "        [4., 5., 6.]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.FloatTensor(\n",
    "    [\n",
    "        [1, 2, 3],\n",
    "        [4, 5, 6]\n",
    "    ]\n",
    ")\n",
    "describe(x)\n",
    "x = x.long()\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "619d0360",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 2., 3.],\n",
      "        [4., 5., 6.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor(\n",
    "    [\n",
    "        [1, 2, 3],\n",
    "        [4, 5, 6]\n",
    "    ]\n",
    ")\n",
    "describe(x)\n",
    "x = x.float()\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55a46c56",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tensor Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "94595a9a",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[-3.1600,  0.3143,  0.8178],\n",
      "        [-0.7756,  0.2878,  0.1736]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[-6.3199,  0.6286,  1.6356],\n",
      "        [-1.5512,  0.5756,  0.3472]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[-6.3199,  0.6286,  1.6356],\n",
      "        [-1.5512,  0.5756,  0.3472]])\n"
     ]
    }
   ],
   "source": [
    "# Tensor Operations: addition\n",
    "x = torch.randn(2, 3)\n",
    "describe(x)\n",
    "describe(torch.add(x, x))\n",
    "describe(x + x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "eea9198f",
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([6])\n",
      "Values: \n",
      "tensor([0, 1, 2, 3, 4, 5])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([3])\n",
      "Values: \n",
      "tensor([3, 5, 7])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([ 3, 12])\n"
     ]
    }
   ],
   "source": [
    "# Dimension Based Tensor Operations\n",
    "x = torch.arange(6)\n",
    "describe(x)\n",
    "x = x.view(2, 3)\n",
    "describe(x)\n",
    "describe(torch.sum(x, dim=0))\n",
    "describe(torch.sum(x, dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f50673e",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Indexing, Slicing & Joining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "62f1330b",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([1, 2])\n",
      "Values: \n",
      "tensor([[0, 1]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([])\n",
      "Values: \n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# Slicing & Indexing a Tensor\n",
    "x = torch.arange(6).view(2, 3)\n",
    "describe(x)\n",
    "describe(x[:1, :2])\n",
    "describe(x[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e6038644",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 2])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[0, 2],\n",
      "        [3, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 0])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [0, 1, 2]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 1])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 1])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 4])\n"
     ]
    }
   ],
   "source": [
    "# Complex Indexing: Noncontiguous Indexing of a tensor\n",
    "indices = torch.LongTensor([0, 2])\n",
    "describe(indices)\n",
    "describe(torch.index_select(x, dim=1, index=indices))\n",
    "\n",
    "indices = torch.LongTensor([0, 0])\n",
    "describe(indices)\n",
    "describe(torch.index_select(x, dim=0, index=indices))\n",
    "\n",
    "row_indices = torch.arange(2).long()\n",
    "col_indices = torch.LongTensor([0, 1])\n",
    "describe(row_indices)\n",
    "describe(col_indices)\n",
    "describe(x[row_indices, col_indices])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b877cdcf",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 6])\n",
      "Values: \n",
      "tensor([[0, 1, 2, 0, 1, 2],\n",
      "        [3, 4, 5, 3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([4, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5],\n",
      "        [0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 2, 3])\n",
      "Values: \n",
      "tensor([[[0, 1, 2],\n",
      "         [3, 4, 5]],\n",
      "\n",
      "        [[0, 1, 2],\n",
      "         [3, 4, 5]]])\n"
     ]
    }
   ],
   "source": [
    "# Concatenating Tensors\n",
    "x = torch.arange(6).view(2, 3)\n",
    "describe(x)\n",
    "describe(torch.cat([x, x], dim=1))\n",
    "describe(torch.cat([x, x], dim=0))\n",
    "describe(torch.stack([x, x]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e23bd9fe",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 2])\n",
      "Values: \n",
      "tensor([[1., 2.],\n",
      "        [1., 2.],\n",
      "        [1., 2.]])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[ 3,  6],\n",
      "        [12, 24]])\n"
     ]
    }
   ],
   "source": [
    "# Linear Algebra on Tensors: Multiplication\n",
    "\n",
    "x1 = torch.arange(6).view(2, 3)\n",
    "describe(x1)\n",
    "x2 = torch.ones(3, 2)\n",
    "x2[:, 1] += 1\n",
    "describe(x2)\n",
    "describe(torch.mm(x1, x2.long()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07586830",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tensors & Computational Graphs\n",
    "\n",
    "Gradient is a value that represents the slope of a function output with respect to the function input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "a54e3851",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "None\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[21., 21.],\n",
      "        [21., 21.]], grad_fn=<AddBackward0>)\n",
      "None\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([])\n",
      "Values: \n",
      "21.0\n",
      "tensor([[2.2500, 2.2500],\n",
      "        [2.2500, 2.2500]])\n"
     ]
    }
   ],
   "source": [
    "# Creating tensors for gradient bookkeeping\n",
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "describe(x)\n",
    "print(x.grad)\n",
    "\n",
    "y = (x + 2) * (x + 5) + 3\n",
    "describe(y)\n",
    "print(x.grad)\n",
    "\n",
    "z = y.mean()\n",
    "describe(z)\n",
    "z.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344ad490",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### CUDA Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "f9f49888",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "cpu\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.1819, 0.0941, 0.2468],\n",
      "        [0.0304, 0.7735, 0.5676],\n",
      "        [0.6932, 0.5248, 0.4273]])\n"
     ]
    }
   ],
   "source": [
    "# Creating CUDA Tensors\n",
    "print(torch.cuda.is_available())\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)\n",
    "x = torch.rand(3, 3).to(device)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "8a23aff9",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.1622, 0.8481, 1.1289],\n",
       "        [0.6761, 1.2577, 1.1936],\n",
       "        [1.5976, 1.1787, 1.3901]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mixing CUDA tensors with CPU bound tensors\n",
    "y = torch.rand(3, 3)\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "dd6c0447",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.1622, 0.8481, 1.1289],\n",
       "        [0.6761, 1.2577, 1.1936],\n",
       "        [1.5976, 1.1787, 1.3901]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = y.to(device)\n",
    "x = x.to(device)\n",
    "x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "673663ff",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "0f028913",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.9948, 0.2192, 0.2748],\n",
      "        [0.5520, 0.3972, 0.0348],\n",
      "        [0.3086, 0.8186, 0.6221]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.9948, 0.2192, 0.2748],\n",
      "        [0.5520, 0.3972, 0.0348],\n",
      "        [0.3086, 0.8186, 0.6221]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([1, 3, 3])\n",
      "Values: \n",
      "tensor([[[0.9948, 0.2192, 0.2748],\n",
      "         [0.5520, 0.3972, 0.0348],\n",
      "         [0.3086, 0.8186, 0.6221]]])\n"
     ]
    }
   ],
   "source": [
    "# 1. Create a 2D tensor and then add a dimension of size 1 inserted at dimension 0.\n",
    "\n",
    "x = torch.rand(3, 3)\n",
    "describe(x)\n",
    "a = x.unsqueeze(0)\n",
    "describe(x)\n",
    "describe(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "dd8c5125",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.9948, 0.2192, 0.2748],\n",
      "        [0.5520, 0.3972, 0.0348],\n",
      "        [0.3086, 0.8186, 0.6221]])\n"
     ]
    }
   ],
   "source": [
    "# 2. Remove the extra dimension you just added to the previous tensor.\n",
    "\n",
    "a = a.squeeze(0)\n",
    "describe(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "254258cc",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([5, 3])\n",
      "Values: \n",
      "tensor([[6.1084, 4.5564, 6.2045],\n",
      "        [6.2692, 5.9804, 4.6833],\n",
      "        [3.1471, 4.1873, 3.7031],\n",
      "        [6.0901, 6.8825, 3.7219],\n",
      "        [6.9564, 3.1241, 4.5057]])\n"
     ]
    }
   ],
   "source": [
    "# 3. Create a random tensor of shape 5x3 in the interval [3, 7)\n",
    "\n",
    "describe(3 + torch.rand(5, 3) * (7 - 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "d7d8274b",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.1263, 0.0134, 0.6546],\n",
      "        [0.9368, 0.0811, 0.2582],\n",
      "        [0.4380, 0.6274, 0.4826]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[ 0.3927, -1.3650, -1.3759],\n",
      "        [ 0.3783,  1.0076,  0.2306],\n",
      "        [-0.1471,  0.0136,  0.5349]])\n"
     ]
    }
   ],
   "source": [
    "# 4. Create a tensor with values from a normal distribution (mean=0, std=1).\n",
    "\n",
    "a = torch.rand(3, 3)\n",
    "describe(a)\n",
    "a.normal_()\n",
    "describe(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "ef7f6c35",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([5])\n",
      "Values: \n",
      "tensor([1., 1., 1., 0., 1.])\n",
      "Type: torch.LongTensor\n",
      "Shape/Size: torch.Size([4, 1])\n",
      "Values: \n",
      "tensor([[0],\n",
      "        [1],\n",
      "        [2],\n",
      "        [4]])\n"
     ]
    }
   ],
   "source": [
    "# 5. Retrieve the indexes of all the nonzero elements in the tensor torch.Tensor([1, 1, 1, 0, 1]).\n",
    "\n",
    "a = torch.Tensor([1, 1, 1, 0, 1])\n",
    "describe(a)\n",
    "describe(a.nonzero())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "40461299",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 1])\n",
      "Values: \n",
      "tensor([[0.1885],\n",
      "        [0.9197],\n",
      "        [0.2011]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 4])\n",
      "Values: \n",
      "tensor([[0.1885, 0.1885, 0.1885, 0.1885],\n",
      "        [0.9197, 0.9197, 0.9197, 0.9197],\n",
      "        [0.2011, 0.2011, 0.2011, 0.2011]])\n"
     ]
    }
   ],
   "source": [
    "# 6. Create a random tensor of size (3,1) and then horizontally stack four copies together.\n",
    "\n",
    "a = torch.rand(3, 1)\n",
    "describe(a)\n",
    "describe(a.expand(3, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "7681b1c0",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 4, 5])\n",
      "Values: \n",
      "tensor([[[0.3564, 0.1567, 0.9715, 0.0778, 0.2263],\n",
      "         [0.9015, 0.1654, 0.2525, 0.4174, 0.4270],\n",
      "         [0.5057, 0.8296, 0.9204, 0.0790, 0.2932],\n",
      "         [0.9185, 0.9262, 0.1190, 0.9915, 0.9216]],\n",
      "\n",
      "        [[0.1639, 0.1751, 0.1474, 0.8851, 0.8939],\n",
      "         [0.2789, 0.7392, 0.6716, 0.3847, 0.8202],\n",
      "         [0.0227, 0.0124, 0.9194, 0.8144, 0.5921],\n",
      "         [0.9525, 0.9549, 0.0662, 0.1525, 0.1277]],\n",
      "\n",
      "        [[0.9203, 0.7173, 0.6867, 0.1012, 0.1112],\n",
      "         [0.6357, 0.2192, 0.2895, 0.2133, 0.8876],\n",
      "         [0.3152, 0.2038, 0.1371, 0.1724, 0.6423],\n",
      "         [0.3159, 0.6188, 0.8418, 0.4031, 0.8775]]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 5, 4])\n",
      "Values: \n",
      "tensor([[[0.7862, 0.1557, 0.8397, 0.6308],\n",
      "         [0.0601, 0.5460, 0.2286, 0.7408],\n",
      "         [0.1278, 0.8643, 0.9329, 0.2279],\n",
      "         [0.2715, 0.4812, 0.1460, 0.2860],\n",
      "         [0.2923, 0.7213, 0.2361, 0.5052]],\n",
      "\n",
      "        [[0.5088, 0.4569, 0.0815, 0.7030],\n",
      "         [0.5685, 0.6959, 0.3629, 0.9032],\n",
      "         [0.4623, 0.1116, 0.7319, 0.2846],\n",
      "         [0.2337, 0.7324, 0.6030, 0.4305],\n",
      "         [0.3837, 0.1924, 0.5878, 0.1796]],\n",
      "\n",
      "        [[0.6449, 0.0109, 0.1624, 0.6048],\n",
      "         [0.7178, 0.9046, 0.5459, 0.7848],\n",
      "         [0.0976, 0.1943, 0.0845, 0.4416],\n",
      "         [0.6295, 0.2312, 0.8712, 0.0404],\n",
      "         [0.4717, 0.9231, 0.2384, 0.6854]]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 4, 4])\n",
      "Values: \n",
      "tensor([[[0.5010, 1.1813, 1.3061, 0.6989],\n",
      "         [0.9892, 0.9578, 1.1921, 1.0839],\n",
      "         [0.6722, 1.5767, 1.5537, 1.3140],\n",
      "         [1.3316, 1.8934, 1.4563, 2.0419]],\n",
      "\n",
      "        [[0.8009, 1.0334, 1.2439, 0.8569],\n",
      "         [1.2772, 1.1563, 1.4965, 1.3678],\n",
      "         [0.8611, 0.8319, 1.5183, 0.7457],\n",
      "         [1.1427, 1.2434, 0.6397, 1.6396]],\n",
      "\n",
      "        [[1.2915, 0.9184, 0.7137, 1.5031],\n",
      "         [1.1485, 1.1302, 0.6448, 1.3013],\n",
      "         [0.7745, 0.8472, 0.4773, 0.8583],\n",
      "         [1.3977, 1.6301, 1.0206, 1.6662]]])\n"
     ]
    }
   ],
   "source": [
    "# 7. Return the batch matrix-matrix product of two three-dimensional matrices (a=torch.rand(3,4,5), b=torch.rand(3,5,4)).\n",
    "\n",
    "a = torch.rand(3, 4, 5)\n",
    "b = torch.rand(3, 5, 4)\n",
    "describe(a)\n",
    "describe(b)\n",
    "describe(torch.bmm(a, b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "9f94d9fc",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 4, 5])\n",
      "Values: \n",
      "tensor([[[0.5307, 0.8954, 0.3231, 0.0083, 0.6625],\n",
      "         [0.0679, 0.2234, 0.6733, 0.4813, 0.3997],\n",
      "         [0.5609, 0.9456, 0.8110, 0.3193, 0.5541],\n",
      "         [0.5484, 0.4430, 0.4566, 0.4791, 0.2792]],\n",
      "\n",
      "        [[0.8887, 0.7611, 0.3875, 0.8849, 0.0490],\n",
      "         [0.8606, 0.4816, 0.3886, 0.0110, 0.7618],\n",
      "         [0.9859, 0.5685, 0.1761, 0.7831, 0.4973],\n",
      "         [0.5241, 0.1780, 0.7988, 0.7058, 0.5777]],\n",
      "\n",
      "        [[0.1751, 0.6022, 0.5638, 0.2586, 0.7211],\n",
      "         [0.8273, 0.4658, 0.0716, 0.5146, 0.9314],\n",
      "         [0.2133, 0.4027, 0.4810, 0.0385, 0.9646],\n",
      "         [0.4136, 0.7462, 0.3778, 0.2873, 0.4125]]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([5, 4])\n",
      "Values: \n",
      "tensor([[0.9210, 0.5668, 0.7346, 0.3976],\n",
      "        [0.3458, 0.3803, 0.1340, 0.1006],\n",
      "        [0.8108, 0.1976, 0.1804, 0.5371],\n",
      "        [0.4929, 0.6887, 0.7709, 0.1628],\n",
      "        [0.3428, 0.8578, 0.2044, 0.2830]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/Size: torch.Size([3, 4, 4])\n",
      "Values: \n",
      "tensor([[[1.2915, 1.2792, 0.7099, 0.6635],\n",
      "         [1.0599, 0.9308, 0.6540, 0.6026],\n",
      "         [1.8484, 1.5330, 1.0444, 0.9625],\n",
      "         [1.3603, 1.1390, 0.9710, 0.6649]],\n",
      "\n",
      "        [[1.8487, 1.5213, 1.5169, 0.7960],\n",
      "         [1.5408, 1.4088, 0.9311, 0.8167],\n",
      "         [1.8038, 1.7758, 1.5376, 0.8120],\n",
      "         [1.7377, 1.5042, 1.2151, 0.9337]],\n",
      "\n",
      "        [[1.2012, 1.2363, 0.6577, 0.6792],\n",
      "         [1.5539, 1.8135, 1.2702, 0.7616],\n",
      "         [1.0754, 1.2230, 0.5243, 0.6630],\n",
      "         [1.2283, 1.1446, 0.7778, 0.6059]]])\n"
     ]
    }
   ],
   "source": [
    "# 8. Return the batch matrix-matrix product of a 3D matrix and a 2D matrix (a=torch.rand(3,4,5), b=torch.rand(5,4)).\n",
    "\n",
    "a = torch.rand(3, 4, 5)\n",
    "b = torch.rand(5, 4)\n",
    "describe(a)\n",
    "describe(b)\n",
    "describe(torch.bmm(a, b.unsqueeze(0).expand(a.size(0), *b.size())))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}