2-Basic Training Loop on Toy Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Let's see a real scenario on a toy dataset"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you are using a GPU (if you have one definitely use it)  \n",
    "You should know one thing about the GPU training.  \n",
    "Your data and your model should be located at the same place to start training.  \n",
    "You can always move your data or model by using .to(device)\n",
    "or give function to as argument"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "# Device configuration\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's set our hyper parameters which are the most important things for our training"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# Hyper-parameters \n",
    "input_size = 28 * 28    # 784\n",
    "num_classes = 10\n",
    "hidden_size = 500\n",
    "num_epochs = 10\n",
    "batch_size = 128\n",
    "learning_rate = 0.001"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's use our datasets"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "# MNIST dataset (images and labels)\n",
    "train_dataset = torchvision.datasets.MNIST(root='data', \n",
    "                                           train=True, \n",
    "                                           transform=transforms.ToTensor(),\n",
    "                                           download=True)\n",
    "\n",
    "test_dataset = torchvision.datasets.MNIST(root='data', \n",
    "                                          train=False, \n",
    "                                          transform=transforms.ToTensor())\n",
    "                                          \n",
    "\n",
    "# Data loader (input pipeline)\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, \n",
    "                                           batch_size=batch_size, \n",
    "                                           shuffle=True)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset, \n",
    "                                          batch_size=batch_size, \n",
    "                                          shuffle=False)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "print(len(train_loader), len(test_loader))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "469 79\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We know how to use trained torch models but what if we want to create a custom model.  \n",
    "P.S. network architecture creation is very hard if your knowledge is limited. So customizing known architectures is good place to start."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "# Fully connected neural network with one hidden layer\n",
    "class NeuralNet(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, num_classes):\n",
    "        super(NeuralNet, self).__init__()\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size) \n",
    "        self.fc2 = nn.Linear(hidden_size, num_classes)  \n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Sometimes you create so much layers. Giving unique name for each one is hard so we use nn.Sequential for those cases."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "# Fully connected neural network with one hidden layer\n",
    "class NeuralNet(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, num_classes):\n",
    "        super(NeuralNet, self).__init__()\n",
    "        self.fc = nn.Sequential(\n",
    "                nn.Linear(input_size, hidden_size),\n",
    "                nn.ReLU(),\n",
    "                nn.Linear(hidden_size, num_classes)\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out = self.fc(x)\n",
    "        return out"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "# We create the model and move it to the device (GPU or CPU) default is CPU\n",
    "model = NeuralNet(input_size, hidden_size, num_classes).to(device)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "# We define our loss function and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now important thing is we should write a training loop.  \n",
    "There could be so many different training loop as you want but every training loop has same simple logic\n",
    "\n",
    "1. Get data and move it to appropriate location\n",
    "2. Make a forward pass\n",
    "3. Calculate loss\n",
    "4. Zero the gradients\n",
    "5. Calculate gradients\n",
    "6. Optimize your network with one step gradient descent"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "total_step = len(train_loader)\n",
    "total_images = len(train_dataset)\n",
    "print(f\"Number of images: {total_images}, Number of batches: {total_step}\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Number of images: 60000, Number of batches: 469\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "# Train the model\n",
    "\n",
    "# At each epoch we iterate over all the dataset\n",
    "for epoch in range(num_epochs):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        # Image shapes are\n",
    "        # [Batch_size, Channel, Height, Width]\n",
    "        # Label shapes are\n",
    "        # [Batch_size]\n",
    "        # Move labels and images to the configured device\n",
    "        images = images.reshape(-1, 28*28).to(device)\n",
    "        labels = labels.to(device)\n",
    "        # Forward pass through the model\n",
    "        outputs = model(images)\n",
    "        # Calculate your loss\n",
    "        loss = criterion(outputs, labels)\n",
    "        \n",
    "        # Backward and optimize\n",
    "        # If you don't accumulate the gradients then you should\n",
    "        # zero grads before calculate them\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        # Make a step with your optimizer\n",
    "        optimizer.step()\n",
    "        \n",
    "        # To monitor the training process we print or log some useful values \n",
    "        if (i+1) % 200 == 0:\n",
    "            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' \n",
    "                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Epoch [1/10], Step [200/469], Loss: 0.2074\n",
      "Epoch [1/10], Step [400/469], Loss: 0.1409\n",
      "Epoch [2/10], Step [200/469], Loss: 0.1043\n",
      "Epoch [2/10], Step [400/469], Loss: 0.1115\n",
      "Epoch [3/10], Step [200/469], Loss: 0.0740\n",
      "Epoch [3/10], Step [400/469], Loss: 0.1678\n",
      "Epoch [4/10], Step [200/469], Loss: 0.0417\n",
      "Epoch [4/10], Step [400/469], Loss: 0.0333\n",
      "Epoch [5/10], Step [200/469], Loss: 0.0418\n",
      "Epoch [5/10], Step [400/469], Loss: 0.0331\n",
      "Epoch [6/10], Step [200/469], Loss: 0.0141\n",
      "Epoch [6/10], Step [400/469], Loss: 0.0065\n",
      "Epoch [7/10], Step [200/469], Loss: 0.0288\n",
      "Epoch [7/10], Step [400/469], Loss: 0.0088\n",
      "Epoch [8/10], Step [200/469], Loss: 0.0199\n",
      "Epoch [8/10], Step [400/469], Loss: 0.0073\n",
      "Epoch [9/10], Step [200/469], Loss: 0.0074\n",
      "Epoch [9/10], Step [400/469], Loss: 0.0358\n",
      "Epoch [10/10], Step [200/469], Loss: 0.0043\n",
      "Epoch [10/10], Step [400/469], Loss: 0.0061\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "# Test the model\n",
    "# In test phase, we don't need to compute gradients (for memory efficiency)\n",
    "# no_grad context saves you about forgetting gradient freezing\n",
    "# Within no_grad context nothing will compute or keep gradients\n",
    "# If you don't use no_grad your memory and  time usage will be higher\n",
    "with torch.no_grad():\n",
    "    model.eval()\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    for images, labels in test_loader:\n",
    "        images = images.reshape(-1, 28*28).to(device)\n",
    "        labels = labels.to(device)\n",
    "        # If you don't use no_grad context you can use\n",
    "        # model.eval() function\n",
    "        # When you use it your model enters to evaluation mode (no grad calculation) \n",
    "        # Be careful some layers (BatchNorm) behaves different in training and evaluation mode \n",
    "        # You know we calculate local gradients when we do forward pass\n",
    "        \n",
    "        outputs = model(images)\n",
    "        \n",
    "        # Get predictions and calculate your accuracy\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "\n",
    "    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Accuracy of the network on the 10000 test images: 98.03 %\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Awesome in a less than a minute you solved digit recognition problem with high accuracy.  \n",
    "Let's save our model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "# Save the model checkpoint\n",
    "torch.save(model.state_dict(), 'model.ckpt')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.5 64-bit ('pytorch': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "metadata": {
   "interpreter": {
    "hash": "ebe9b73ffc5772acfd044802dad682a126aff30cac12fb04966daedd090a2fff"
   }
  },
  "interpreter": {
   "hash": "ebe9b73ffc5772acfd044802dad682a126aff30cac12fb04966daedd090a2fff"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}