05_rnn_concepts.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sumani\n",
    "# 1-9-2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequential Models in PyTorch: RNN, LSTM, and GRU\n",
    "\n",
    "This tutorial will cover the basics of sequential models in PyTorch, specifically focusing on Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Gated Recurrent Units (GRUs). We'll start with a brief overview of each model, followed by simple examples using tensors to illustrate their operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction to Sequential Models\n",
    "\n",
    "Sequential models are designed to process sequences of data, such as time series or natural language. Unlike feedforward neural networks, sequential models retain information from previous inputs in the sequence, making them suitable for tasks where the order of inputs matters.\n",
    "\n",
    "### 1.1 Recurrent Neural Networks (RNN)\n",
    "\n",
    "RNNs are a type of neural network designed to handle sequential data by maintaining a hidden state that captures information from previous steps in the sequence.\n",
    "\n",
    "### 1.2 Long Short-Term Memory Networks (LSTM)\n",
    "\n",
    "LSTMs are a special type of RNN designed to address the vanishing gradient problem. They use gates to control the flow of information, making them capable of learning long-term dependencies.\n",
    "\n",
    "### 1.3 Gated Recurrent Units (GRU)\n",
    "\n",
    "GRUs are similar to LSTMs but are simpler and faster to train. They combine the forget and input gates into a single update gate, reducing the number of parameters.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. PyTorch Implementation of RNN, LSTM, and GRU\n",
    "\n",
    "We'll implement each of these models using PyTorch and simple tensors to demonstrate how they work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Recurrent Neural Networks (RNN)\n",
    "\n",
    "In this example, we define a simple RNN with one input feature and two hidden units. The input sequence consists of three time steps. The RNN processes each time step and updates the hidden state accordingly. The final output and hidden state are printed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# Define the input tensor (batch_size, sequence_length, input_size)\n",
    "input_seq = torch.tensor([[[0.1], [0.2], [0.3]]], dtype=torch.float32)\n",
    "\n",
    "# Define an RNN layer\n",
    "rnn = nn.RNN(input_size=1, hidden_size=2, batch_first=True)\n",
    "\n",
    "# Initialize the hidden state\n",
    "hidden_state = torch.zeros(1, 1, 2)\n",
    "\n",
    "# Forward pass through the RNN\n",
    "output, hidden_state = rnn(input_seq, hidden_state)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"RNN Output:\", output)\n",
    "print(\"RNN Hidden State:\", hidden_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's pass the same input again\n",
    "output, hidden_state = rnn(input_seq, hidden_state)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"RNN Output:\", output)\n",
    "print(\"RNN Hidden State:\", hidden_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change Hidden State Size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the input tensor (batch_size, sequence_length, input_size)\n",
    "input_seq = torch.tensor([[[0.1], [0.2], [0.3]]], dtype=torch.float32)\n",
    "\n",
    "# Define an RNN layer with hidden_size=4\n",
    "rnn_hidden_4 = nn.RNN(input_size=1, hidden_size=4, batch_first=True)\n",
    "\n",
    "# Initialize the hidden state (num_layers, batch_size, hidden_size)\n",
    "hidden_state_4 = torch.zeros(1, 1, 4)\n",
    "\n",
    "# Forward pass through the RNN\n",
    "output_4, hidden_state_4 = rnn_hidden_4(input_seq, hidden_state_4)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"RNN with hidden_size=4 Output:\", output_4)\n",
    "print(\"RNN with hidden_size=4 Hidden State:\", hidden_state_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi Layer RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the input tensor (batch_size, sequence_length, input_size)\n",
    "input_seq = torch.tensor([[[0.1], [0.2], [0.3]]], dtype=torch.float32)\n",
    "\n",
    "# Define an RNN layer with 3 stacked layers and hidden_size=2\n",
    "rnn_stacked = nn.RNN(input_size=1, hidden_size=2, num_layers=3, batch_first=True)\n",
    "\n",
    "# Initialize the hidden state (num_layers, batch_size, hidden_size)\n",
    "hidden_state_stacked = torch.zeros(3, 1, 2)  # 3 layers, batch_size=1, hidden_size=2\n",
    "\n",
    "# Forward pass through the Multi Layer RNN\n",
    "output_stacked, hidden_state_stacked = rnn_stacked(input_seq, hidden_state_stacked)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"Multi Layer RNN Output:\", output_stacked)\n",
    "print(\"Multi Layer RNN Hidden State:\", hidden_state_stacked)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Long Short-Term Memory Networks (LSTM)\n",
    "\n",
    "The LSTM example is similar to the RNN example, but with an additional cell state that helps maintain long-term dependencies. The output, hidden state, and cell state are printed after processing the input sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define an LSTM layer\n",
    "lstm = nn.LSTM(input_size=1, hidden_size=2, batch_first=True)\n",
    "\n",
    "# Initialize the hidden state and cell state\n",
    "hidden_state = torch.zeros(1, 1, 2)\n",
    "cell_state = torch.zeros(1, 1, 2)\n",
    "\n",
    "# Forward pass through the LSTM\n",
    "output, (hidden_state, cell_state) = lstm(input_seq, (hidden_state, cell_state))\n",
    "\n",
    "# Print the output, hidden state, and cell state\n",
    "print(\"LSTM Output:\", output)\n",
    "print(\"LSTM Hidden State:\", hidden_state)\n",
    "print(\"LSTM Cell State:\", cell_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forward pass through the LSTM\n",
    "output, (hidden_state, cell_state) = lstm(input_seq, (hidden_state, cell_state))\n",
    "\n",
    "# Print the output, hidden state, and cell state\n",
    "print(\"LSTM Output:\", output)\n",
    "print(\"LSTM Hidden State:\", hidden_state)\n",
    "print(\"LSTM Cell State:\", cell_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Gated Recurrent Units (GRU)\n",
    "\n",
    "In the GRU example, we define a GRU layer with one input feature and two hidden units. The GRU is simpler than the LSTM, with fewer parameters. The output and hidden state are printed after processing the input sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a GRU layer\n",
    "gru = nn.GRU(input_size=1, hidden_size=2, batch_first=True)\n",
    "\n",
    "# Initialize the hidden state\n",
    "hidden_state = torch.zeros(1, 1, 2)\n",
    "\n",
    "# Forward pass through the GRU\n",
    "output, hidden_state = gru(input_seq, hidden_state)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"GRU Output:\", output)\n",
    "print(\"GRU Hidden State:\", hidden_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forward pass through the GRU\n",
    "output, hidden_state = gru(input_seq, hidden_state)\n",
    "\n",
    "# Print the output and hidden state\n",
    "print(\"GRU Output:\", output)\n",
    "print(\"GRU Hidden State:\", hidden_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}