00_tensor_fundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sumani\n",
    "# 20-7-2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction to Tensors\n",
    "\n",
    "A tensor represents a (possibly multidimensional) array of numerical values. \n",
    "1. In the zero dimensional case, a tensor is called a scalar. \n",
    "2. In the one dimensional case, i.e., when only one axis is needed for the data, a tensor is called a vector.\n",
    "3. With two axes, a tensor is called a matrix.\n",
    "4. With 𝑘 > 2 axes, we drop the specialized names and just refer to the object as a 𝑘 th order tensor.\n",
    "\n",
    "Complex computations often involve combining multiple tensor operations. For example, adding tensors, performing element-wise operations, reshaping, and aggregating results in a single sequence. Understanding how different operations interact and combining them effectively is crucial for efficient tensor manipulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To start, we import the PyTorch library. Note that the package name is torch.\n",
    "import torch\n",
    "torch.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.1 Scalars\n",
    "Scalars: Scalars are single numerical values. They have magnitude but no direction. Examples include temperature, mass, and time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scalar = torch.tensor(7)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scalar.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Vectors\n",
    "Vectors: Vectors are quantities with both magnitude and direction. They are represented by arrows. Examples include velocity, force, and displacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector = torch.tensor([7, 3, 7])\n",
    "vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Matrix\n",
    "Matrices: Matrices are 2-D arrays of numbers. They are used to represent data or transformations. A matrix has rows and columns. For example:\n",
    "\n",
    "[[7, 8], [9, 6]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matrix\n",
    "MATRIX = torch.tensor([[7, 8], \n",
    "                       [9, 10]])\n",
    "MATRIX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MATRIX.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Tensors\n",
    "\n",
    "Tensors generalize scalars, vectors, and matrices to higher dimensions. They are used in deep learning and represent multi-dimensional data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensor\n",
    "TENSOR = torch.tensor([[[[1, 2, 3],\n",
    "                        [3, 6, 9],\n",
    "                        [2, 4, 5]]]])\n",
    "TENSOR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TENSOR.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Creating Tensors\n",
    "\n",
    " Tensors can be created from lists, numpy arrays, or by using various PyTorch functions like `torch.tensor()`, `torch.zeros()`, `torch.ones()`, `torch.rand()`, etc. These functions allow creating tensors of specific shapes filled with zeros, ones, or random values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Random, ones and zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with zeros\n",
    "zeros_tensor = torch.zeros((3, 4))\n",
    "print(\"Zeros Tensor:\\n\", zeros_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with ones\n",
    "ones_tensor = torch.ones((2, 3))\n",
    "print(\"Ones Tensor:\\n\", ones_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor with random values\n",
    "random_tensor = torch.rand((3, 3))\n",
    "print(\"Random Tensor:\\n\", random_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor from a list\n",
    "list_tensor = torch.tensor([1, 2, 3, 4])\n",
    "print(\"Tensor from list:\\n\", list_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 arange\n",
    "\n",
    "By invoking arange(n), we can create a vector of evenly spaced values, starting at 0 (included) and ending at n (not included).\n",
    "By default, the interval size is 1. Unless otherwise specified, new tensors are stored in main memory and designated for CPU-based\n",
    "computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.arange(12, dtype=torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.reshape((3, 4))\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these values is called an element of the tensor. The tensor x contains 12 elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise\n",
    "# Using arrange and reshape, create a tonsor of (3, 6, 4)\n",
    "# Print 3rd element in the first column of the second matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Attributes of Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a random tensor with shape 3x4\n",
    "tensor = torch.rand(3,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can inspect the total number of elements in a tensor via its numel method\n",
    "print(f\"total number of elements: {tensor.numel()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can access a tensor’sshape (the length along each axis) by inspecting its shape attribute\n",
    "print(f\"Shape of tensor: {tensor.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print data type of tensor\n",
    "print(f\"Datatype of tensor: {tensor.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise - Print the device tensor is stored on\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Operations on tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Tensor Indexing and Slicing\n",
    "\n",
    "Similar to numpy arrays, elements of tensors can be accessed and modified using indexing and slicing. Indexing allows accessing specific elements, rows, or columns, while slicing enables accessing sub-tensors by specifying ranges for each dimension.\n",
    "\n",
    "We can access tensor elements by indexing (starting with 0). To access an element based on its position relative to the end of the list, we can use negative indexing.\n",
    "\n",
    "We can access whole ranges of indices via slicing (e.g., X[start:stop]), where the returned value includes the first index (start) but not the last (stop). \n",
    "\n",
    "Finally, when only one index (or slice) is specified for a 𝑘 th-order tensor, it is applied along axis 0. Thus, in the following code, [-1] selects the last row and [1:3] selects the second and third rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an example tensor\n",
    "tensor = torch.tensor([[2, 4, 6, 8], [1, 3, 5, 9], [3, 5, 7, 9]])\n",
    "\n",
    "# Print the original tensor\n",
    "print(\"Original tensor:\\n\", tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access and print specific elements\n",
    "print(\"Element at row 1, column 2:\", tensor[1, 2].item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access and print the first row of the tensor\n",
    "print('First row: ', tensor[0])\n",
    "\n",
    "# Access and print the first column of the tensor\n",
    "print('First column: ', tensor[:, 0], \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise - print the last row and the last column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Slice and print sub-tensors\n",
    "print(\"First two rows:\\n\", tensor[:2, :])\n",
    "print(\"Elements from row 1 to end, columns 2 to end:\\n\", tensor[1:, 2:], \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update a specific element in the tensor\n",
    "tensor[2, 3] = 23\n",
    "print(\"Tensor after updating the element at row 2, column 3 to 23:\\n\", tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update an entire column in the tensor\n",
    "tensor[:, 1] = 10\n",
    "print(\"Tensor after updating the entire second column to 10:\\n\", tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we want to assign multiple elements the same value, we apply the indexing on the lefthand side of the assignment operation.\n",
    "# For instance, [:2, :] accesses the first and second rows, where : takes all the elements along axis 1 (column).\n",
    "\n",
    "# Update multiple elements at once (first two rows)\n",
    "tensor[:2, :] = 18\n",
    "print(\"Tensor after updating the first two rows to 18:\\n\", tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise - Print the second to fourth numbers in the third row in the tensor T.\n",
    "T = torch.rand(5, 6)\n",
    "T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Arithmatic operations\n",
    "\n",
    "PyTorch supports various arithmetic operations such as addition, subtraction, multiplication, and division. These operations can be performed element-wise on tensors of the same shape. Scalar operations (adding, subtracting, multiplying, or dividing a tensor by a scalar) are also supported.\n",
    "\n",
    "The common standard arithmetic operators for addition (+), subtraction (-), multiplication (*), division (/), and exponentiation (**) have all been lifted to elementwise operations for identically-shaped tensors of arbitrary shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor_a = torch.tensor([[1, 2], [3, 4]])\n",
    "tensor_b = torch.tensor([[5, 6], [7, 8]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Element-wise addition\n",
    "print(\"Element-wise addition:\\n\", tensor_a + tensor_b, \"\\n\")\n",
    "\n",
    "# Element-wise subtraction\n",
    "print(\"Element-wise subtraction:\\n\", tensor_a - tensor_b, \"\\n\")\n",
    "\n",
    "# Element-wise multiplication\n",
    "print(\"Element-wise multiplication:\\n\", tensor_a * tensor_b, \"\\n\")\n",
    "\n",
    "# Element-wise division\n",
    "print(\"Element-wise division:\\n\", tensor_a / tensor_b, \"\\n\")\n",
    "\n",
    "# Element-wise division\n",
    "print(\"Element-wise powering:\\n\", tensor_a ** tensor_b, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 Tensor Aggregation Operations\n",
    "\n",
    "Aggregation operations include functions that reduce the tensor to a single value or a smaller tensor. Common operations are `sum()`, `mean()`, `max()`, `min()`, and their variants along specific dimensions (e.g., summing all elements along rows or columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = torch.tensor([[1.0, 2, 3], [4, 5, 6], [7, 8, 9]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sum of all elements\n",
    "print(\"Sum of all elements:\", tensor.sum().item())\n",
    "\n",
    "# Mean of all elements\n",
    "print(\"Mean of all elements:\", tensor.mean().item())\n",
    "\n",
    "# Maximum value\n",
    "print(\"Maximum value:\", tensor.max().item())\n",
    "\n",
    "# Minimum value\n",
    "print(\"Minimum value:\", tensor.min().item())\n",
    "\n",
    "# Sum along a specific dimension\n",
    "print(\"Sum along columns:\\n\", tensor.sum(dim=0))\n",
    "\n",
    "# Mean along a specific dimension\n",
    "print(\"Mean along rows:\\n\", tensor.mean(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 Tensor Reshaping Operations\n",
    "\n",
    "Reshaping operations change the shape of a tensor without altering its data. Key operations include `view()`, `reshape()`, `transpose()`, and `permute()`. These functions allow converting a tensor to a different shape, transposing dimensions, or flattening the tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = torch.rand(2, 3, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape the tensor to a different shape\n",
    "reshaped_tensor = tensor.view(3, 8)\n",
    "print(\"Reshaped Tensor (3x8):\\n\", reshaped_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transpose the tensor (swap dimensions)\n",
    "transposed_tensor = tensor.transpose(0, 1)\n",
    "print(\"Transposed Tensor (swap first and second dimensions):\\n\", transposed_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten the tensor (convert to 1D)\n",
    "flattened_tensor = tensor.view(-1)\n",
    "print(\"Flattened Tensor:\\n\", flattened_tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5 Tensor Broadcasting\n",
    "\n",
    "Broadcasting allows performing arithmetic operations on tensors of different shapes. PyTorch automatically expands the smaller tensor along the missing dimensions to match the shape of the larger tensor, enabling element-wise operations without explicitly reshaping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor_a = torch.tensor([[1.0, 2, 3], [4, 5, 6]])\n",
    "tensor_b = torch.tensor([1.0, 2, 3])\n",
    "\n",
    "# Broadcasting addition\n",
    "print(\"Broadcasting addition:\\n\", tensor_a + tensor_b, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6 Tensor Device Management\n",
    "\n",
    "PyTorch supports operations on both CPU and GPU. Tensors can be moved between devices using `.to('cpu')` or `.to('cuda')` methods. This is essential for leveraging GPU acceleration for faster computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor on CPU\n",
    "tensor = torch.tensor([1, 2, 3])\n",
    "print(\"Tensor on CPU:\\n\", tensor, \"\\n\")\n",
    "\n",
    "# Check if CUDA is available and move the tensor to GPU\n",
    "if torch.cuda.is_available():\n",
    "    tensor = tensor.to('cuda')\n",
    "    print(\"Tensor on GPU:\\n\", tensor, \"\\n\")\n",
    "else:\n",
    "    print(\"CUDA is not available.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7 Matrix Multiplication and Dot Product\n",
    "\n",
    "In addition to elementwise computations, we can also perform linear algebraic operations, such as dot products and matrix multiplications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = torch.tensor([[2,4,6,8],[1,3,5,9],[3,5,7,9]])\n",
    "\n",
    "# Print the original tensor\n",
    "print(\"Original tensor:\\n\", tensor, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dot Product\n",
    "\n",
    "# Dot product (element-wise multiplication)\n",
    "print(\"Element-wise multiplication (tensor * tensor):\\n\", tensor * tensor, \"\\n\")\n",
    "\n",
    "# Element-wise multiplication using the mul method\n",
    "print(\"Element-wise multiplication using tensor.mul(tensor):\\n\", tensor.mul(tensor), \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matrix multiplication (tensor with its transpose)\n",
    "print(\"Matrix multiplication (tensor.matmul(tensor.T)):\\n\", tensor.matmul(tensor.T))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.8 Joining tensors (concatination)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors can be joined (concatenated) along a specified dimension using functions like `torch.cat()` and `torch.stack()`. Concatenation combines tensors along an existing dimension, while stacking adds a new dimension. Tensors can also be split into smaller tensors using `torch.split()` or `torch.chunk()`.\n",
    "\n",
    "You can use `torch.cat()` to concatenate a sequence of tensors along a given dimension. `torch.stack()` is a related tensor joining option that concatenates a sequence of tensors along a new dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.arange(12, dtype=torch.float32).reshape((3,4))\n",
    "Y = torch.tensor([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])\n",
    "\n",
    "X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = torch.cat((X, Y), dim=0)\n",
    "print (t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t2 = torch.cat((X, Y), dim=1)\n",
    "print (t2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = torch.arange(24).reshape((2, 3, 4))\n",
    "t2 = torch.arange(start=10, end=34).reshape((2, 3, 4))\n",
    "t1, t2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate tensors t1 and t2 along the first dimension (rows).\n",
    "# This results in a tensor with more rows, stacking t2 below t1.\n",
    "t3 = torch.cat((t1, t2), dim=0)\n",
    "\n",
    "# Concatenate tensors t1 and t2 along the second dimension (columns).\n",
    "# This results in a tensor with more columns, placing t2 to the right of t1.\n",
    "t4 = torch.cat((t1, t2), dim=1)\n",
    "\n",
    "# Concatenate tensors t1 and t2 along the third dimension.\n",
    "# This dimension is typically for depth in 3D tensors, adding depth from t2 to t1.\n",
    "t5 = torch.cat((t1, t2), dim=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the concatenated tensors with messages\n",
    "print(\"Tensor t3 (concatenated along the first dimension):\\n\", t3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nTensor t4 (concatenated along the second dimension):\\n\", t4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nTensor t5 (concatenated along the third dimension):\\n\", t5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.9 Logical Operation\n",
    "Sometimes, we want to construct a binary tensor via logical statements. Take X == Y as an\n",
    "example. For each position i, j, if X[i, j] and Y[i, j] are equal, then the corresponding\n",
    "entry in the result takes value 1, otherwise it takes value 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X == Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise - Consider a tensor of shape (2, 3, 4). What are the shapes of the summation outputs along axes 0, 1, and 2?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.10 Inplace Operations\n",
    "\n",
    "In-place operations modify the tensor directly without creating a new tensor. They are denoted by an underscore at the end of the method name (e.g., `add_()`, `mul_()`). In-place operations are memory efficient but can lead to unintended side effects if not used carefully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = torch.tensor([[1,2,3], [4,5,6], [7,8,9]])\n",
    "\n",
    "# Print the original tensor\n",
    "print(\"Original tensor:\\n\", tensor, \"\\n\")\n",
    "\n",
    "# Print the tensor after adding 5 to each element (non-in-place operation)\n",
    "print(\"Tensor after adding 5 (non-in-place operation):\\n\", tensor.add(5), \"\\n\")\n",
    "\n",
    "# Print the original tensor again to show it remains unchanged\n",
    "print(\"The original tensor after non-in-place addition (should be unchanged):\\n\", tensor, \"\\n\")\n",
    "\n",
    "# Add 5 to each element of the tensor (in-place operation)\n",
    "tensor.add_(5)\n",
    "\n",
    "# Print the tensor after the in-place addition\n",
    "print(\"Tensor after adding 5 (in-place operation):\\n\", tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.11 Tensor Cloning and Detaching\n",
    "\n",
    "Cloning creates a copy of the tensor with its own data and computation history, allowing modifications without affecting the original tensor. Detaching a tensor from the computation graph using `.detach()` creates a new tensor that shares data but is not tracked for gradients, which is useful in certain parts of the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = torch.tensor([1, 2, 3], requires_grad=True)\n",
    "\n",
    "# Clone the tensor\n",
    "cloned_tensor = tensor.clone()\n",
    "print(\"Cloned Tensor:\\n\", cloned_tensor, \"\\n\")\n",
    "\n",
    "# Detach the tensor from the computation graph\n",
    "detached_tensor = tensor.detach()\n",
    "print(\"Detached Tensor:\\n\", detached_tensor, \"\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}