Tutorial1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to TensorFlow 2.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install tensorflow --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Content\n",
    "- [Intro to TensorFlow2.x](#Intro-to-TensorFlow2.x)\n",
    "    - [History between Keras and TensorFlow](#History-between-Keras-and-TensorFlow:)\n",
    "    - [Keras and tf.keras](#Keras-and-tf.keras)\n",
    "- [Eager Execution](#Eager-Execution)\n",
    "    - [Eager execution](#Eager-execution)\n",
    "    - [Graph execution](#Graph-execution)\n",
    "    - [Why TensorFlow adopted Eager Execution?](#Why-TensorFlow-adopted-Eager-Execution?)\n",
    "- [Tensors](#Tensors)\n",
    "    - [Create tensors](#Create-tensors)\n",
    "    - [indexing](#indexing)\n",
    "    - [Operations with tensors](#Operations-with-tensors)\n",
    "- [Variables](#Variables)\n",
    "    - [Operations with variables](#Operations-with-variables)\n",
    "    - [Extend](#Extend)\n",
    "    - [Hardware selection for variables](#Hardware-selection-for-variables)\n",
    "- [function](#function)\n",
    "- [gradient](#gradient)\n",
    "- [Some common seen commands in TensorFlow1.x](#Some-common-seen-commands-in-TensorFlow1.x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TensorFlow** is an end-to-end framework and platform designed to build and train machine learning models, especially deep learning models.\n",
    "\n",
    "**Keras** is a deep learning API written in Python, running on top of the machine learning platform TensorFlow.\n",
    "\n",
    "## History between Keras and TensorFlow:\n",
    "\n",
    "- Keras was first developed by Francois Chollet for his own reserach in March 27th, 2015. Back then, TensorFlow hasn't been developed. There weren't too many deep learning libraries available - the popular ones included Torch, Theano, and Caffe. \n",
    "- In order to train the neural networks, Keras required a **backend** (computational engine). Keras's default backend was Theano until v1.1.0. At the same time, Google released TensorFlow. Keras started supporting it as a backend, and became the default starting from the release of Keras v1.1.0.\n",
    "     - Keras started making TensorFlow default due to its popularity; and TensorFlow users were also becoming increasingly drawn to the simplicity of the high-level Keras API.\n",
    "- Google announced TensorFlow 2.0 in June 2019, they declared that Keras is now the official high-level API of TensorFlow.\n",
    "\n",
    "[Link](https://www.pyimagesearch.com/2019/10/21/keras-vs-tf-keras-whats-the-difference-in-tensorflow-2-0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4.1\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras and tf.keras\n",
    "\n",
    "- tf.keras submodule was introduced in TensorFlow v1.10.0, integrating Keras directly within TensorFlow package itself.\n",
    "- Release of Keras 2.3.0: \n",
    "     - It is the first release of Keras that brings the Keras package in sync with tf.keras\n",
    "     - It is the final release of Keras that will support multiple backends \n",
    "     - The keras package will only support bug fixes. Developers should start using tf.keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras == keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.Sequential == keras.models.Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.models.Model == tf.keras.Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Eager Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://towardsdatascience.com/eager-execution-vs-graph-execution-which-is-better-38162ea4dbf6\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eager execution is a powerful execution environment that evaluates operations immediately. It does not build graphs, and the operations return actual values instead of computational graphs to run later. With Eager execution, TensorFlow calculates the values of tensors as they occur in your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In Tensorflow 2.0, eager execution is enabled by default.\n",
    "tf.executing_eagerly() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello, [[4.]]\n"
     ]
    }
   ],
   "source": [
    "x = [[2.]]\n",
    "m = tf.matmul(x,x)\n",
    "print(\"hello, {}\".format(m))     #Return results immediately "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorFlow1.x, the above code will return:\n",
    "\n",
    "      hello, Tensor(\"MatMul:0\", shape=(1, 1), dtype=float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eager execution\n",
    "\n",
    "<font color= red>Easy-to-build\\&test</font>\n",
    "\n",
    "Eager execution is easy to implement, and simplify the model building experience. Advantages of eager execution:\n",
    "\n",
    "- **An intuitive interface** with natural Python code and data structures;\n",
    "- **Easier debugging** with calling operations directly to inspect and test models;\n",
    "- **Natural control flow** with Python, instead of graph control flow; and\n",
    "- Support for **GPU & TPU acceleration**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph execution\n",
    "\n",
    "<font color= red>Efficient and fast</font>\n",
    "\n",
    "Eager execution is slower than graph execution since eager execution runs all operations one-by-one in Python, it cannot take advantage of potential acceleration opportunities.\n",
    "\n",
    "Graphs are easy-to-optimize, simplify arithmetic operations. In graph execution, evaluation of all the operations happens only after we've called our program entirely. So, graph execution is\n",
    "- **Very Fast**\n",
    "- **Very Flexible**\n",
    "- **Runs in parallel**, even in sub-operation level;\n",
    "- **Very efficient**, on multiple devices\n",
    "- with **GPU & TPU acceleration** capability\n",
    "\n",
    "Graph execution is ideal for large model training. For small model training, beginners, eager execution is better suited."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why TensorFlow adopted Eager Execution?\n",
    "\n",
    "- Before version 2.0, TensorFlow prioritized graph execution because it was fast, effiicent, and flexible. But it is difficult to implement.\n",
    "- PyTorch adopted a different approach and prioritized dynamic computation graphs, which is a similar concept to eager execution. Although they are not that efficient, it is intuitive and easy to implement. Then PyTorch became attractive for the newcommers.\n",
    "- TensorFlow felt threatened, so the TensorFlow team adopted eager execution as the default execution method, and graph execution is optional.\n",
    "\n",
    "In TensorFlow 2.0, you can decorate a Python function using tf.function() to run it as a single graph object. With this new method, you can easily build models and gain all the graph execution benefits.\n",
    "\n",
    "Note: TensorFlow2.x, no session, no initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import timeit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eager_function(x):\n",
    "    result = x**2\n",
    "    print(result)\n",
    "    return result\n",
    "\n",
    "x = tf.constant([1.0,2.0,3.0,4.0,5.0])\n",
    "\n",
    "graph_function = tf.function(eager_function)  #run the same function with graph execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5,), dtype=float32, numpy=array([ 1.,  4.,  9., 16., 25.], dtype=float32)>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### eager execution\n",
    "eager_function(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"pow:0\", shape=(5,), dtype=float32)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5,), dtype=float32, numpy=array([ 1.,  4.,  9., 16., 25.], dtype=float32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### graph execution\n",
    "graph_function(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "Eager time: 0.00047989003360271454\n",
      "Graph time: 0.0005872081965208054\n"
     ]
    }
   ],
   "source": [
    "### compare the execution times\n",
    "print(\"Eager time:\", timeit.timeit(lambda: eager_function(x), number=1))\n",
    "print(\"Graph time:\", timeit.timeit(lambda: graph_function(x), number=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simple operations, graph execution does not always perform well because it has to spend the initial computing power to build a graph. If the code is ran 100 times, the results will change dramatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)\n",
      "Eager time: 0.03065408207476139\n",
      "Graph time: 0.010782018303871155\n"
     ]
    }
   ],
   "source": [
    "### compare the execution times\n",
    "print(\"Eager time:\", timeit.timeit(lambda: eager_function(x), number=100))\n",
    "print(\"Graph time:\", timeit.timeit(lambda: graph_function(x), number=100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a dummy neural network to compare the performances of eager and graph executions. No need to worry about the process of building the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eager time: 19.675863103941083\n",
      "Graph time: 8.229794163256884\n"
     ]
    }
   ],
   "source": [
    "# TensorFlow imports\n",
    "from tensorflow.keras import Input, Model\n",
    "from tensorflow.keras.layers import Flatten, Dense\n",
    "\n",
    "# Model building\n",
    "inputs = Input(shape=(28, 28)) \n",
    "x = Flatten()(inputs) \n",
    "x = Dense(256, \"relu\")(x)\n",
    "x = Dense(256, \"relu\")(x) \n",
    "x = Dense(256, \"relu\")(x) \n",
    "outputs = Dense(10, \"softmax\")(x) \n",
    "\n",
    "input_data = tf.random.uniform([100, 28, 28])\n",
    "\n",
    "# Eager Execution\n",
    "eager_model = Model(inputs=inputs, outputs=outputs)\n",
    "print(\"Eager time:\", timeit.timeit(lambda: eager_model(input_data), number=10000))\n",
    "\n",
    "#Graph Execution \n",
    "graph_model = tf.function(eager_model) # Wrap the model with tf.function \n",
    "print(\"Graph time:\", timeit.timeit(lambda: graph_model(input_data), number=10000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensors\n",
    "\n",
    "Tensors are TensorFlow's multi-dimensional arrays with uniform type. They are ***immutable***.\n",
    "\n",
    "[Ref](https://towardsdatascience.com/mastering-tensorflow-tensors-in-5-easy-steps-35f21998bb86)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[1 2 3 4 5]], shape=(1, 5), dtype=int32)\n",
      "tf.Tensor([[1. 1. 1. 1. 1.]], shape=(1, 5), dtype=float32)\n",
      "tf.Tensor([[0. 0. 0. 0. 0.]], shape=(1, 5), dtype=float32)\n",
      "tf.Tensor([1 2 3 4 5], shape=(5,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "t1 = tf.constant([[1,2,3,4,5]])\n",
    "t2 = tf.ones((1,5))\n",
    "t3 = tf.zeros((1,5))\n",
    "t4 = tf.range(start=1,limit=6,delta=1)\n",
    "print(t1)\n",
    "print(t2)\n",
    "print(t3)\n",
    "print(t4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([1 1 1 1 1], shape=(5,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "o = tf.constant([1,2,3,73,2])\n",
    "print(tf.ones_like(o))     #same shape as o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5,), dtype=int32, numpy=array([ 1,  2,  3, 73,  2], dtype=int32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.identity(o)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/job:localhost/replica:0/task:0/device:CPU:0'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "o.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    }
   ],
   "source": [
    "## copy tensors to devices\n",
    "x = tf.random.uniform((3,2))\n",
    "with tf.device('/cpu:0'):\n",
    "    x_cpu = tf.identity(x)\n",
    "print(x_cpu.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[1, 2, 3], [4, 5], [6]]>\n",
      "tf.Tensor([b'element1' b'element2' b'element3'], shape=(3,), dtype=string)\n",
      "tf.Tensor(\n",
      "[[ 25   0   0   0   0]\n",
      " [  0   0   0   0   0]\n",
      " [  0   0  50   0   0]\n",
      " [  0   0   0   0   0]\n",
      " [  0   0   0   0 100]], shape=(5, 5), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "## ragged tensor\n",
    "ragged_tensor = tf.ragged.constant([[1, 2, 3],[4, 5],[6]])\n",
    "print(ragged_tensor)\n",
    "## string tensor\n",
    "string_tensor = tf.constant([\"element1\",\n",
    "                             \"element2\",\n",
    "                             \"element3\"])\n",
    "print(string_tensor)\n",
    "## sparse tensor\n",
    "## when you have holes in your data, Sparse Tensors are to-go objects.\n",
    "sparse_tensor = tf.sparse.SparseTensor(indices=[[0,0],[2,2],[4,4]],\n",
    "                                      values = [25,50,100],\n",
    "                                      dense_shape=[5,5])\n",
    "print(tf.sparse.to_dense(sparse_tensor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attribute of tensors:\n",
    "- `.ndim` or `tf.rank(...).numpy()`\n",
    "- `.shape`\n",
    "    - `tf.reshape(tensor1,[...])`\n",
    "- `tf.size(...).numpy()`\n",
    "- `.dtype`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[ 0  1  2]\n",
      "  [ 3  4  5]]\n",
      "\n",
      " [[ 6  7  8]\n",
      "  [ 9 10 11]]], shape=(2, 2, 3), dtype=int32)\n",
      "3\n",
      "3\n",
      "(2, 2, 3)\n",
      "12\n",
      "<dtype: 'int32'>\n"
     ]
    }
   ],
   "source": [
    "rank_3_tensor = tf.constant([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]])\n",
    "print(rank_3_tensor)\n",
    "## rank\n",
    "print(rank_3_tensor.ndim)\n",
    "print(tf.rank(rank_3_tensor).numpy())\n",
    "## shape\n",
    "print(rank_3_tensor.shape)\n",
    "## size\n",
    "print(tf.size(rank_3_tensor).numpy())\n",
    "## dtype\n",
    "print(rank_3_tensor.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[ 0  1]\n",
      "  [ 2  3]\n",
      "  [ 4  5]]\n",
      "\n",
      " [[ 6  7]\n",
      "  [ 8  9]\n",
      "  [10 11]]], shape=(2, 3, 2), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[ 0  1  2  3  4  5]\n",
      " [ 6  7  8  9 10 11]], shape=(2, 6), dtype=int32)\n",
      "tf.Tensor([ 0  1  2  3  4  5  6  7  8  9 10 11], shape=(12,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "## reshape\n",
    "reshape_tensor1 = tf.reshape(rank_3_tensor,[2,3,2])\n",
    "print(reshape_tensor1)\n",
    "reshape_tensor2 = tf.reshape(rank_3_tensor,[2,-1])\n",
    "print(reshape_tensor2)\n",
    "reshape_tensor3 = tf.reshape(rank_3_tensor,[-1])  #flatten the tensor\n",
    "print(reshape_tensor3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 0  1  2  3  4  5  6  7  8  9 10 11], shape=(12,), dtype=int32)\n",
      "[ 0  1  2  3  4  5  6  7  8  9 10 11]\n",
      "0\n",
      "11\n",
      "[ 1  2  3  4  5  6  7  8  9 10]\n"
     ]
    }
   ],
   "source": [
    "rank_1_tensor = tf.constant([0,1,2,3,4,5,6,7,8,9,10,11])\n",
    "print(rank_1_tensor)\n",
    "## only values\n",
    "print(rank_1_tensor.numpy())\n",
    "## indexing\n",
    "print(rank_1_tensor[0].numpy())\n",
    "print(rank_1_tensor[-1].numpy())\n",
    "print(rank_1_tensor[1:-1].numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 0  1  2  3  4  5]\n",
      " [ 6  7  8  9 10 11]], shape=(2, 6), dtype=int32)\n",
      "[0 1 2 3 4 5]\n",
      "[ 6  7  8  9 10 11]\n",
      "0\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "rank_2_tensor = tf.constant([[0,1,2,3,4,5],[6,7,8,9,10,11]])\n",
    "print(rank_2_tensor)\n",
    "## indexing\n",
    "print(rank_2_tensor[0].numpy())\n",
    "print(rank_2_tensor[1].numpy())\n",
    "print(rank_2_tensor[0,0].numpy())\n",
    "print(rank_2_tensor[0,2].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations with tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant([[2, 4], \n",
    "                 [6, 8]], dtype=tf.float32)   #if we didn't specify the dtype, it would be tf.int32\n",
    "b = tf.constant([[1, 3], \n",
    "                 [5, 7]], dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int8, numpy=\n",
       "array([[2, 4],\n",
       "       [6, 8]], dtype=int8)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## We cannot change a tensor's dtype with a mutator\n",
    "## We can only use tf.cast to create a new tensor with the desired data type\n",
    "\n",
    "aa = tf.cast(a,dtype=tf.int8)\n",
    "aa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 3.  7.]\n",
      " [11. 15.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "## addition\n",
    "add_tensors = tf.add(a,b)    #the same with a+b \n",
    "print(add_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 3.  7.]\n",
      " [11. 15.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(a.__add__(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 2. 12.]\n",
      " [30. 56.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "## element-wise multiplication\n",
    "multiply_tensors = tf.multiply(a,b)    #the same with a*b\n",
    "print(multiply_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 2. 12.]\n",
      " [30. 56.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(a.__mul__(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 34.]\n",
      " [46. 74.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "## matrix multiplication\n",
    "matmul_tensors = tf.matmul(a,b)\n",
    "print(matmul_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 34.]\n",
      " [46. 74.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(a @ b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 34.]\n",
      " [46. 74.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(a.__matmul__(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.0\n"
     ]
    }
   ],
   "source": [
    "## max value\n",
    "print(tf.reduce_max(b).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16.0\n"
     ]
    }
   ],
   "source": [
    "## sum\n",
    "print(tf.reduce_sum(b).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1]\n"
     ]
    }
   ],
   "source": [
    "## argmx\n",
    "print(tf.argmax(b).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.11920291 0.880797  ]\n",
      " [0.11920291 0.880797  ]]\n"
     ]
    }
   ],
   "source": [
    "## in softmax function\n",
    "print(tf.nn.softmax(b).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 5 10]\n",
      " [15 20]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "m = tf.constant([5])\n",
    "n = tf.constant([[1,2],[3,4]])\n",
    "\n",
    "print(tf.multiply(m, n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variables\n",
    "\n",
    "A TensorFlow Variable is the preferred object type representing a shared and persistent state that you can ***manipulate with any operation***, including TensorFlow models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0. 1.]\n",
      " [2. 3.]], shape=(2, 2), dtype=float32)\n",
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[0., 1.],\n",
      "       [2., 3.]], dtype=float32)>\n",
      "<tf.Variable 'Variable:0' shape=() dtype=int32, numpy=10000>\n",
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[0., 1.],\n",
      "       [2., 3.]], dtype=float32)>\n",
      "tf.Tensor(\n",
      "[[ True  True]\n",
      " [ True  True]], shape=(2, 2), dtype=bool)\n",
      "<tf.Variable 'Variable:0' shape=() dtype=string, numpy=b'string'>\n",
      "<tf.Variable 'Variable:0' shape=(2,) dtype=string, numpy=array([b'string1', b'string2'], dtype=object)>\n"
     ]
    }
   ],
   "source": [
    "## type1: pass a tf.constant()\n",
    "a = tf.constant([[0.0,1.0],[2.0,3.0]])\n",
    "print(a)\n",
    "var_a = tf.Variable(a)\n",
    "print(var_a)\n",
    "## type2: pass a single integer\n",
    "var_b = tf.Variable(10000)\n",
    "print(var_b)\n",
    "## type3: pass a list\n",
    "var_c = tf.Variable([[0.0,1.0],[2.0,3.0]])\n",
    "print(var_c)\n",
    "print(var_a == var_c)\n",
    "## type4: pass a single string\n",
    "var_d = tf.Variable('string')\n",
    "print(var_d)\n",
    "## type5: pass a list of strings\n",
    "var_e = tf.Variable(['string1','string2'])\n",
    "print(var_e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variable objects are built on top of Tensor objects.\n",
    "\n",
    "Attribute of variables:\n",
    "- `.shape`\n",
    "    - `tf.reshape(var1,[...])`\n",
    "- `tf.size(...).numpy()`\n",
    "- `tf.rank(...).numpy()`\n",
    "- `.dtype`\n",
    "- `.value()`      \n",
    "     - every variable must specify its initial value\n",
    "- `.name`\n",
    "     - If you don't specify a `name`, TensorFlow assigns a default name\n",
    "- `.trainable`\n",
    "- `.device`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[0., 1.],\n",
      "       [2., 3.]], dtype=float32)>\n",
      "(2, 2)\n",
      "4\n",
      "2\n",
      "tf.Tensor(\n",
      "[[0. 1.]\n",
      " [2. 3.]], shape=(2, 2), dtype=float32)\n",
      "[[0. 1.]\n",
      " [2. 3.]]\n"
     ]
    }
   ],
   "source": [
    "print(var_a)\n",
    "print(var_a.shape)\n",
    "print(tf.size(var_a).numpy())\n",
    "print(tf.rank(var_a).numpy())\n",
    "\n",
    "## The values stroed in the variables \n",
    "print(var_a.value())\n",
    "## The values stroed in the variables as NumPy object\n",
    "print(var_a.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]], shape=(4, 1), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[0.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]], shape=(4, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "reshape_a1 = tf.reshape(var_a,[4,1])\n",
    "print(reshape_a1)\n",
    "reshape_a2 = tf.reshape(var_a,[4,-1])\n",
    "print(reshape_a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable:0\n",
      "myname:0\n",
      "True\n",
      "/job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    }
   ],
   "source": [
    "## name\n",
    "\n",
    "var1 = tf.Variable([[0.0,1.0],[2.0,3.0]])\n",
    "print(var1.name)\n",
    "var2 = tf.Variable([[0.0,1.0],[2.0,3.0]],name='myname')\n",
    "print(var2.name)\n",
    "\n",
    "print(var_a.trainable)\n",
    "print(var_a.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations with variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Plus\n",
      "tf.Tensor(\n",
      "[[3. 4.]\n",
      " [5. 6.]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Minus\n",
      "tf.Tensor(\n",
      "[[-1.  0.]\n",
      " [ 1.  2.]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Multiply\n",
      "tf.Tensor(\n",
      "[[2. 4.]\n",
      " [6. 8.]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Divide\n",
      "tf.Tensor(\n",
      "[[0.5 1. ]\n",
      " [1.5 2. ]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Matrix Multiply\n",
      "tf.Tensor(\n",
      "[[ 7. 10.]\n",
      " [15. 22.]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Modulo\n",
      "tf.Tensor(\n",
      "[[1. 0.]\n",
      " [1. 0.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "var_a = tf.Variable([[1.0,2.0],[3.0,4.0]])\n",
    "print('Plus')\n",
    "print(var_a + 2)\n",
    "print('\\nMinus')\n",
    "print(var_a - 2)\n",
    "print('\\nMultiply')\n",
    "print(var_a * 2)\n",
    "print('\\nDivide')\n",
    "print(var_a / 2)\n",
    "print('\\nMatrix Multiply')\n",
    "print(var_a @ var_a)   #matrix multipllication\n",
    "print('\\nModulo')\n",
    "print(var_a % 2)       #modulo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[100., 200.],\n",
      "       [300., 400.]], dtype=float32)>\n",
      "tf.Tensor(\n",
      "[[100. 200.]\n",
      " [300. 400.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "## variables can be updated with .assign, .assign_add, .assign_sub\n",
    "## assign\n",
    "var_a = tf.Variable([[1.0,2.0],[3.0,4.0]])\n",
    "var_a.assign([[100,200],[300,400]])\n",
    "print(var_a)\n",
    "print(var_a.value())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[101., 202.],\n",
      "       [303., 404.]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "var_a.assign_add([[1,2],[3,4]])\n",
    "print(var_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101. 202.]\n",
      "[303. 404.]\n",
      "101.0\n",
      "202.0\n",
      "tf.Tensor(\n",
      "[[ 5 10]\n",
      " [15 20]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "## indexing\n",
    "print(var_a[0].numpy())\n",
    "print(var_a[1].numpy())\n",
    "print(var_a[0,0].numpy())\n",
    "print(var_a[0,1].numpy())\n",
    "## broadcasting\n",
    "var_b = tf.Variable([5])\n",
    "var_c = tf.Variable([[1,2],[3,4]])\n",
    "print(var_b * var_c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hardware selection for variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    }
   ],
   "source": [
    "## Print what type of device our variable is processed with\n",
    "print(var_a.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/job:localhost/replica:0/task:0/device:CPU:0\n",
      "/job:localhost/replica:0/task:0/device:CPU:0\n",
      "/job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    }
   ],
   "source": [
    "## tf.device: Set the device we want a particular calculation to be processed with\n",
    "with tf.device('CPU:0'):\n",
    "    a = tf.Variable([[1.0,2.0,3.0],[4.0,5.0,6.0]])\n",
    "    b = tf.Variable([[1.0,2.0,3.0]])\n",
    "    print(a.device)\n",
    "    print(b.device)\n",
    "    \n",
    "with tf.device('GPU:0'):\n",
    "    k = a * b\n",
    "    print(k.device)      #should be GPU if it exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# function\n",
    "\n",
    "`tf.gradients` is only valid in a graph context. In particular, it is valid in the context of a `tf.function` wrapper, where code is executing as a graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=1.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=1.0>]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@tf.function\n",
    "def example():\n",
    "    a = tf.constant(0.)\n",
    "    b = 2 * a **2\n",
    "    return tf.gradients(a + b, [a, b], stop_gradients=[a, b])\n",
    "\n",
    "example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# gradient\n",
    "\n",
    "In TensorFlow, we use the `tf.GradientTape` context to trace what's happened inside, so that we can calculate the gradients afterward with the `.gradient()` from the tape object.\n",
    "\n",
    "\n",
    "[Ref](https://www.tensorflow.org/guide/advanced_autodiff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n",
      "dz/dy: None\n"
     ]
    }
   ],
   "source": [
    "# tf.gradients\n",
    "\n",
    "x = tf.Variable(2.0)\n",
    "y = tf.Variable(3.0)\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "    x_sq = x * x\n",
    "    with t.stop_recording():  #See Ref for details\n",
    "        y_sq = y * y  #not trackable due to stop_recording\n",
    "    z = x_sq + y_sq\n",
    "\n",
    "grad = t.gradient(z, {'x': x, 'y': y})\n",
    "\n",
    "print('dz/dx:', grad['x'])  # 2*x => 4\n",
    "print('dz/dy:', grad['y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Tensor: shape=(), dtype=float32, numpy=4.0>, None]\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable(2.0)\n",
    "y = tf.Variable(3.0)\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "    x_sq = x * x\n",
    "    with t.stop_recording():  \n",
    "        y_sq = y * y\n",
    "    z = x_sq + y_sq\n",
    "    \n",
    "print(t.gradient(z,[x,y]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[None, <tf.Tensor: shape=(), dtype=float32, numpy=6.0>]\n"
     ]
    }
   ],
   "source": [
    "## Note that once we extracted the gradients from the tape\n",
    "## the resources it holds will be released\n",
    "with tf.GradientTape() as t:\n",
    "    f = x * x + y * y\n",
    "    \n",
    "print(t.gradient(f,[x,y]))\n",
    "# print(t.gradient(f,[x,y]))   #uncomment this line to see the error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the context will only keep track of the **variables** but not the tensors, meaning that we can oly ask for gradients with respect to the variables by default (throught the `watch_accessed_variables` argument which defaults to `True`). But we can explicitly ask the tape to <font color=red>watch</font> things for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Tensor: shape=(), dtype=float32, numpy=4.0>, <tf.Tensor: shape=(), dtype=float32, numpy=6.0>]\n"
     ]
    }
   ],
   "source": [
    "x = tf.constant(2.0)\n",
    "y = tf.Variable(3.0)\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "    t.watch(x)             #comment this line to see the difference\n",
    "    f = x * x + y * y\n",
    "    \n",
    "print(t.gradient(f,[x,y]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(4.0, shape=(), dtype=float32)\n",
      "tf.Tensor(6.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "## we can create a persistent gradient tape object to compute\n",
    "## multiple gradients and manually release the resources\n",
    "\n",
    "x = tf.Variable(2.0)\n",
    "y = tf.Variable(3.0)\n",
    "\n",
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    f = x*x+y*y\n",
    "\n",
    "print(tape.gradient(f,x))\n",
    "print(tape.gradient(f,y))\n",
    "\n",
    "del tape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "multiple tapes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "0.25\n"
     ]
    }
   ],
   "source": [
    "x0 = tf.constant(0.0)\n",
    "x1 = tf.constant(0.0)\n",
    "\n",
    "with tf.GradientTape() as tape0, tf.GradientTape() as tape1:\n",
    "    tape0.watch(x0)\n",
    "    tape1.watch(x1)\n",
    "\n",
    "    y0 = tf.math.sin(x0)\n",
    "    y1 = tf.nn.sigmoid(x1)\n",
    "\n",
    "    y = y0 + y1\n",
    "    ys = tf.reduce_sum(y)\n",
    "\n",
    "\n",
    "print(tape0.gradient(ys, x0).numpy())  # cos(x) => 1.0\n",
    "\n",
    "print(tape1.gradient(ys, x1).numpy())   # sigmoid(x1)*(1-sigmoid(x1)) => 0.25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "higher-order gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dy_dx: 3.0\n",
      "d2y_dx2: 6.0\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable(1.0)  # Create a Tensorflow variable initialized to 1.0\n",
    "\n",
    "with tf.GradientTape() as t2:\n",
    "    with tf.GradientTape() as t1:\n",
    "        y = x * x * x\n",
    "\n",
    "    # Compute the gradient inside the outer `t2` context manager\n",
    "    # which means the gradient computation is differentiable as well.\n",
    "    dy_dx = t1.gradient(y, x)\n",
    "d2y_dx2 = t2.gradient(dy_dx, x)\n",
    "\n",
    "print('dy_dx:', dy_dx.numpy())  # 3 * x**2 => 3.0\n",
    "print('d2y_dx2:', d2y_dx2.numpy())  # 6 * x => 6.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some common seen commands in TensorFlow1.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tf.Session()`, `tf.global_variables_initializer()`, `tf.placeholder` are being removed in TensorFlow2.x\n",
    "\n",
    "If you do want to use them, use `tf.compat.v1.Session()`, `tf.compat.v1.global_variables_initializer()`, `tf.compat.v1.placeholder` instead\n",
    "\n",
    "`tf.gradients` is not supported when eager execution is enabled. Use `tf.GradientTape` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[ 3.,  7.],\n",
       "       [11., 15.]], dtype=float32)>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## When eager execution is on, the result will be directly returned\n",
    "a = tf.constant([[2, 4], \n",
    "                 [6, 8]], dtype=tf.float32)   \n",
    "b = tf.constant([[1, 3], \n",
    "                 [5, 7]], dtype=tf.float32)\n",
    "tf.add(a,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Add:0' shape=(2, 2) dtype=float32>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## When eager execution is off, the result is hidden\n",
    "tf.compat.v1.disable_eager_execution()\n",
    "a = tf.constant([[2, 4], \n",
    "                 [6, 8]], dtype=tf.float32)   #if we didn't specify the dtype, it would be tf.int32\n",
    "b = tf.constant([[1, 3], \n",
    "                 [5, 7]], dtype=tf.float32)\n",
    "tensor_sum = tf.add(a,b)\n",
    "tensor_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.  7.]\n",
      " [11. 15.]]\n"
     ]
    }
   ],
   "source": [
    "## In TensorFlow1.x, this is the way to really run the code\n",
    "## session is removed in TensorFlow2.x\n",
    "with tf.compat.v1.Session() as sess:\n",
    "    init = tf.compat.v1.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    print(tensor_sum.eval())        #way1 to print result\n",
    "#     print(sess.run(tensor_sum))     #way2 to print result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(In TensorFlow1.x) **Placeholders** in TensorFlow are similar to variables and you can declare it using tf.placeholder. You dont have to provide an initial value and you can specify it at runtime with feed_dict argument inside Session.run , whereas in tf.Variable you can to provide initial value when you declare it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.2372893 1.2466476 1.2848123 1.3270197 1.408318 ]\n",
      " [1.5434198 1.2169448 1.565929  1.3188149 1.4794407]\n",
      " [1.9835036 1.7217739 2.3466644 2.2340496 2.2978036]\n",
      " [0.8756916 1.0449702 1.8397173 1.8703374 1.4189218]\n",
      " [1.2161818 1.3499663 2.0907462 2.0469189 1.649347 ]]\n"
     ]
    }
   ],
   "source": [
    "## Demonstration of placeholder\n",
    "x = tf.compat.v1.placeholder(tf.float32, shape=(5, 5))\n",
    "y = tf.matmul(x, x)\n",
    "\n",
    "with tf.compat.v1.Session() as sess:\n",
    "#     print(sess.run(y))  # ERROR: will fail because x was not fed.\n",
    "    rand_array = np.random.rand(5, 5)\n",
    "    print(sess.run(y, feed_dict={x: rand_array}))  # Will succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}