onnx/onnx-inference-emotion-recognition.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Facial Expression Recognition (Emotion FER+) using ONNX Runtime on Azure ML\n",
        "\n",
        "This example shows how to deploy an image classification neural network using the Facial Expression Recognition ([FER](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)) dataset and Open Neural Network eXchange format ([ONNX](http://aka.ms/onnxdocarticle)) on the Azure Machine Learning platform. This tutorial will show you how to deploy a FER+ model from the [ONNX model zoo](https://github.com/onnx/models), use it to make predictions using ONNX Runtime Inference, and deploy it as a web service in Azure.\n",
        "\n",
        "Throughout this tutorial, we will be referring to ONNX, a neural network exchange format used to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools (CNTK, PyTorch, Caffe, MXNet, TensorFlow) and choose the combination that is best for them. ONNX is developed and supported by a community of partners including Microsoft AI, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai) and [open source files](https://github.com/onnx).\n",
        "\n",
        "[ONNX Runtime](https://aka.ms/onnxruntime-python) is the runtime engine that enables evaluation of trained machine learning (Traditional ML and Deep Learning) models with high performance and low resource utilization. We use the CPU version of ONNX Runtime in this tutorial, but will soon be releasing an additional tutorial for deploying this model using ONNX Runtime GPU.\n",
        "\n",
        "#### Tutorial Objectives:\n",
        "\n",
        "1. Describe the FER+ dataset and pretrained Convolutional Neural Net ONNX model for Emotion Recognition, stored in the ONNX model zoo.\n",
        "2. Deploy and run the pretrained FER+ ONNX model on an Azure Machine Learning instance\n",
        "3. Predict labels for test set data points in the cloud using ONNX Runtime and Azure ML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites\n",
        "\n",
        "### 1. Install Azure ML SDK and create a new workspace\n",
        "Please follow [Azure ML configuration notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/00.configuration.ipynb) to set up your environment.\n",
        "\n",
        "### 2. Install additional packages needed for this Notebook\n",
        "You need to install the popular plotting library `matplotlib`, the image manipulation library `opencv`, and the `onnx` library in the conda environment where Azure Maching Learning SDK is installed.\n",
        "\n",
        "```sh\n",
        "(myenv) $ pip install matplotlib onnx opencv-python\n",
        "```\n",
        "\n",
        "**Debugging tip**: Make sure that to activate your virtual environment (myenv) before you re-launch this notebook using the `jupyter notebook` comand. Choose the respective Python kernel for your new virtual environment using the `Kernel > Change Kernel` menu above. If you have completed the steps correctly, the upper right corner of your screen should state `Python [conda env:myenv]` instead of `Python [default]`.\n",
        "\n",
        "### 3. Download sample data and pre-trained ONNX model from ONNX Model Zoo.\n",
        "\n",
        "In the following lines of code, we download [the trained ONNX Emotion FER+ model and corresponding test data](https://github.com/onnx/models/tree/master/emotion_ferplus) and place them in the same folder as this tutorial notebook. For more information about the FER+ dataset, please visit Microsoft Researcher Emad Barsoum's [FER+ source data repository](https://github.com/ebarsoum/FERPlus)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# urllib is a built-in Python library to download files from URLs\n",
        "\n",
        "# Objective: retrieve the latest version of the ONNX Emotion FER+ model files from the\n",
        "# ONNX Model Zoo and save it in the same folder as this tutorial\n",
        "\n",
        "import urllib.request\n",
        "\n",
        "onnx_model_url = \"https://www.cntk.ai/OnnxModels/emotion_ferplus/opset_7/emotion_ferplus.tar.gz\"\n",
        "\n",
        "urllib.request.urlretrieve(onnx_model_url, filename=\"emotion_ferplus.tar.gz\")\n",
        "\n",
        "# the ! magic command tells our jupyter notebook kernel to run the following line of \n",
        "# code from the command line instead of the notebook kernel\n",
        "\n",
        "# We use tar and xvcf to unzip the files we just retrieved from the ONNX model zoo\n",
        "\n",
        "!tar xvzf emotion_ferplus.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy a VM with your ONNX model in the Cloud\n",
        "\n",
        "### Load Azure ML workspace\n",
        "\n",
        "We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Check core SDK version number\n",
        "import azureml.core\n",
        "\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Workspace\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Registering your model with Azure ML"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_dir = \"emotion_ferplus\" # replace this with the location of your model files\n",
        "\n",
        "# leave as is if it's in the same folder as this notebook"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.model import Model\n",
        "\n",
        "model = Model.register(model_path = model_dir + \"/\" + \"model.onnx\",\n",
        "                       model_name = \"onnx_emotion\",\n",
        "                       tags = {\"onnx\": \"demo\"},\n",
        "                       description = \"FER+ emotion recognition CNN from ONNX Model Zoo\",\n",
        "                       workspace = ws)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Optional: Displaying your registered models\n",
        "\n",
        "This step is not required, so feel free to skip it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "models = ws.models()\n",
        "for name, m in models.items():\n",
        "    print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### ONNX FER+ Model Methodology\n",
        "\n",
        "The image classification model we are using is pre-trained using Microsoft's deep learning cognitive toolkit, [CNTK](https://github.com/Microsoft/CNTK), from the [ONNX model zoo](http://github.com/onnx/models). The model zoo has many other models that can be deployed on cloud providers like AzureML without any additional training. To ensure that our cloud deployed model works, we use testing data from the well-known FER+ data set, provided as part of the [trained Emotion Recognition model](https://github.com/onnx/models/tree/master/emotion_ferplus) in the ONNX model zoo.\n",
        "\n",
        "The original Facial Emotion Recognition (FER) Dataset was released in 2013 by Pierre-Luc Carrier and Aaron Courville as part of a [Kaggle Competition](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data), but some of the labels are not entirely appropriate for the expression. In the FER+ Dataset, each photo was evaluated by at least 10 croud sourced reviewers, creating a more accurate basis for ground truth. \n",
        "\n",
        "You can see the difference of label quality in the sample model input below. The FER labels are the first word below each image, and the FER+ labels are the second word below each image.\n",
        "\n",
        "![](https://raw.githubusercontent.com/Microsoft/FERPlus/master/FER+vsFER.png)\n",
        "\n",
        "***Input: Photos of cropped faces from FER+ Dataset***\n",
        "\n",
        "***Task: Classify each facial image into its appropriate emotions in the emotion table***\n",
        "\n",
        "```    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, 'anger':4, 'disgust':5, 'fear':6, 'contempt':7} ```\n",
        "\n",
        "***Output: Emotion prediction for input image***\n",
        "\n",
        "\n",
        "Remember, once the application is deployed in Azure ML, you can use your own images as input for the model to classify."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# for images and plots in this notebook\n",
        "import matplotlib.pyplot as plt  \n",
        "from IPython.display import Image\n",
        "\n",
        "# display images inline\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Model Description\n",
        "\n",
        "The FER+ model from the ONNX Model Zoo is summarized by the graphic below. You can see the entire workflow of our pre-trained model in the following image from Barsoum et. al's paper [\"Training Deep Networks for Facial Expression Recognition\n",
        "with Crowd-Sourced Label Distribution\"](https://arxiv.org/pdf/1608.01041.pdf), with our (64 x 64) input images and our output probabilities for each of the labels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![](https://raw.githubusercontent.com/vinitra/FERPlus/master/emotion_model_img.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Specify our Score and Environment Files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We are now going to deploy our ONNX Model on AML with inference in ONNX Runtime. We begin by writing a score.py file, which will help us run the model in our Azure ML virtual machine (VM), and then specify our environment by writing a yml file. You will also notice that we import the onnxruntime library to do runtime inference on our ONNX models (passing in input and evaluating out model's predicted output). More information on the API and commands can be found in the [ONNX Runtime documentation](https://aka.ms/onnxruntime).\n",
        "\n",
        "### Write Score File\n",
        "\n",
        "A score file is what tells our Azure cloud service what to do. After initializing our model using azureml.core.model, we start an ONNX Runtime inference session to evaluate the data passed in on our function calls."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile score.py\n",
        "import json\n",
        "import numpy as np\n",
        "import onnxruntime\n",
        "import sys\n",
        "import os\n",
        "from azureml.core.model import Model\n",
        "import time\n",
        "\n",
        "def init():\n",
        "    global session, input_name, output_name\n",
        "    model = Model.get_model_path(model_name = 'onnx_emotion')\n",
        "    session = onnxruntime.InferenceSession(model, None)\n",
        "    input_name = session.get_inputs()[0].name\n",
        "    output_name = session.get_outputs()[0].name \n",
        "    \n",
        "def run(input_data):\n",
        "    '''Purpose: evaluate test input in Azure Cloud using onnxruntime.\n",
        "        We will call the run function later from our Jupyter Notebook \n",
        "        so our azure service can evaluate our model input in the cloud. '''\n",
        "\n",
        "    try:\n",
        "        # load in our data, convert to readable format\n",
        "        data = np.array(json.loads(input_data)['data']).astype('float32')\n",
        "        \n",
        "        start = time.time()\n",
        "        r = session.run([output_name], {input_name : data})\n",
        "        end = time.time()\n",
        "        \n",
        "        result = emotion_map(postprocess(r[0]))\n",
        "        \n",
        "        result_dict = {\"result\": result,\n",
        "                      \"time_in_sec\": [end - start]}\n",
        "    except Exception as e:\n",
        "        result_dict = {\"error\": str(e)}\n",
        "    \n",
        "    return json.dumps(result_dict)\n",
        "\n",
        "def emotion_map(classes, N=1):\n",
        "    \"\"\"Take the most probable labels (output of postprocess) and returns the \n",
        "    top N emotional labels that fit the picture.\"\"\"\n",
        "    \n",
        "    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, \n",
        "                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}\n",
        "    \n",
        "    emotion_keys = list(emotion_table.keys())\n",
        "    emotions = []\n",
        "    for i in range(N):\n",
        "        emotions.append(emotion_keys[classes[i]])\n",
        "    return emotions\n",
        "\n",
        "def softmax(x):\n",
        "    \"\"\"Compute softmax values (probabilities from 0 to 1) for each possible label.\"\"\"\n",
        "    x = x.reshape(-1)\n",
        "    e_x = np.exp(x - np.max(x))\n",
        "    return e_x / e_x.sum(axis=0)\n",
        "\n",
        "def postprocess(scores):\n",
        "    \"\"\"This function takes the scores generated by the network and \n",
        "    returns the class IDs in decreasing order of probability.\"\"\"\n",
        "    prob = softmax(scores)\n",
        "    prob = np.squeeze(prob)\n",
        "    classes = np.argsort(prob)[::-1]\n",
        "    return classes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Write Environment File"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.conda_dependencies import CondaDependencies \n",
        "\n",
        "myenv = CondaDependencies()\n",
        "myenv.add_pip_package(\"numpy\")\n",
        "myenv.add_pip_package(\"azureml-core\")\n",
        "myenv.add_pip_package(\"onnxruntime\")\n",
        "\n",
        "\n",
        "with open(\"myenv.yml\",\"w\") as f:\n",
        "    f.write(myenv.serialize_to_string())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create the Container Image\n",
        "\n",
        "This step will likely take a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.image import ContainerImage\n",
        "\n",
        "image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
        "                                                  runtime = \"python\",\n",
        "                                                  conda_file = \"myenv.yml\",\n",
        "                                                  description = \"Emotion ONNX Runtime container\",\n",
        "                                                  tags = {\"demo\": \"onnx\"})\n",
        "\n",
        "\n",
        "image = ContainerImage.create(name = \"onnxtest\",\n",
        "                              # this is the model object\n",
        "                              models = [model],\n",
        "                              image_config = image_config,\n",
        "                              workspace = ws)\n",
        "\n",
        "image.wait_for_creation(show_output = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In case you need to debug your code, the next line of code accesses the log file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(image.image_build_log_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We're all done specifying what we want our virtual machine to do. Let's configure and deploy our container image.\n",
        "\n",
        "### Deploy the container image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.webservice import AciWebservice\n",
        "\n",
        "aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
        "                                               memory_gb = 1, \n",
        "                                               tags = {'demo': 'onnx'}, \n",
        "                                               description = 'ONNX for emotion recognition model')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.webservice import Webservice\n",
        "\n",
        "aci_service_name = 'onnx-demo-emotion'\n",
        "print(\"Service\", aci_service_name)\n",
        "\n",
        "aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
        "                                           image = image,\n",
        "                                           name = aci_service_name,\n",
        "                                           workspace = ws)\n",
        "\n",
        "aci_service.wait_for_deployment(True)\n",
        "print(aci_service.state)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following cell will likely take a few minutes to run as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if aci_service.state != 'Healthy':\n",
        "    # run this command for debugging.\n",
        "    print(aci_service.get_logs())\n",
        "\n",
        "    # If your deployment fails, make sure to delete your aci_service before trying again!\n",
        "    # aci_service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Success!\n",
        "\n",
        "If you've made it this far, you've deployed a working VM with a facial emotion recognition model running in the cloud using Azure ML. Congratulations!\n",
        "\n",
        "Let's see how well our model deals with our test images."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Testing and Evaluation\n",
        "\n",
        "### Useful Helper Functions\n",
        "\n",
        "We preprocess and postprocess our data (see score.py file) using the helper functions specified in the [ONNX FER+ Model page in the Model Zoo repository](https://github.com/onnx/models/tree/master/emotion_ferplus)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def emotion_map(classes, N=1):\n",
        "    \"\"\"Take the most probable labels (output of postprocess) and returns the \n",
        "    top N emotional labels that fit the picture.\"\"\"\n",
        "    \n",
        "    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, \n",
        "                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}\n",
        "    \n",
        "    emotion_keys = list(emotion_table.keys())\n",
        "    emotions = []\n",
        "    for i in range(N):\n",
        "        emotions.append(emotion_keys[classes[i]])\n",
        "        \n",
        "    return emotions\n",
        "\n",
        "def softmax(x):\n",
        "    \"\"\"Compute softmax values (probabilities from 0 to 1) for each possible label.\"\"\"\n",
        "    x = x.reshape(-1)\n",
        "    e_x = np.exp(x - np.max(x))\n",
        "    return e_x / e_x.sum(axis=0)\n",
        "\n",
        "def postprocess(scores):\n",
        "    \"\"\"This function takes the scores generated by the network and \n",
        "    returns the class IDs in decreasing order of probability.\"\"\"\n",
        "    prob = softmax(scores)\n",
        "    prob = np.squeeze(prob)\n",
        "    classes = np.argsort(prob)[::-1]\n",
        "    return classes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Load Test Data\n",
        "\n",
        "These are already in your directory from your ONNX model download (from the model zoo).\n",
        "\n",
        "Notice that our Model Zoo files have a .pb extension. This is because they are [protobuf files (Protocol Buffers)](https://developers.google.com/protocol-buffers/docs/pythontutorial), so we need to read in our data through our ONNX TensorProto reader into a format we can work with, like numerical arrays."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# to manipulate our arrays\n",
        "import numpy as np \n",
        "\n",
        "# read in test data protobuf files included with the model\n",
        "import onnx\n",
        "from onnx import numpy_helper\n",
        "\n",
        "# to use parsers to read in our model/data\n",
        "import json\n",
        "import os\n",
        "\n",
        "test_inputs = []\n",
        "test_outputs = []\n",
        "\n",
        "# read in 3 testing images from .pb files\n",
        "test_data_size = 3\n",
        "\n",
        "for i in np.arange(test_data_size):\n",
        "    input_test_data = os.path.join(model_dir, 'test_data_set_{0}'.format(i), 'input_0.pb')\n",
        "    output_test_data = os.path.join(model_dir, 'test_data_set_{0}'.format(i), 'output_0.pb')\n",
        "    \n",
        "    # convert protobuf tensors to np arrays using the TensorProto reader from ONNX\n",
        "    tensor = onnx.TensorProto()\n",
        "    with open(input_test_data, 'rb') as f:\n",
        "        tensor.ParseFromString(f.read())\n",
        "    \n",
        "    input_data = numpy_helper.to_array(tensor)\n",
        "    test_inputs.append(input_data)\n",
        "    \n",
        "    with open(output_test_data, 'rb') as f:\n",
        "        tensor.ParseFromString(f.read())\n",
        "    \n",
        "    output_data = numpy_helper.to_array(tensor)\n",
        "    output_processed = emotion_map(postprocess(output_data))[0]\n",
        "    test_outputs.append(output_processed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nbpresent": {
          "id": "c3f2f57c-7454-4d3e-b38d-b0946cf066ea"
        }
      },
      "source": [
        "### Show some sample images\n",
        "We use `matplotlib` to plot 3 test images from the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbpresent": {
          "id": "396d478b-34aa-4afa-9898-cdce8222a516"
        }
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize = (20, 20))\n",
        "for test_image in np.arange(3):\n",
        "    test_inputs[test_image].reshape(1, 64, 64)\n",
        "    plt.subplot(1, 8, test_image+1)\n",
        "    plt.axhline('')\n",
        "    plt.axvline('')\n",
        "    plt.text(x = 10, y = -10, s = test_outputs[test_image], fontsize = 18)\n",
        "    plt.imshow(test_inputs[test_image].reshape(64, 64), cmap = plt.cm.gray)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Run evaluation / prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "plt.figure(figsize = (16, 6), frameon=False)\n",
        "plt.subplot(1, 8, 1)\n",
        "\n",
        "plt.text(x = 0, y = -30, s = \"True Label: \", fontsize = 13, color = 'black')\n",
        "plt.text(x = 0, y = -20, s = \"Result: \", fontsize = 13, color = 'black')\n",
        "plt.text(x = 0, y = -10, s = \"Inference Time: \", fontsize = 13, color = 'black')\n",
        "plt.text(x = 3, y = 14, s = \"Model Input\", fontsize = 12, color = 'black')\n",
        "plt.text(x = 6, y = 18, s = \"(64 x 64)\", fontsize = 12, color = 'black')\n",
        "plt.imshow(np.ones((28,28)), cmap=plt.cm.Greys)    \n",
        "\n",
        "\n",
        "for i in np.arange(test_data_size):\n",
        "    \n",
        "    input_data = json.dumps({'data': test_inputs[i].tolist()})\n",
        "\n",
        "    # predict using the deployed model\n",
        "    r = json.loads(aci_service.run(input_data))\n",
        "    \n",
        "    if \"error\" in r:\n",
        "        print(r['error'])\n",
        "        break\n",
        "        \n",
        "    result = r['result'][0]\n",
        "    time_ms = np.round(r['time_in_sec'][0] * 1000, 2)\n",
        "    \n",
        "    ground_truth = test_outputs[i]\n",
        "    \n",
        "    # compare actual value vs. the predicted values:\n",
        "    plt.subplot(1, 8, i+2)\n",
        "    plt.axhline('')\n",
        "    plt.axvline('')\n",
        "\n",
        "    # use different color for misclassified sample\n",
        "    font_color = 'red' if ground_truth != result else 'black'\n",
        "    clr_map = plt.cm.Greys if ground_truth != result else plt.cm.gray\n",
        "\n",
        "    # ground truth labels are in blue\n",
        "    plt.text(x = 10, y = -70, s = ground_truth, fontsize = 18, color = 'blue')\n",
        "    \n",
        "    # predictions are in black if correct, red if incorrect\n",
        "    plt.text(x = 10, y = -45, s = result, fontsize = 18, color = font_color)\n",
        "    plt.text(x = 5, y = -22, s = str(time_ms) + ' ms', fontsize = 14, color = font_color)\n",
        "\n",
        "    \n",
        "    plt.imshow(test_inputs[i].reshape(64, 64), cmap = clr_map)\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Try classifying your own images!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Preprocessing functions take your image and format it so it can be passed\n",
        "# as input into our ONNX model\n",
        "\n",
        "import cv2\n",
        "\n",
        "def rgb2gray(rgb):\n",
        "    \"\"\"Convert the input image into grayscale\"\"\"\n",
        "    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])\n",
        "\n",
        "def resize_img(img):\n",
        "    \"\"\"Resize image to MNIST model input dimensions\"\"\"\n",
        "    img = cv2.resize(img, dsize=(64, 64), interpolation=cv2.INTER_AREA)\n",
        "    img.resize((1, 1, 64, 64))\n",
        "    return img\n",
        "\n",
        "def preprocess(img):\n",
        "    \"\"\"Resize input images and convert them to grayscale.\"\"\"\n",
        "    if img.shape == (64, 64):\n",
        "        img.resize((1, 1, 64, 64))\n",
        "        return img\n",
        "    \n",
        "    grayscale = rgb2gray(img)\n",
        "    processed_img = resize_img(grayscale)\n",
        "    return processed_img"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Replace the following string with your own path/test image\n",
        "# Make sure your image is square and the dimensions are equal (i.e. 100 * 100 pixels or 28 * 28 pixels)\n",
        "\n",
        "# Any PNG or JPG image file should work\n",
        "# Make sure to include the entire path with // instead of /\n",
        "\n",
        "# e.g. your_test_image = \"C://Users//vinitra.swamy//Pictures//emotion_test_images//img_1.png\"\n",
        "\n",
        "import matplotlib.image as mpimg\n",
        "\n",
        "if your_test_image != \"<path to file>\":\n",
        "    img = mpimg.imread(your_test_image)\n",
        "    plt.subplot(1,3,1)\n",
        "    plt.imshow(img, cmap = plt.cm.Greys)\n",
        "    print(\"Old Dimensions: \", img.shape)\n",
        "    img = preprocess(img)\n",
        "    print(\"New Dimensions: \", img.shape)\n",
        "else:\n",
        "    img = None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if img is None:\n",
        "    print(\"Add the path for your image data.\")\n",
        "else:\n",
        "    input_data = json.dumps({'data': img.tolist()})\n",
        "\n",
        "    try:\n",
        "        r = json.loads(aci_service.run(input_data))\n",
        "        result = r['result'][0]\n",
        "        time_ms = np.round(r['time_in_sec'][0] * 1000, 2)\n",
        "    except Exception as e:\n",
        "        print(str(e))\n",
        "\n",
        "    plt.figure(figsize = (16, 6))\n",
        "    plt.subplot(1,8,1)\n",
        "    plt.axhline('')\n",
        "    plt.axvline('')\n",
        "    plt.text(x = -10, y = -40, s = \"Model prediction: \", fontsize = 14)\n",
        "    plt.text(x = -10, y = -25, s = \"Inference time: \", fontsize = 14)\n",
        "    plt.text(x = 100, y = -40, s = str(result), fontsize = 14)\n",
        "    plt.text(x = 100, y = -25, s = str(time_ms) + \" ms\", fontsize = 14)\n",
        "    plt.text(x = -10, y = -10, s = \"Model Input image: \", fontsize = 14)\n",
        "    plt.imshow(img.reshape((64, 64)), cmap = plt.cm.gray)    \n",
        "     "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# remember to delete your service after you are done using it!\n",
        "\n",
        "aci_service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Conclusion\n",
        "\n",
        "Congratulations!\n",
        "\n",
        "In this tutorial, you have:\n",
        "- familiarized yourself with ONNX Runtime inference and the pretrained models in the ONNX model zoo\n",
        "- understood a state-of-the-art convolutional neural net image classification model (FER+ in ONNX) and deployed it in the Azure ML cloud\n",
        "- ensured that your deep learning model is working perfectly (in the cloud) on test data, and checked it against some of your own!\n",
        "\n",
        "Next steps:\n",
        "- If you have not already, check out another interesting ONNX/AML application that lets you set up a state-of-the-art [handwritten image classification model (MNIST)](https://github.com/Azure/MachineLearningNotebooks/tree/master/onnx/onnx-inference-mnist.ipynb) in the cloud! This tutorial deploys a pre-trained ONNX Computer Vision model for handwritten digit classification in an Azure ML virtual machine.\n",
        "- Keep an eye out for an updated version of this tutorial that uses ONNX Runtime GPU.\n",
        "- Contribute to our [open source ONNX repository on github](http://github.com/onnx/onnx) and/or add to our [ONNX model zoo](http://github.com/onnx/models)"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "viswamy"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "msauthor": "vinitra.swamy"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}