4 - Keyword Spotting Demo with TensorFlow Lite.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/sensiml/AnalyticStudioTutorials/blob/master/sensiml_keyworld_spotting_recognition_with_tensorflow_lite_micro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keyword Spotting Demo with TensorFlow Lite\n",
    "\n",
    "*Note: You must have a SensiML Account to use this tutorial. You can sign up for free at https://sensiml.com/plans/community-edition/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-nH64mMtH7wO"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "assW5g5DH9YY"
   },
   "source": [
    "### SensiML Python SDK\n",
    "\n",
    "Install the SensiML Python SDK in the Google Collab environment by running the following cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ikLK27oprj8G"
   },
   "outputs": [],
   "source": [
    "!pip install sensiml -U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will initialize SensiML Python SDK by running the following cell. After connecting, you will be able to use the SensiML Python SDK to manage the data in your project, create queries, build and test models, and download firmware.\n",
    "\n",
    "*Further documentation for using the SensiML Python SDK can be found at https://sensiml.com/documentation/sensiml-python-sdk/api-methods/overview.html*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17
    },
    "id": "CkM3SHoDJIFs",
    "outputId": "37200b2f-b4ab-49aa-a513-9d86d46fc7ed"
   },
   "outputs": [],
   "source": [
    "from sensiml import *\n",
    "import sensiml.tensorflow.utils as sml_tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s9D22jJQr0mr"
   },
   "source": [
    "Next we are going to connect to our **Keyword Spotting - Demo** Project. Run the following cell to connect to the project using the SensiML Python SDK. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SKYKeIkr6SD"
   },
   "outputs": [],
   "source": [
    "client = SensiML()\n",
    "client.project = '<Your Project Name>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GzcU8vESsIMV"
   },
   "source": [
    "### Pipelines\n",
    "\n",
    "Pipelines are a key component of the SensiML workflow. Pipelines store the preprocessing, feature extraction and model building steps. When training a model, these steps are executed on the SensiML server. Once the model has been trained, the pipeline is converted to firmware code that will run on your target embedded device. For more documentation on pipelines see the [Getting Started with the SensiML Python SDK Documentation](https://sensiml.com/documentation/sensiml-python-sdk/getting-started-with-the-sensiml-python-sdk.html). To create a new empty pipeline, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qdoxoC0PQXaC",
    "outputId": "c63e35ad-ce54-4836-abd1-34d64f90f5ff"
   },
   "outputs": [],
   "source": [
    "client.pipeline = '<Your Pipeline name>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NbsKPtKQrucZ"
   },
   "source": [
    "## Query Your Data\n",
    "\n",
    "To select the data you want to use in your pipeline you need to add a query step. Queries give us a way to select and filter the data we want to use in our pipeline.\n",
    "\n",
    "1. Create a query for all labeled sensor data in your project\n",
    "\n",
    "We recommend using the **Prepare Data** page in the Analytics Studio at https://app.sensiml.cloud/ to create your query. Alternatively, you can also use the create_query API by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qVDsvm3dILjn"
   },
   "outputs": [],
   "source": [
    "client.create_query(\n",
    "    name=\"Q1\",\n",
    "    label_column=\"Label\",\n",
    "    columns=[\n",
    "        \"Microphone\",\n",
    "    ],\n",
    "    metadata_columns=[\"Subject\"],\n",
    "    segmenter=\"Training Session\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the segments in your query by running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "id": "VX53A75aIoHu",
    "outputId": "6d6af329-7f08-4c96-be20-b23b3451c613"
   },
   "outputs": [],
   "source": [
    "q = client.get_query(\"Q1\")\n",
    "q.statistics_segments().groupby('Label').size().plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9i5tnXO-sVf2"
   },
   "source": [
    "\n",
    "### Assembling a pipeline \n",
    "\n",
    "The pipeline for this tutorial will consist of the following steps:\n",
    "\n",
    "1.   The **Input Query** which specifies what data is being fed into the model\n",
    "2.   The **Feature Generators** which specify which features should be extracted from the raw time series data\n",
    "3.   The **Feature Transform** which specify how to transform the features after extraction. In this case it is to scale them to 1 byte each\n",
    "4.   The **Feature Selector** which selects the best features. In this case we are using the custom feature selector to downsample the data. \n",
    "\n",
    "\n",
    "The code in the following cell sets our initial variables, then specifies each step in the pipeline. For now, you don't need to dig into each of these steps, but just know that the end result will be a feature vector scaled to 1 byte values for each of the segments that were labeled in the Data Capture Lab. We will use these features as input to our TensorFlow model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iaolQVs-JKUc",
    "outputId": "8b11206c-2476-41fb-9699-7815c2f4ad0a"
   },
   "outputs": [],
   "source": [
    "client.project.columns()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vg3J1NdlSFQz"
   },
   "outputs": [],
   "source": [
    "num_cols=12\n",
    "row_size=10\n",
    "\n",
    "client.pipeline.reset()\n",
    "client.pipeline.set_input_query(\"Q1\")\n",
    "\n",
    "client.pipeline.add_transform(\"Windowing\", params={\"window_size\": 400,\n",
    "                                                \"delta\": 400,\n",
    "                                                \"train_delta\": 0,\n",
    "                                                \"return_segment_index\": False,\n",
    "                                                })\n",
    "\n",
    "client.pipeline.add_feature_generator([{'name':'MFCC', 'params':{\"columns\": [\"Microphone\"],\n",
    "                                    \"sample_rate\": 16000,\n",
    "                                    \"cepstra_count\": num_cols,\n",
    "                                    }}])\n",
    "\n",
    "client.pipeline.add_transform(\"Feature Cascade\", params={\"num_cascades\": row_size ,\n",
    "                                \"slide\": True,\n",
    "                                })\n",
    "\n",
    "client.pipeline.add_transform(\"Min Max Scale\", params={\"min_bound\": 0,\n",
    "                                \"max_bound\": 255,\n",
    "                                \"pad\": 0,\n",
    "                                \"feature_min_max_defaults\":{'minimum':-500000, 'maximum':500000.0},\n",
    "                                })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mxTbqlgguO_b"
   },
   "source": [
    "To see the steps currently added to your pipeline you can use the describe method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qErWtKxUuLoJ",
    "outputId": "7cccf4ba-b316-4868-9eb3-b80b3b759213"
   },
   "outputs": [],
   "source": [
    "client.pipeline.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bXLiev6csdcZ"
   },
   "source": [
    "### Executing a Pipeline\n",
    "\n",
    "At this point the pipeline has not been executed yet, we are just assembling the steps. To run the pipeline, execute the following cell. This will print out a summary of the pipeline steps that are being run followed by the status of the pipeline execution on the server.\n",
    "\n",
    "Once the pipeline is finished running, the results will be stored in the variable *fv_t*. A summary of the execution is stored in the *s_t* variable. \n",
    "\n",
    "\n",
    "The pipeline will report status updates that tell you which step is currently running, how many batches (parallel steps) have been run, and how many steps are left. This process is run on the server, so if you get disconnected you can run the client.pipeline.get_results() to check the status of your pipeline.\n",
    "\n",
    "Another thing to note is that intermediate steps are cached. you can retrieve data from intermediate steps by using client.pipeline.data(<index_of_step>). If you make any changes to your pipeline, only steps after the change will need to rerun.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F1IBGufgKFJl",
    "outputId": "53706bde-bae2-4c21-b949-f08d25b86a95"
   },
   "outputs": [],
   "source": [
    "fv_s, s= client.pipeline.execute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I7-lm1z6sqDv"
   },
   "source": [
    "## SensiML Features and TensorFlow Model\n",
    "\n",
    "Now we have our features for this model, we will go ahead and train a TensorFlow Model in the colab environment. We will start by splitting our dataset into train, test and validate groups. The SensiML Python SDK has a built-in function for performing this split. You can also pass in the validation data test sizes. By default, they are set to 10% each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s5OJ9DTVSPbz",
    "outputId": "301ab1ce-cc14-4d2e-a38c-a5704faf4f4e"
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "x_train, x_validate, x_test, y_train, y_validate, y_test,  class_map = \\\n",
    "    client.pipeline.features_to_tensor(fv_s, test=.1, validate=.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xtp0HjuLss-B"
   },
   "source": [
    "### Feature Visualization\n",
    "Let's take a quick look at the features that we have created. Run the following cell to see the heatmap (left) along with the raw features (right).  To switch which event you are looking at change the event_index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "id": "MuNnS2Y_GgI-",
    "outputId": "f8944625-2f84-4894-b28e-ccb496ebe478"
   },
   "outputs": [],
   "source": [
    "import seaborn as sn\n",
    "event_index=50\n",
    "sn.heatmap(x_train[event_index].reshape(row_size,-1).T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I7PGOlc1s0Yd"
   },
   "source": [
    "### Data Augmentation\n",
    "\n",
    "It is often useful to perform data augmentation to make models more robust to unseen data. One data augmentation strategy is to apply masks in the feature space. For this tutorial, we will add masks randomly across time and sensors. To implement the masking in TensorFlow we use the tf.data API. The steps in the following cell are described by:\n",
    "\n",
    "1.   Specify variables which describe the properties of our masking\n",
    "2.   Convert our training dataset that is currently a dataframe into a tf.data bject\n",
    "3.   Add a noise layer which randomly adds noise to the features.\n",
    "4.   Apply a time mask will randomly mask up to rand_max time slices.\n",
    "5.   Apply a sensor mask which will randomly mask up to rand_max sensors\n",
    "6.   Apply a shuffler which will randomize our dataset after each training iteration.\n",
    "\n",
    "For more information about the tf.data.DataSet see the documentation [here](https://www.tensorflow.org/guide/data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4kNcQ9ie87zX"
   },
   "outputs": [],
   "source": [
    "rand_max = 1\n",
    "min_noise=-10\n",
    "max_noise=10\n",
    "mask_value=127\n",
    "batch_size =32\n",
    "\n",
    "data  = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "noise_ds = data.map(lambda image, label: sml_tf.add_image_noise(image, label, rand_max, min_noise, max_noise))\n",
    "freq_ds = noise_ds.map(lambda image, label: sml_tf.mask_image_row(image, label, rand_max, num_cols, mask_value))\n",
    "masked_ds = freq_ds.map(lambda image, label: sml_tf.mask_image_column(image, label, row_size, rand_max, num_cols, mask_value))\n",
    "\n",
    "\n",
    "# Augmented Data\n",
    "shuffle_aug = masked_ds.shuffle(buffer_size=x_train.shape[0], reshuffle_each_iteration=True).batch(batch_size)\n",
    "\n",
    "# Shuffled Data\n",
    "shuffle_ds = data.shuffle(buffer_size=x_train.shape[0], reshuffle_each_iteration=True).batch(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l0nPxn6fs3p3"
   },
   "source": [
    "To visualize the tf data you can use this helper function, switch the index to look at different feature maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "I4Q5cxE8GSDH",
    "outputId": "2d70cd51-6763-4fd8-9523-a8c4fb0e7da3"
   },
   "outputs": [],
   "source": [
    "sml_tf.visualize_tf_data(masked_ds, index=7, row_size=row_size, num_cols=num_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Q_5CRXys5sP"
   },
   "source": [
    "### Specifying the TensorFlow Model\n",
    "\n",
    "The Next step is to define what our TensorFlow model looks like. For this tutorial we are going to use the TensorFlow Keras API to create a [Convolution Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN). When you are building a model to deploy on a microcontroller, it is important to remember that all functions of tensorflow are not suitable for a micro controller. Additionally, only a subset of TensorFlow functions are available as part of TensorFlow Lite Micro. For a full list of available functions see the [all_ops_resolver.cc](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/kernels/all_ops_resolver.cc). \n",
    "\n",
    "For this tutorial use a 1D convolution followed by a 2D convolution and finally a fully connected layer to efficiently classify the boxing gestures. Our aim is to limit the number and size of every layer in the model to only those necessary to get our desired accuracy. Often you will find that you need to make a trade off between latency/memory usage and accuracy in order to get a model that will work well on your microcontroller.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gyn9FdTmSQzQ",
    "outputId": "d23608b9-2cf9-4626-8fa3-b8858f13c577"
   },
   "outputs": [],
   "source": [
    "optimization_metric = \"accuracy\"\n",
    "# We'll use Keras to create a simple model architecture\n",
    "from tensorflow.keras import layers\n",
    "import tensorflow as tf\n",
    "\n",
    "tf_model = tf.keras.Sequential()\n",
    "tf_model.add(layers.Reshape((10, 12, 1), input_shape=(120,)))\n",
    "tf_model.add(layers.Conv2D(32, (2, 6), padding=\"valid\", activation=\"relu\",  strides=[2,2]))\n",
    "tf_model.add(layers.MaxPool2D((2, 2)))\n",
    "tf_model.add(layers.Dropout(0.1))\n",
    "tf_model.add(layers.Conv2D(16, (2, 2), padding=\"valid\", activation=\"relu\"))\n",
    "#tf_model.add(layers.Dropout(0.1))\n",
    "#print(tf_model.summary())\n",
    "#tf_model.add(layers.MaxPool2D((1, 2), padding=\"valid\"))\n",
    "tf_model.add(layers.Flatten())\n",
    "# Final layer is a single neuron, since we want to output a single value\n",
    "tf_model.add(layers.Dense(18, activation='relu', ))\n",
    "tf_model.add(layers.Dense(len(class_map.keys()), activation='sigmoid'))\n",
    "# Compile the model using a standard optimizer and loss function for regression\n",
    "tf_model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=[optimization_metric])\n",
    "tf_model.summary()\n",
    "\n",
    "train_history = {'loss':[], 'val_loss':[], 'accuracy':[], 'val_accuracy':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RR3rjeKZtC7H"
   },
   "source": [
    "### Training the TensorFlow Model\n",
    "\n",
    "After defining the model graph, it is time to train the model. Training a NN consists of iterating through batches of your training dataset multiple times, each time it loops through the entire training set is called an epoch. For each batch of data, the loss function is computed and the weights of the layers in the network are adjusted.  \n",
    "\n",
    "The following cell will loop through the training data num_iterations of times. Each time running a specific number of epochs. After each iteration, the visualizations for the loss, accuracy and confusion matrix will be updated for both the validation and training data sets. You can use this to see how the model is progressing. \n",
    "\n",
    "Sometimes the initialization of the model gets stuck in a bad state, we put a check in that will call reset_weights to reinitialize the network weights if the model is stuck in a low accuracy state. Other methods could be to adjust the learning rate. If your reaches the desired accuracy before the training completes, hit the stop button at the left side of the cell and you can continue the tutorial with the model at its current state. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 999
    },
    "id": "VxAMIhbNSVxK",
    "outputId": "56438391-b96e-4a34-f29a-132d0dfb89ca"
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "num_iterations=10\n",
    "for i in range(num_iterations):\n",
    "    history = tf_model.fit( shuffle_aug, #shuffle_ds\n",
    "                              epochs=50,\n",
    "                              batch_size=batch_size, \n",
    "                              validation_data=(x_validate, y_validate),\n",
    "                              verbose=0)\n",
    "    clear_output()    \n",
    "\n",
    "    for key in train_history:\n",
    "        train_history[key].extend(history.history[key])\n",
    "\n",
    "    sml_tf.plot_training_results(tf_model, train_history, x_train, y_train, x_validate, y_validate)\n",
    "\n",
    "    if train_history['accuracy'][-1] <= .3 and train_history['accuracy'][-25] <= (train_history['accuracy'][-1]+5):\n",
    "        print('reinitializing weights')\n",
    "        sml_tf.reset_weights(tf_model)\n",
    "        train_history = {'loss':[], 'val_loss':[], 'accuracy':[], 'val_accuracy':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5DNI0IlzSoBF"
   },
   "source": [
    "### Quantize the TensorFlow Model\n",
    "\n",
    "Now that you have trained a neural network with TensorFlow, we are going to use the built-in tools to quantize it. Quantization of NN allows use to reduce the model size by up to 4x by converting the network weights from 4-byte floating point values to 1-byte uint8. This can be done without sacrificing much in terms of accuracy. The best way to perform quantization is still an active area of research. For this tutorial, we will use the built-in methods that are provided as part of TensorFlow. \n",
    "\n",
    "*   The ```representative_dataset_generator()``` function is necessary to provide statistical information about your dataset in order to quantize the model weights appropriately. \n",
    "*   The TFLiteConverter is used to convert a TensorFlow model into a TensorFlow Lite model. The TensorFlow Lite model is stored as a [flatbuffer](https://google.github.io/flatbuffers/) which allows us to easily store and access it on embedded systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mrzypWkwbDlZ",
    "outputId": "95753430-23b0-4b43-8435-e03c835d7654"
   },
   "outputs": [],
   "source": [
    "def representative_dataset_generator():  \n",
    "  for value in x_test:   \n",
    "  # Each scalar value must be inside of a 2D array that is wrapped in a list    \n",
    "    yield [np.array(value, dtype=np.float32, ndmin=2)]\n",
    "\n",
    "# Unquantized Model\n",
    "converter = tf.lite.TFLiteConverter.from_keras_model(tf_model)\n",
    "tflite_model_full = converter.convert()\n",
    "print(\"Full Model Size\", len(tflite_model_full))\n",
    "\n",
    "# Quantized Model\n",
    "converter = tf.lite.TFLiteConverter.from_keras_model(tf_model)\n",
    "converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]\n",
    "converter.experimental_new_converter = False\n",
    "converter.representative_dataset = representative_dataset_generator\n",
    "tflite_model_quant = converter.convert()\n",
    "\n",
    "print(\"Quantized Model Size\", len(tflite_model_quant))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NsHN50BLAbn0"
   },
   "source": [
    "As you can see the Quantized model is significantly smaller than the unquantized model. An additional benefit of quantizing the model is that tensorflow lite micro is able to take advantage of specialized instructions on cortex M chips using the [cmsis-nn](http://www.keil.com/pack/doc/cmsis/NN/html/index.html) DSP library which gives another huge boost in performance. For more information on TensorFlow Lite for microcontrollers you can check out the excellent [tinyml](https://www.oreilly.com/library/view/tinyml/9781492052036/) book by Pete Warden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UTV_LGyZtJAz"
   },
   "source": [
    "Now that you have trained your model, its time to upload it as the final step in your pipeline. From here you will be able to test the entire pipeline against test data as well as download the firmware which can be flashed to run locally on your embedded device. To do this we will use the **Load Model TF Micro** function. We will convert the tflite model to and upload it to the SensiML Cloud server. \n",
    "\n",
    "*Note: An additional parameter that can specified here is the threshold. We set it to .2 in this example. Any classification with a value less than the threshold will be returned as 0 which is reserved for an **unknown** classification.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wMonr9XySkR-",
    "outputId": "49ffaad4-692c-4af5-ec10-5b1ea2d6b69b"
   },
   "outputs": [],
   "source": [
    "class_map_tmp = {k:v+1 for k,v in class_map.items()} #increment by 1 as 0 corresponds to unknown\n",
    "\n",
    "client.pipeline.set_training_algorithm(\"Load Model TF Micro\",\n",
    "                                    params={\"model_parameters\": {'tflite': sml_tf.convert_tf_lite(tflite_model_quant)},\n",
    "                                            \"class_map\": class_map_tmp,\n",
    "                                            \"estimator_type\": \"classification\",\n",
    "                                            \"threshold\": 0.9, # must be above this value otherwise is unknown\n",
    "                                            })\n",
    "\n",
    "client.pipeline.set_validation_method(\"Recall\", params={})\n",
    "\n",
    "client.pipeline.set_classifier(\"TF Micro\", params={})\n",
    "\n",
    "client.pipeline.set_tvo()\n",
    "\n",
    "results, stats = client.pipeline.execute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R27ebOmCtM6n"
   },
   "source": [
    "### Model summary\n",
    "After executing the pipeline, the cloud computes a model summary as well as a confusion matrix. The model summary gives a quick overview of the model performance so we can see what the accuracy of the quantized model was across our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kXyFXjOqScPa",
    "outputId": "c1c1d079-7b7d-49e2-9be2-923b0526ea5c"
   },
   "outputs": [],
   "source": [
    "results.summarize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jMJk8VhZtPio"
   },
   "source": [
    "### Confusion Matrix\n",
    "\n",
    "The confusion matrix provides information not only about the accuracy, but also what sort of misclassifications occurred. The confusion matrix is often one of the best ways to understand how your model is performing, as you can see which classes are difficult to distinguish between. The confusion matrix here also includes the Sensitivity and Positive Predictivity for each class along with the overall accuracy. If you raise the threshold value, you will notice that some of the class start showing up as having UNK values. This corresponds to an unknown class and is useful for filtering out False Positives or detecting anomalous states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZjplLfn3TGsq",
    "outputId": "4586a696-7a59-41d8-9de0-87ed012010a2"
   },
   "outputs": [],
   "source": [
    "model = results.configurations[0].models[0]\n",
    "model.confusion_matrix_stats['validation']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m_UfQrlktR8H"
   },
   "source": [
    "*Finally*, we save the model knowledge pack with a name. This tells the server to persist this model and will let us retrieve and view the metrics in the future. Models that are not saved will be deleted when the pipeline is rerun. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bx4OtmMrvErx"
   },
   "outputs": [],
   "source": [
    "model.knowledgepack.save(\"TensorFlow_model_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RbnjKa6etYPn"
   },
   "source": [
    "### Offline Model Validation\n",
    "\n",
    "After saving your model, you can return back to the Analytics Studio at https://app.sensiml.cloud to continue validating and generating your firmware. To test your model against any of the captured data files select the pipeline and \n",
    "\n",
    "1.   Go to the Test Model tab of the Analytics Studio.\n",
    "2.   Select the model you want to test.\n",
    "3.   Select any of the capture files in the Project.\n",
    "4.   Click Compute Accuracy to classify that capture using the selected model.\n",
    "\n",
    "The model will be compiled in the SensiML Cloud and the output of the classification will be returned. The graph shows the segment start and segment classified for all of the detected events.\n",
    "\n",
    "We can also test the model through the SensiML Python SDK by selecting a capture and running it through the model recognize_signal API below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vvFHsXbOZg0j"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "uAkYdp63TOpo",
    "outputId": "6b35475d-93fb-42c8-d7fc-597d625d3802"
   },
   "outputs": [],
   "source": [
    "client.list_captures()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-r-7FYXhTSzD",
    "outputId": "7e8b8248-2fa1-4704-d78e-c918204854af"
   },
   "outputs": [],
   "source": [
    "res,s = model.recognize_signal(capturefile=\"off_017c4098_nohash_2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "Zx5AuKVRyrYh",
    "outputId": "cc3b9a77-a110-491c-d954-8d9c8ac7d95e"
   },
   "outputs": [],
   "source": [
    "res"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "sensiml_audio_recognition.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}