notebooks/210-ct-scan-live-inference/210-ct-scan-live-inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Live Inference and Benchmark CT-scan Data with OpenVINO\n",
    "\n",
    "This tutorial is part of a series on how to train, optimize, quantize and show live inference on a medical segmentation model. The goal is to accelerate inference on a kidney segmentation model. The [UNet](https://arxiv.org/abs/1505.04597) model is trained from scratch; the data is from [Kits19](https://github.com/neheller/kits19).\n",
    "\n",
    "This tutorial shows how to \n",
    "\n",
    "- Benchmark performance of the model\n",
    "- Show live inference with OpenVINO's async API and MULTI plugin\n",
    "\n",
    "To learn how this model was quantized, please see the [Convert and Quantize a UNet Model and Show Live Inference](../110-ct-segmentation-quantize/110-ct-segmentation-quantize.ipynb) tutorial. \n",
    "\n",
    "## Instructions\n",
    "\n",
    "This notebook needs a quantized OpenVINO IR model. We provide a pretrained model trained for 20 epochs with the full [Kits-19](https://github.com/neheller/kits19) frames dataset, which has an F1 score on the validation set of 0.9. The training code is available in the [PyTorch Monai Training](../110-ct-segmentation-quantize/pytorch-monai-training.ipynb) notebook. It also needs images from the Kits19 dataset, converted to 2D images. For demonstration purposes, this tutorial will download one converted CT scan to use for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "from monai.transforms import LoadImage\n",
    "from openvino.inference_engine import IECore\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from models.custom_segmentation import SegmentationModel\n",
    "from notebook_utils import benchmark_model, download_file, show_live_inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings\n",
    "\n",
    "To use the pretrained models, set `IR_PATH` to `\"pretrained_model/unet44.xml\"` and `COMPRESSED_MODEL_PATH` to `\"pretrained_model/quantized_unet44.xml\"`. To use a model that you trained or optimized yourself, adjust the model paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The directory that contains the IR model (xml and bin) files\n",
    "MODEL_PATH = \"pretrained_model/quantized_unet_kits19.xml\"\n",
    "# Uncomment the next line to use the FP16 model instead of the quantized model\n",
    "# MODEL_PATH = \"pretrained_model/unet_kits19.xml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark Model Performance\n",
    "To measure the inference performance of the IR model, we use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html), OpenVINO's inference performance measurement tool. Benchmark tool is a command line application that can be run in the notebook with `! benchmark_app` or `%sx benchmark_app`. \n",
    "\n",
    "In this tutorial, we use a wrapper function from [Notebook Utils](https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/utils/notebook_utils.ipynb). It prints the `benchmark_app` command with the chosen parameters.\n",
    "\n",
    "> NOTE: For the most accurate performance estimation, we recommended running `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app --help` to see all command line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = IECore()\n",
    "# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.\n",
    "device = \"MULTI:CPU,GPU\" if \"GPU\" in ie.available_devices else \"CPU\"\n",
    "# Uncomment one of the options below to benchmark on other devices\n",
    "# device = \"GPU\"\n",
    "# device = \"CPU\"\n",
    "# device = \"AUTO\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "# Benchmark model\n",
    "benchmark_model(model_path=MODEL_PATH, device=device, seconds=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and Prepare Data\n",
    "\n",
    "Download one validation video for live inference. We reuse the KitsDataset class that was also used in the training and quantization notebook that will be released later.\n",
    "\n",
    "Data is expected in `BASEDIR` defined in the Settings cell. `BASEDIR` should contain directories named `case_00000` to `case_00299`. If data for the case specified above does not exist yet, it will be downloaded and extracted in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directory that contains the CT scan data. This directory should contain subdirectories\n",
    "# case_00XXX where XXX is between 000 and 299\n",
    "BASEDIR = Path(\"kits19_frames_1\")\n",
    "# The CT scan case number. For example: 16 for data from the case_00016 directory\n",
    "# Currently only 117 is supported\n",
    "CASE = 117\n",
    "\n",
    "case_path = BASEDIR / f\"case_{CASE:05d}\"\n",
    "\n",
    "if not case_path.exists():\n",
    "    filename = download_file(\n",
    "        f\"https://storage.openvinotoolkit.org/data/test_data/openvino_notebooks/kits19/case_{CASE:05d}.zip\"\n",
    "    )\n",
    "    with zipfile.ZipFile(filename, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(path=BASEDIR)\n",
    "    os.remove(filename)  # remove zipfile\n",
    "    print(f\"Downloaded and extracted data for case_{CASE:05d}\")\n",
    "else:\n",
    "    print(f\"Data for case_{CASE:05d} exists\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Show Live Inference\n",
    "\n",
    "To show live inference on the model in the notebook, we use the asynchronous processing feature of OpenVINO Inference Engine.\n",
    "\n",
    "If you use a GPU device, with `device=\"GPU\"` or `device=\"MULTI:CPU,GPU\"` to do inference on an integrated graphics card, model loading will be slow the first time you run this code. The model will be cached, so after the first time model loading will be fast. See the [OpenVINO API tutorial](../002-openvino-api/002-openvino-api.ipynb) for more information on Inference Engine, including Model Caching.\n",
    "\n",
    "We use the `show_live_inference` function from [Notebook Utils](../utils/notebook_utils.ipynb) to show live inference. This function uses [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/)'s AsyncPipeline and Model API to perform asynchronous inference. After inference on the specified CT scan has completed, the total time and throughput (fps), including preprocessing and displaying, will be printed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Model and List of Image Files\n",
    "\n",
    "Load the segmentation model to Inference Engine with `SegmentationModel`, based on the [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/) Model API. This model implementation includes pre and post processing for the model. For `SegmentationModel` this includes the code to create an overlay of the segmentation mask on the original image/frame. Uncomment the next cell to see the implementation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SegmentationModel??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "122wcKhzXn3z",
    "tags": []
   },
   "outputs": [],
   "source": [
    "ie = IECore()\n",
    "segmentation_model = SegmentationModel(\n",
    "    ie=ie, model_path=Path(MODEL_PATH), sigmoid=True, rotate_and_flip=True\n",
    ")\n",
    "image_paths = sorted(case_path.glob(\"imaging_frames/*jpg\"))\n",
    "\n",
    "print(f\"{case_path.name}, {len(image_paths)} images\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Inference\n",
    "\n",
    "In the next cell, we run the `show live_inference` function, which loads the `segmentation_model` to the specified `device` (using caching for faster model loading on GPU devices), loads the images, performs inference, and displays the results on the frames loaded in `images` in real-time.\n",
    "\n",
    "We use `reader=LoadImage()` to read the images in the same way as in the [training](../110-ct-segmentation-quantize/pytorch-monai-training.ipynb) tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "=image_paths": "=image_paths[:5]"
    }
   },
   "outputs": [],
   "source": [
    "# Possible options for device include \"CPU\", \"GPU\", \"AUTO\", \"MULTI\"\n",
    "device = \"MULTI:CPU,GPU\" if \"GPU\" in ie.available_devices else \"CPU\"\n",
    "reader = LoadImage(image_only=True, dtype=np.uint8)\n",
    "\n",
    "show_live_inference(\n",
    "    ie=ie, image_paths=image_paths, model=segmentation_model, device=device, reader=reader\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "openvino_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}