notebooks/207-vision-paddlegan-superresolution/207-vision-paddlegan-superresolution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7607ce35-db52-4e1c-add7-8abed748de6a",
   "metadata": {},
   "source": [
    "# Super Resolution with PaddleGAN and OpenVINO\n",
    "\n",
    "This notebook demonstrates converting the RealSR (real-world super-resolution) model from [PaddlePaddle/PaddleGAN](https://github.com/PaddlePaddle/PaddleGAN) to OpenVINO's Intermediate Representation (IR) format, and shows inference results on both the PaddleGAN and IR models.\n",
    "\n",
    "For more information about the various PaddleGAN superresolution models, see [PaddleGAN's documentation](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/tutorials/single_image_super_resolution.md). For more information about RealSR, see the [research paper](https://openaccess.thecvf.com/content_CVPRW_2020/papers/w31/Ji_Real-World_Super-Resolution_via_Kernel_Estimation_and_Noise_Injection_CVPRW_2020_paper.pdf) from CVPR 2020.\n",
    "\n",
    "This notebook works best with small images (up to 800x600)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a9e69d-27cc-421a-b526-7cc31cbb06bd",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e41056f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import time\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import paddle\n",
    "from IPython.display import HTML, FileLink, ProgressBar, clear_output, display\n",
    "from IPython.display import Image as DisplayImage\n",
    "from openvino.inference_engine import IECore\n",
    "from paddle.static import InputSpec\n",
    "from PIL import Image\n",
    "from ppgan.apps import RealSRPredictor\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import NotebookAlert"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8aaf4da-a840-4c07-bd4c-b703fa5c58fa",
   "metadata": {},
   "source": [
    "## Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "764178fb-c9b2-4005-8dc8-84018b12c439",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The filenames of the downloaded and converted models\n",
    "MODEL_NAME = \"paddlegan_sr\"\n",
    "MODEL_DIR = Path(\"model\")\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "model_path = MODEL_DIR / MODEL_NAME\n",
    "ir_path = model_path.with_suffix(\".xml\")\n",
    "onnx_path = model_path.with_suffix(\".onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ad5fe05",
   "metadata": {},
   "source": [
    "## Inference on PaddlePaddle model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e51e484-6d4d-4ce1-a757-de8790ee6669",
   "metadata": {},
   "source": [
    "### Investigate PaddleGAN model\n",
    "\n",
    "The [PaddleGAN documentation](https://github.com/PaddlePaddle/PaddleGAN) explains to run the model with `sr.run()`. Let's see what that function does, and check other relevant functions that are called from that function. Adding `??` to the methods shows the docstring and source code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "151fd351",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Running this cell will download the model weights if they have not been downloaded before\n",
    "# This may take a while\n",
    "sr = RealSRPredictor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6ece28c-9f84-4f91-9c7b-48376ddb1e9b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sr.run??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d78cc588-fb6f-43b3-a97a-bef8ae92b011",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sr.run_image??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f701eaca-781b-4ae5-86b1-76465448f811",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sr.norm??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a8cc86a-6e13-432f-9b1b-ca0a5588973c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sr.denorm??"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f3417c1-5cac-4d88-bd37-15982525ebb4",
   "metadata": {},
   "source": [
    "The run checks whether the input is an image or a video. For an image, it loads the image as an RGB image, normalizes it, and converts it to a Paddle tensor. It is propagated to the network by calling `self.model()` and then \"denormalized\". The normalization function simply divides all image values by 255. This converts an image with integer values in the range of 0 to 255 to an image with floating point values in the range of 0 to 1. The denormalization function transforms the output from network shape (C,H,W) to image shape (H,W,C). It then clips the image values between 0 and 255, and converts the image to a standard RGB image with integer values in the range of 0 to 255.\n",
    "\n",
    "To get more information about the model, we can check what it looks like with `sr.model??`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "289f5d1e-ffa7-4729-a68e-873289043585",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# sr.model??"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf962925-7176-4f25-bf7a-59cf86433aa8",
   "metadata": {},
   "source": [
    "### Do Inference\n",
    "\n",
    "To show inference on the PaddlePaddle model, set PADDLEGAN_INFERENCE to True in the cell below. Performing inference may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95f6554f-62ec-4e7d-aeb8-c43e09e219b7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set PADDLEGAN_INFERENCE to True to show inference on the PaddlePaddle model.\n",
    "# This may take a long time, especially for larger images.\n",
    "#\n",
    "PADDLEGAN_INFERENCE = False\n",
    "if PADDLEGAN_INFERENCE:\n",
    "    # load the input image and convert to tensor with input shape\n",
    "    IMAGE_PATH = Path(\"data/coco_tulips.jpg\")\n",
    "    image = cv2.cvtColor(cv2.imread(str(IMAGE_PATH)), cv2.COLOR_BGR2RGB)\n",
    "    input_image = image.transpose(2, 0, 1)[None, :, :, :] / 255\n",
    "    input_tensor = paddle.to_tensor(input_image.astype(np.float32))\n",
    "    if max(image.shape) > 400:\n",
    "        NotebookAlert(\n",
    "            f\"This image has shape {image.shape}. Doing inference will be slow \"\n",
    "            \"and the notebook may stop responding. Set PADDLEGAN_INFERENCE to False \"\n",
    "            \"to skip doing inference on the PaddlePaddle model.\",\n",
    "            \"warning\",\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f49d0770-d885-4e78-addf-1fc51022540f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if PADDLEGAN_INFERENCE:\n",
    "    # Do inference, and measure how long it takes\n",
    "    print(f\"Start superresolution inference for {IMAGE_PATH.name} with shape {image.shape}...\")\n",
    "    start_time = time.perf_counter()\n",
    "    sr.model.eval()\n",
    "    with paddle.no_grad():\n",
    "        result = sr.model(input_tensor)\n",
    "    end_time = time.perf_counter()\n",
    "    duration = end_time - start_time\n",
    "    result_image = (\n",
    "        (result.numpy().squeeze() * 255).clip(0, 255).astype(\"uint8\").transpose((1, 2, 0))\n",
    "    )\n",
    "    print(f\"Superresolution image shape: {result_image.shape}\")\n",
    "    print(f\"Inference duration: {duration:.2f} seconds\")\n",
    "    plt.imshow(result_image);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adb6be9f",
   "metadata": {},
   "source": [
    "## Convert PaddleGAN Model to ONNX and OpenVINO IR\n",
    "\n",
    "To convert the PaddlePaddle model to OpenVINO IR, we first convert the model to ONNX, and then convert the ONNX model to the IR format.\n",
    "\n",
    "### Convert PaddlePaddle Model to ONNX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6735bf0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Ignore PaddlePaddle warnings:\n",
    "# The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1)\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "sr.model.eval()\n",
    "# ONNX export requires an input shape in this format as parameter\n",
    "x_spec = InputSpec([None, 3, 299, 299], \"float32\", \"x\")\n",
    "paddle.onnx.export(sr.model, str(model_path), input_spec=[x_spec], opset_version=13)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f8c13a",
   "metadata": {},
   "source": [
    "### Convert ONNX Model to OpenVINO IR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d192786-908a-4cdb-9cc6-695c2b24ff54",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Uncomment the command below to show Model Optimizer help, which shows the possible arguments for Model Optimizer\n",
    "# ! mo --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed6cc19f-4b8c-4754-8473-1d905b89d476",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not ir_path.exists():\n",
    "    print(\"Exporting ONNX model to IR... This may take a few minutes.\")\n",
    "    ! mo --input_model $onnx_path --input_shape \"[1,3,299,299]\" --model_name $MODEL_NAME --output_dir \"$MODEL_DIR\" --data_type \"FP16\" --log_level \"CRITICAL\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5317e89-0b52-4d4a-af5d-d0b4a136c0a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-03T15:46:41.605511Z",
     "iopub.status.busy": "2021-08-03T15:46:41.605374Z",
     "iopub.status.idle": "2021-08-03T15:46:41.607174Z",
     "shell.execute_reply": "2021-08-03T15:46:41.606871Z",
     "shell.execute_reply.started": "2021-08-03T15:46:41.605498Z"
    }
   },
   "source": [
    "## Do Inference on IR Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f0f83cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Read network and get input and output names\n",
    "ie = IECore()\n",
    "net = ie.read_network(ir_path)\n",
    "input_layer = next(iter(net.input_info.keys()))\n",
    "output_layer = next(iter(net.outputs.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dba4b8d3-6048-4d91-896f-922d17e574c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load and show image\n",
    "IMAGE_PATH = Path(\"data/coco_tulips.jpg\")\n",
    "image = cv2.cvtColor(cv2.imread(str(IMAGE_PATH)), cv2.COLOR_BGR2RGB)\n",
    "if max(image.shape) > 800:\n",
    "    NotebookAlert(\n",
    "        f\"This image has shape {image.shape}. The notebook works best with images with \"\n",
    "        \"a maximum side of 800x600. Larger images may work well, but inference may \"\n",
    "        \"be slow\",\n",
    "        \"warning\",\n",
    "    )\n",
    "plt.imshow(image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e5c959b-4456-440e-9bd7-5a900b49ac6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reshape network to image size\n",
    "net.reshape({input_layer: [1, 3, image.shape[0], image.shape[1]]})\n",
    "# Load network to the CPU device (this may take a few seconds)\n",
    "exec_net = ie.load_network(net, \"CPU\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "499cfbc6-e935-4f06-b786-43b3020e72a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Convert image to network input shape and divide pixel values by 255\n",
    "# See \"Investigate PaddleGAN model\" section\n",
    "input_image = image.transpose(2, 0, 1)[None, :, :, :] / 255\n",
    "start_time = time.perf_counter()\n",
    "# Do inference\n",
    "ir_result = exec_net.infer({input_layer: input_image})\n",
    "end_time = time.perf_counter()\n",
    "duration = end_time - start_time\n",
    "print(f\"Inference duration: {duration:.2f} seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f8d9340-4151-475e-a074-9907adba7c9b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get result array in CHW format\n",
    "result_array = ir_result[output_layer].squeeze()\n",
    "# Convert array to image with same method as PaddleGAN:\n",
    "# Multiply by 255, clip values between 0 and 255, convert to HWC INT8 image\n",
    "# See \"Investigate PaddleGAN model\" section\n",
    "image_super = (result_array * 255).clip(0, 255).astype(\"uint8\").transpose((1, 2, 0))\n",
    "# Resize image with bicubic upsampling for comparison\n",
    "image_bicubic = cv2.resize(image, tuple(image_super.shape[:2][::-1]), interpolation=cv2.INTER_CUBIC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ef4f90a-1d3a-4e9c-a667-a300fe7b1559",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.imshow(image_super)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563ef395-63f1-4b2f-856d-a1fd9a114c8d",
   "metadata": {},
   "source": [
    "### Show Animated Gif\n",
    "\n",
    "To visualize the difference between the bicubic image and the superresolution image, we create an imated gif that switches between both versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "540d1cef-0c64-47f3-86f3-e86442093dc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result_pil = Image.fromarray(image_super)\n",
    "bicubic_pil = Image.fromarray(image_bicubic)\n",
    "gif_image_path = OUTPUT_DIR / Path(IMAGE_PATH.stem + \"_comparison.gif\")\n",
    "final_image_path = OUTPUT_DIR / Path(IMAGE_PATH.stem + \"_super.png\")\n",
    "\n",
    "result_pil.save(\n",
    "    fp=str(gif_image_path),\n",
    "    format=\"GIF\",\n",
    "    append_images=[bicubic_pil],\n",
    "    save_all=True,\n",
    "    duration=1000,\n",
    "    loop=0,\n",
    ")\n",
    "\n",
    "result_pil.save(fp=str(final_image_path), format=\"png\")\n",
    "DisplayImage(open(gif_image_path, \"rb\").read(), width=1920 // 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ab71ad-e254-477a-a334-3bf1ee509454",
   "metadata": {},
   "source": [
    "### Create Comparison Video\n",
    "\n",
    "Create a video with a \"slider\", showing the bicubic image to the right and the superresolution image on the left. \n",
    "\n",
    "For the video, the superresolution and bicubic image are resized by a factor two to improve processing speed. This gives an indication of the superresolution effect. The video is saved as an .avi video. You can click on the link to download the video, or open it directly from the images directory, and play it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df667cd6-333b-49b0-bad6-684cdd7b8ada",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "FOURCC = cv2.VideoWriter_fourcc(*\"MJPG\")\n",
    "IMAGE_PATH = Path(IMAGE_PATH)\n",
    "result_video_path = OUTPUT_DIR / Path(f\"{IMAGE_PATH.stem}_comparison_paddlegan.avi\")\n",
    "video_target_height, video_target_width = (\n",
    "    image_super.shape[0] // 2,\n",
    "    image_super.shape[1] // 2,\n",
    ")\n",
    "\n",
    "out_video = cv2.VideoWriter(\n",
    "    str(result_video_path),\n",
    "    FOURCC,\n",
    "    90,\n",
    "    (video_target_width, video_target_height),\n",
    ")\n",
    "\n",
    "resized_result_image = cv2.resize(image_super, (video_target_width, video_target_height))[\n",
    "    :, :, (2, 1, 0)\n",
    "]\n",
    "resized_bicubic_image = cv2.resize(image_bicubic, (video_target_width, video_target_height))[\n",
    "    :, :, (2, 1, 0)\n",
    "]\n",
    "\n",
    "progress_bar = ProgressBar(total=video_target_width)\n",
    "progress_bar.display()\n",
    "\n",
    "for i in range(2, video_target_width):\n",
    "    # Create a frame where the left part (until i pixels width) contains the\n",
    "    # superresolution image, and the right part (from i pixels width) contains\n",
    "    # the bicubic image\n",
    "    comparison_frame = np.hstack(\n",
    "        (\n",
    "            resized_result_image[:, :i, :],\n",
    "            resized_bicubic_image[:, i:, :],\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # create a small black border line between the superresolution\n",
    "    # and bicubic part of the image\n",
    "    comparison_frame[:, i - 1 : i + 1, :] = 0\n",
    "    out_video.write(comparison_frame)\n",
    "    progress_bar.progress = i\n",
    "    progress_bar.update()\n",
    "out_video.release()\n",
    "clear_output()\n",
    "\n",
    "video_link = FileLink(result_video_path)\n",
    "video_link.html_link_str = \"<a href='%s' download>%s</a>\"\n",
    "display(HTML(f\"The video has been saved to {video_link._repr_html_()}\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0faf010-1262-4c84-b4da-642ab4ffdba1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "openvino_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}