examples/JUPYTER/Demo_labeledexample_MouseReaching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N0gDJMy1ywm8"
   },
   "source": [
    "# DeepLabCut Toolbox - DEMO (mouse reaching)\n",
    "https://github.com/AlexEMG/DeepLabCut\n",
    "\n",
    "#### The notebook accompanies the following user-guide:\n",
    "\n",
    "Nath\\*, Mathis\\* et al. *Using DeepLabCut for markerless pose estimation during behavior across species* Nature Protocols, 2019: https://www.nature.com/articles/s41596-019-0176-0\n",
    "\n",
    "This notebook starts from an already initialized project with labeled data.\n",
    "\n",
    "**Data:** dataset is from Mathis et al. *Somatosensory Cortex Plays an Essential Role in Forelimb Motor Adaptation in Mice* Neuron, 2017: DOI:https://doi.org/10.1016/j.neuron.2017.02.049\n",
    "\n",
    "This notebook illustrates how to:\n",
    "- plot the labeled images\n",
    "- train a network\n",
    "- evaluate a network\n",
    "- analyze a novel video\n",
    "- create an automatically labeled video \n",
    "- plot the trajectories \n",
    "- identify outlier frames\n",
    "- annotate the outlier frames manually\n",
    "- merge the data sets and update the training set\n",
    "- train a network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the toolbox:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hTtJxcQ7ywnB"
   },
   "outputs": [],
   "source": [
    "import deeplabcut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set a variable to point to the project configuration file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# Note that parameters of this project can be seen at: *Reaching-Mackenzie-2018-08-30/config.yaml*\n",
    "from pathlib import Path\n",
    "\n",
    "#create a variable to set the config.yaml file path:\n",
    "path_config_file = os.path.join(os.getcwd(),'Reaching-Mackenzie-2018-08-30/config.yaml')\n",
    "print(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: When you use DeepLabCut on your own data, you will (1) create a project, (2) extract frames to label, and (3) label you data. \n",
    "**In this demo, this is all done for you!**\n",
    "The purpose of the demo to for you to get familiar with part of the workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the pre-labeled data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WOEHc0MeywnJ"
   },
   "outputs": [],
   "source": [
    "#let's load some demo data, and create a training set \n",
    "#(note, this function is not used when you create your own project):\n",
    "\n",
    "deeplabcut.load_demo_data(path_config_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ROlflqQLywnP"
   },
   "outputs": [],
   "source": [
    "#Perhaps plot the labels to see how the frames were annotated:\n",
    "\n",
    "deeplabcut.check_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h9H7eqDLywnV"
   },
   "source": [
    "## Start training of Feature Detectors\n",
    "This function trains the network for a specific shuffle of the training dataset. **The user can set various parameters in /Reaching-Mackenzie-2018-08-30/dlc-models/ReachingAug30-trainset95shuffle1/iteration-0/train/pose_cfg.yaml.**\n",
    "\n",
    "Training can be stopped at any time. Note that the weights are only stored every 'save_iters' steps. For this demo the it is advisable to store & display the progress very often (i.e. display every 20, save every 100). In practice this is inefficient (in reality, you will train until ~200K, so we save every 50K).\n",
    "\n",
    "**We recommend just training for 10-20 min, as you aren't running this demo to use DLC, just to work through the steps. In total, this demo should take you LESS THAN 1 HOUR!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jg96O2acywnW",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "deeplabcut.train_network(path_config_file, shuffle=1, saveiters=300, displayiters=10)\n",
    "#notice the variables \"saveiters\" and \"dsiplayiters\" that can be set in the function\n",
    "\n",
    "#you just need to run this until you get at least 1 snapshot, which is set by: \"save_iters\" \n",
    "#(so in this case you could stop after 500!) How do I stop? Click the STOP button!\n",
    "# To train until ~2,000 iterations on a CPU should be ~30 min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note, that if it reaches the end (defualt 1M) or you stop it (by \"stop\" or by CTRL+C), \n",
    "you will see an keyboard interrupt \"error\", but it is not a real error, i.e. you can ignore this.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CCzxVT_gywnc"
   },
   "source": [
    "## Evaluate the trained network\n",
    "\n",
    "This function evaluates a trained model for a specific shuffle/shuffles at a particular training state (snapshot) or on all the states. The network is evaluated on the data set (images) and stores the results as .csv file in a subdirectory under **evaluation-results**.\n",
    "\n",
    "You can change various parameters in the ```config.yaml``` file of this project. For the evaluation one can change pcutoff. This cutoff also influences how likely estimated postions need to be so that they are shown in the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kuprPKDdywne",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "deeplabcut.evaluate_network(path_config_file,plotting=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE: depending on your set up sometimes you get some \"matplotlib errors, but these are not important**\n",
    "\n",
    "Now you can go check out the images. Given the limted data input and it took ~20 mins to test this out, it is not meant to track well, so don't be alarmed. This is just to get you familiar with the workflow... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XeqYWGaXywnj"
   },
   "source": [
    "## Analyzing videos\n",
    "This function extracts the pose based on a trained network from videos. The user can choose the trained network - by default the most recent snapshot is used to analyse the videos. However, the user can also specify the snapshot index for the variable **snapshotindex** in the **config.yaml** file).\n",
    "\n",
    "The results are stored in hd5 file in the same directory, where the video resides. The pose array (pose vs. frame index) can also be exported as csv file (set flag to...). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Vv9iHHLlywnl"
   },
   "outputs": [],
   "source": [
    "# Set the video path:\n",
    "#The video can be the one you trained with and new videos that look similar, i.e. same experiments, etc.\n",
    "# You can add individual videos, OR just a folder - it will skip videos that are already analyzed once.\n",
    "\n",
    "#i.e you can run 'reachingvideo1' and/or 'MovieS2_Perturbation_noLaser_compressed'\n",
    "\n",
    "videofile_path = os.path.join(os.getcwd(),'Reaching-Mackenzie-2018-08-30/videos/reachingvideo1.avi')                   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CFbPPD4hywnq",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(\"Start Analyzing the video!\")\n",
    "deeplabcut.analyze_videos(path_config_file,[videofile_path])\n",
    "# this video takes ~ 8 min to analyze with a CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*NOTE: Yes, this is slow on a CPU (a GPU is MUCH faster)... see https://www.biorxiv.org/content/early/2018/10/30/457242 if you are interested!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QQ3T3oykywnw"
   },
   "source": [
    "## Create labeled video\n",
    "\n",
    "This function is for the visualization purpose and can be used to create a video in .mp4 format with the predicted labels. This video is saved in the same directory, where the (unlabeled) video resides. \n",
    "\n",
    "Various parameters can be set with regard to the colormap and the dotsize (matplotlib is used in the backend). See the config.yaml file for how to set these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GhI9KLs4ywn0",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video(path_config_file,[videofile_path], draw_skeleton=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IErvm1K5ywn5"
   },
   "source": [
    "## Plot the trajectories of the analyzed videos\n",
    "This function plots the trajectories of all the body parts across the entire video. Each body part is identified by a unique color. The underlying functions can easily be customized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mP2useJgywn7",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "deeplabcut.plot_trajectories(path_config_file,[videofile_path],showfigures=True)\n",
    "\n",
    "#These plots can are interactive and can be customized (see https://matplotlib.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a5nOaWzXywoB"
   },
   "source": [
    "## Extract outlier frames, where the predictions are off.\n",
    "\n",
    "This is optional step allows to add more training data when the evaluation results are poor. In such a case, the user can use the following function to extract frames where the labels are incorrectly predicted. Make sure to provide the correct value of the \"iterations\" as it will be used to create the unique directory where the extracted frames will be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note, if you have questions on parameters, remember \"?\" gives you answers:\n",
    "deeplabcut.extract_outlier_frames?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RJGiDKuUywoC",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "deeplabcut.extract_outlier_frames(path_config_file,videofile_path,outlieralgorithm='uncertain',p_bound=.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jHjpscPcywoG"
   },
   "source": [
    "The user can run this iteratively, and (even) extract additional frames from the same video."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uaNUm3NSywoH"
   },
   "source": [
    "## Manually correct labels\n",
    "\n",
    "This step allows the user to correct the labels in the extracted frames. Navigate to the folder with the videos and use the GUI as described in the protocol to update the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OJDvJMcrywoI",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#GUI pops up! \n",
    "#sometimes you need to restart the kernel for the GUI to launch.\n",
    "%gui wx\n",
    "deeplabcut.refine_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mcuqoeRbywoL"
   },
   "outputs": [],
   "source": [
    "# Now merge datasets (once you refined all frames)\n",
    "deeplabcut.merge_datasets(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WRB9KgGsywoP"
   },
   "source": [
    "## Create a new iteration of training dataset, check it and train...\n",
    "\n",
    "Following the refine labels, append these frames to the original dataset to create a new iteration of training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y7efellnywoT"
   },
   "outputs": [],
   "source": [
    "#Perhaps plot the labels to see how how all the frames are annotated (including the refined ones)\n",
    "deeplabcut.check_labels(path_config_file)\n",
    "# if they are off, you can load them in the labeling_gui to adjust!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NGHghXfdywoQ"
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_training_dataset(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8fhL6nG2ywoW"
   },
   "source": [
    "Now one can train the network again... (with the expanded data set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cAUxg5sgywoX"
   },
   "outputs": [],
   "source": [
    "deeplabcut.train_network(path_config_file)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Demo-labeledexample-MouseReaching.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [conda env:DLC2]",
   "language": "python",
   "name": "conda-env-DLC2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}