examples/Demo_3D_DeepLabCut.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3D DeepLabCut Toolbox\n",
    "https://github.com/AlexEMG/DeepLabCut\n",
    "\n",
    "This notebook will highlight the functionality of the newly released 3D project option (as of **2.0.7+**).\n",
    "\n",
    "We recommend you look through these commands, and then you can easily modify this notebook for your own projects.\n",
    "\n",
    "**In summary, the new functionality will:**\n",
    "- allow you to calibrate your cameras using checkerboard images\n",
    "- compute and fix camera distortions\n",
    "- convert your 2D DeepLabCut tracking to 3D\n",
    "- allow for plotting in 3D space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a New 3D Project:\n",
    "\n",
    "You will run this first function **one** time per project; a project is defined as a given set of cameras and calibration images. You can always analyze new videos within this project.\n",
    "\n",
    "The function create_new_project_3d creates a new project directory specifically for converting the 2D pose to 3D pose, required subdirectories, and a basic 3D project configuration file. Each project is identified by the name of the project (e.g. Task1), name of the experimenter (e.g. YourName), as well as the date at creation.\n",
    "\n",
    "Thus, this function requires the user to input the enter the name of the project, the name of the experimenter and number of cameras to be used. Currently, DeepLabCut supports triangulation using 2 cameras, but will expand to more than 2 cameras in a future version.\n",
    "\n",
    "Optional arguments specify the working directory, i.e. where the project directory will be created. If the optional argument working_directory is unspecified, the project directory is created in the current working directory. Please note that the Full path of the 3D project configuration file will be referenced as ``config_path3d``.\n",
    "\n",
    "NOTE: Be sure you are inside your DLC anaconda environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deeplabcut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Setup your project variables:\n",
    "YourName = 'teamDLC'\n",
    "YourExperimentName = 'testing'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created \"/home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples/testing-teamDLC-2019-10-29-3d/camera_matrix\"\n",
      "Created \"/home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples/testing-teamDLC-2019-10-29-3d/calibration_images\"\n",
      "Created \"/home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples/testing-teamDLC-2019-10-29-3d/undistortion\"\n",
      "Created \"/home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples/testing-teamDLC-2019-10-29-3d/corners\"\n",
      "Generated \"/home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples/testing-teamDLC-2019-10-29-3d/config.yaml\"\n",
      "\n",
      "A new project with name testing-teamDLC-2019-10-29-3d is created at /home/mackenzie/Desktop/DeepLabCut2.0-2.2/examples and a configurable file (config.yaml) is stored there. If you have not calibrated the cameras, then use the function 'calibrate_camera' to start calibrating the camera otherwise use the function ``triangulate`` to triangulate the dataframe\n"
     ]
    }
   ],
   "source": [
    "config_path = deeplabcut.create_new_project_3d(YourExperimentName,YourName,num_cameras=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TIP 1:** you can also pass ``working_directory='Full path of the working directory'`` if you want to place this folder somewhere beside the current directory you are working in.\n",
    "\n",
    "**TIP 2:** you can also place config_path3d in front of deeplabcut.create_new_project_3d to create a variable that holds the path to the config.yaml file, i.e. ``config_path3d=deeplabcut.create_new_project_3d(...``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#If you're loading an already created project, just set the 3D Project config_path variable:\n",
    "#import os\n",
    "#from pathlib import Path\n",
    "#config_path3d = os.path.join(os.getcwd(),'testing3D-DeepLabCutTeam-2019-06-05-3d/config.yaml')\n",
    "#print(config_path3d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calibrate your cameras! \n",
    "\n",
    "(**CRITICAL!**) You must take images of a checkerboard to calibrate your images. Here are example boards you could print and use (mount them on a flat, hard surface!): https://markhedleyjones.com/projects/calibration-checkerboard-collection. \n",
    "- You must save the image pairs as **.jpg** files. \n",
    "- They should be named with the camera-# as the prefix, i.e. **camera-1-01.jpg** and **camera-2-01.jpg** for the first pair of images. \n",
    "- While taking the images:\n",
    "     - Keep the orientation of the chessboard same and do not rotate more than 30 degrees. Rotating the chessboard circularly will change the origin across the frames and may result in incorrect order of detected corners.\n",
    "\n",
    "     - Cover several distances, and within each distance, cover all parts of the image view (all corners and center).\n",
    "\n",
    "     - Use a checkerboard as big as possible, ideally with at least 8x6 squares.\n",
    "\n",
    "     - Aim for taking at least 70 pair of images as after corner detection, some of the images might need to be discarded due to either incorrect corner detection or incorrect order of detected corners.\n",
    "     \n",
    "#### DEMO images:\n",
    " \n",
    "Here, we used a standard set along with this notebook. These images are a part of the Camera Calibration ToolBox for Matlab; specifically example 5: http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html \n",
    "\n",
    "\n",
    "If you wish to run this DEMO notebook, download the files and place inside the **calibration_images** directory. (To note, pairs 1 and 6 are not detected correctly, so please delete these images!). \n",
    "\n",
    "\n",
    "When acquiring calibration images from your own setup, it is recommended to take multiple pairs of image (around 50-70 pairs!).\n",
    "\n",
    "The camera calibration is an **iterative process**, where the user needs to select a set of calibration images where the grid pattern is correctly detected. The function:``deeplabcut.calibrate_cameras(config_path)`` \n",
    "extracts the grid pattern from the calibration images and store them under the `corners` directory. The grid pattern could be 8x8 or 5x5 etc. We use a pattern of the 8x6 grid to find the internal corners of the checkerboard.\n",
    "\n",
    "In some cases, it may happen that the corners are not detected correctly or the order of corners detected in the camera-1 image and camera-2 image is incorrect. We need to remove these pair of images as they will reduce the calibration accuracy.\n",
    "\n",
    "**NEXT/if not done:** please place your (or demo) images into the **calibration_images** directory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edit the config.yaml file:\n",
    "- change the camera names; i.e. if you do \"\"cam1, cam2\"\", or \"camera-1, camera-2\", or \"left, right\", etc.\n",
    "- Note, that once this is set, you cannot edit these (they are used for other steps)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.calibrate_cameras(config_path3d, cbrow =9,cbcol =6,calibrate=False,alpha=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: you need to specify how many rows (``cbrow``) and columns (``cbcol``) your checkerboard has (i.e. the intersection points - see demo images if unclear). Also, first set the variable ``calibrate`` to **False**, so you can remove any faulty images. You need to  visually inspect the output to check for the detected corners and select those pair of images where the corners are correctly detected. \n",
    "\n",
    "Once all the set of images are selected (namely, delete from the folder any bad pairs!) where the corners and their orders are detected correctly, then the two cameras can be calibrated using:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**They will look like this (one example frame):**\n",
    "<p align=\"center\">\n",
    "<img src=\"https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1559776966423-RATM6ZQT8JXHYAN768F6/ke17ZwdGBToddI8pDm48kKmw982fUOZVIQXHUCR1F55Zw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZUJFbgE-7XRK3dMEBRBhUpw5XnxLBmEFHJGf_0qFdDpmIncOw4kq9OpCHNTYqzGO-E1YJr-Thht9Tdog4YtCwrE/right02_corner.jpg?format=500w\" width=\"30%\">\n",
    "</p> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.calibrate_cameras(config_path3d, cbrow = 9,cbcol = 6, calibrate=True, alpha=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check for Undistortion:\n",
    "\n",
    "In order to check how well the stereo calibration is, it is recommended to undistort the calibration images and the corner points using camera matrices and project these undistorted points on the undistorted images to check if they align correctly. This can be done in deeplabcut as:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "%matplotlib inline\n",
    "\n",
    "deeplabcut.check_undistortion(config_path3d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each calibration image is undistorted and saved under the directory ``undistortion``. A plot with a pair of undistorted camera images with its undistorted corner points overlaid is also stored. Please visually inspect this image. All the undistorted corner points from all the calibration images are triangulated and plotted for the user to visualize for any undistortion related errors. If they are not correct, go check and revise the calibration images (then repeat the calirbation and this step)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Triangulation --> converting your 2D to 3D! \n",
    "\n",
    "If there are no errors in the undistortion, then the pose from the 2 cameras can be triangulated to get the 3D DeepLabCut coordinates!\n",
    "\n",
    "(**CRITICAL!**) Name the video files in such a way that the file name contains the name of the cameras as specified in the ``config file``. e.g. if the cameras as named as ``camera-1`` and ``camera-2`` (or ``cam-1``, ``cam-2`` etc.) then the video filename must contain this naming, i.e. this could be named as ``rig-1-mouse-day1-camera-1.avi`` and ``rig-1-mouse-day1-camera-2.avi``. Notably, the videos do not need to be the same pixel size, but be sure they are similar in size to the calibration images (and they must be the same cameras used for calibration).\n",
    "\n",
    "##  (**CRITICAL!**) Edit the config.yaml file: \n",
    "You must also edit the **3D project config.yaml** file to denote which DeepLabCut projects have the information for the 2D views. \n",
    "\n",
    " - Of critical importance is that you need to input the **same** body part names as in the config.yaml file of the 2D project.\n",
    "- You must set the snapshot to use inside the 2D config file (default is -1, namely the last training snapnot of the network). \n",
    "- You need to set a \"scorer 3D\" name; this will point to the project file and be set in future 3D output file names.\n",
    "- You should define a \"skeleton\" here as well (note, this is not rigid, it just connects the points in the plotting step). Not every point needs to be \"skeletonized\", i.e. these points can be a subset of the full body parts list. The other points will just be plotted into the 3D space.\n",
    "\n",
    "\n",
    "\n",
    "**Next,** pass the ``config_path3d`` and now the ``video_path``, which is the path to the **folder** where all the videos from two cameras are stored. The triangulation can be done in deeplabcut by typing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Of course, this does not work on the demo calibration images, \n",
    "# but when you are ready for your own dataset, edit and then run the following!\n",
    "\n",
    "video_path = '/home/yourname/videoFolder'\n",
    "\n",
    "deeplabcut.triangulate(config_path3d,video_path, videotype='mp4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **triangulated file** is now saved under the same directory where the video files reside (or you passed a destination folder path)! These files can now be used for future analysis. This step can now be run at anytime as you collect new videos, and easily added to your automated analysis pipeline, i.e. such as using``deeplabcut.triangulate(config_path3d, video_path)`` instead of ``deeplabcut.analyze_videos``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize your 3D DeepLabCut Videos:\n",
    "\n",
    "In order to visualize the pose in 3D, the user can create a 3D video for certain frames (these are large files, so we advise just looking at a subset of frames). The user can specify the path of the triangulated file and specify the start and end frame indices to create a 3D labeled video. Note that the triangulated_file is the newly created file that ends with yourDLC_3D_scorername.h5. This can be done using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video_3d(config_path,['triangulated_file_folder'],start=50,end=250, trailpoints=3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:DLC2]",
   "language": "python",
   "name": "conda-env-DLC2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}