Master thesis

This thesis work was done as a part of research at Unmanned System Research Group (USRG) led by Professor David Hyunchul Shim, KAIST, South Korea.

General information

Important materials

1. Presentation
2. Dissertation

Title

The development of the scale-aware monocular depth estimation aided monocular visual SLAM system for real-time robot navigation

Motivation

Human pilots could fly any drone by relying only on the monocular visual signal in the goggle. In this work, we want to use high performance computer to replace the human brain to navigate the drone by relying only on the monocular camera sensor.

Objective

To enable the mobile robot to autonomously navigate in an unknown environment using only a monocular camera as its sensor

Approaches

1. RGB-Deep D visual SLAM: Augmented the monocular depth estimation (Packnet-sfm) to provide the pseudo depth information to the monocular visual SLAM (ORB_SLAM2) to solves its inherent problems: scale-ambiguity, tracking robustness, and map initialization delay (See presentation file page 5).

2. Navigation stack: Following the see-think-act scheme for autonomous mobile robot, we integrated visual SLAM (see, RGB-Deep D SLAM, localization/ mapping), local path planner (think, Fast-Planner, planning/ obstacle avoidance), and waypoint tracking controller (act, PD controller, control).

Module	Employed system
Visual SLAM	ORB_SLAM2
Monocular Depth Estimation	packnet-sfm
Local path planner	Fast-Planner
Controller	PD controller

Dependencies

We make 3 seperated workspaces. The installation instruction for each package is in the correspoding link.

1. catkin_ws

catkin_ws/src
├── [drone_controller](https://github.com/surfii3z/drone_controller/tree/thesis)
├── [Fast-Planner](https://github.com/surfii3z/Fast-Planner/tree/thesis)
├── [image_proc](https://github.com/surfii3z/image_proc/tree/thesis)
├── [image_undistort](https://github.com/surfii3z/image_undistort/tree/thesis)
└── [tello_driver](https://github.com/surfii3z/tello_driver/tree/thesis)

2. packnet_ros_ws

packnet_ros_ws/src/
├── [packnet_sfm_ros](https://github.com/surfii3z/packnet_sfm_ros)
└── vision_opencv

3. ORB_SLAM2

[ORB_SLAM2](https://github.com/surfii3z/ORB_SLAM2/tree/thesis)

How to run

I experimented this system on DGX-Station (V100), tested with Ubuntu 18.04 LTS, python 3.6.9, TensorRT-7.1.3.4, PyTorch 1.4.0, Cuda 11.0 and CuDNN 8.0.5. We need to use 8 terminals to run

# -------------------------
# 00_roscore
roscore

# -------------------------
# 01_depth_estimation_module
# pwd: /path/to/packnet_ros_ws
source install/setup.bash --extend
rosrun packnet_sfm_ros trt_packnet_node

# -------------------------
# 02_tello_driver
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch tello_driver thesis_tello_node.launch

# -------------------------
# 03_visual_SLAM
# pwd: /path/to/ORB_SLAM2/ros
source devel/setup.bash
roslaunch orb_slam2 rgbd_tello_crop.launch

# -------------------------
# 04_path_planner_visualization
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch plan_manage rviz.launch

# -------------------------
# 05_path_planner
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch plan_manage tello_kino_replan.launch


# -------------------------
# 06_drone_controller
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch drone_controller waypoint_controller.launch

# -------------------------
# 07_mission
# pwd: /path/to/catkin_ws
source devel/setup.bash
rosrun drone_controller thesis_mission.py

NOTE: The monocular depth estimation network must be trained based on the application.

Results

Please see this playlist on Youtube