A Multi-Modal Fusion-Based 3D Multi-Object Tracking Framework With Joint Detection

This is the offical implementation of the paper "A Multi-Modal Fusion-Based 3D Multi-Object Tracking Framework With Joint Detection "

Firstly, a new multi-object tracking framework is proposed in this paper based on multi-modal fusion. By integrating object detection and multi-object tracking into the same model, this framework avoids the complex data association process in the classical TBD paradigm, and requires no additional training. Secondly, confidence of historical trajectory regression is explored, possible states of a trajectory in the current frame (weak object or strong object) are analyzed and a confidence fusion module is designed to guide non-maximum suppression of trajectory and detection for ordered association. Finally, extensive experiments are conducted on the KITTI and Waymo datasets. The results show that the proposed method can achieve robust tracking by using only two modal detectors and it is more accurate than many of the latest TBD paradigm-based multi-modal tracking methods.

Update

• 2024-12-13: This paper has been accepted as a journal paper in Robotics and Automation Letters (RA-L).

Overview

• Update log
• Framework
• Model Zoo
• Getting Started
• Acknowledgementn
• Citation

Update Log

• 2023/4/3 Initial update
• 2023/4/19 Publication of paper and code

Framework

The multi-modal fusion-based multi-object tracking method proposed in this paper is structured as shown in Fig. 2. The structure consists of three modules: data input module, detection and trajectory regression module, and trajectory management module. The detection and trajectory regression module includes 2D/3D detectors, state prediction, and fusion confidence mechanism.

Model Zoo

KITTI Dataset

• The results are obtained by combining different 3D detectors with 2D detector Faster-RCNN for the Car category on the KITTI val set.

• Currently, this repo supports most of the two -stage 3D detectors, such as TED, CasA, PV-RCNN, Voxel-RCNN, PV-RCNN++, etc.

Detectors	HOTA(Car)	MOTA(Car)	IDs(Car)	weights
CasA	76.37	80.40	89.42	google / baiduyun(6666)
OpenPCDet/Voxel-RCNN	77.52	82.11	90.75	google / baiduyun(6666)
OpenPCDet/PV-RCNN	75.88	76.11	88.58	google / baiduyun(6666)
OpenPCDet/Point-RCNN	54.25	48.34	66.22	google / baiduyun(6666)
Faster RCNN	----	----	---	google / baiduyun(6666)

Waymo Dataset

Detectors	Set	Vehicle_L2	Pedestrian_L2	Cyclist_L2
OpenPCDet/PV-RCNN + Faster RCNN	val	53.02	47.45	40.45
OpenPCDet/PV-RCNN + Faster RCNN	test	60.48	50.08	56.55

We could not provide the above pretrained models due to Waymo Dataset License Agreement, but you could easily achieve similar performance by training with the default configs of the detector(such as: casa, pcdet).

Getting Started

Dependency

• Ubuntu 20.04
• NVIDIA CUDA 11.3
• Python 3.9.16
• PyTorch 1.8.1
• Spconv 2.3.3 # pip install spconv-cu113
• 2x RTX4090 GPUs

Prepare dataset

Tips: It is recommended to separate the "data" files from the YONTD-MOT source code files, so as to avoid lagging when opening PyCharm.

KITTI Dataset

To run this code, you should organize kitti tracking dataset as below:

# Kitti Tracking Dataset 

├── data
│   ├── kitti
│   │   │── tracking
│   │   │   │──testing
|   │   │   │   ├──calib
|   │   │   │   |    ├──0000.txt
|   │   │   │   |    ├──....txt
|   │   │   │   |    └──0028.txt
|   │   │   │   ├──image_02
|   │   │   │   |    ├──0000
|   │   │   │   |    ├──....
|   │   │   │   |    └──0028
|   │   │   │   ├──oxts
|   │   │   │   |    ├──0000.txt
|   │   │   │   |    ├──....
|   │   │   │   |    └──0028.txt
|   │   │   │   ├──label_02
|   │   │   │   |    ├──0000.txt
|   │   │   │   |    ├──....txt
|   │   │   │   |    └──0028.txt
|   │   │   │   ├──velodyne
|   │   │   │   |    ├──0000
|   │   │   │   |    ├──....
|   │   │   │   |    └──0028  
│   │   │   │──training  # the structure is same as testing set
|   │   │   │   ├──calib
|   │   │   │   ├──image_02
|   │   │   │   ├──pose
|   │   │   │   ├──label_02
|   │   │   │   └──velodyne 

Waymo Dataset

Step 1, after downloading the data from the Waymo official website and extracting it, it should be in the following format.

├── data
│   ├── waymo
│   │   │── testing
│   │   │   │── segment-xxxxxxxx.tfrecord
│   │   │   │── ...
│   │   │   │── There are a total of 150 files.
│   │   │── training
│   │   │   │── segment-xxxxxxxx.tfrecord
│   │   │   │── ...
│   │   │   │── There are a total of 798 files.
│   │   │── validation
│   │   │   │── segment-xxxxxxxx.tfrecord
│   │   │   │── ...
│   │   │   │── There are a total of 202 files.

Step 2, create a folder named raw_data, and then move all segment-xxxxxxxx.tfrecord files under the "testing," "training," and "validation" folders to the raw_data folder. Then, your file directory should look like the following.

├── data
│   ├── waymo
│   │   │── testing
│   │   │── training
│   │   │── validation
│   │   │── raw_data
│   │   │   │── segment-xxxxxxxx.tfrecord
│   │   │   │── ...
│   │   │   │── There are a total of 1150 files(include testing, training and validation).

Step 3, Run following command to creat dataset infos:

python3 -m dataset_utils.waymo.waymo_dataset 

Finally, your file directory will become as follows. You need to download the ImageSets file from google or baiduyun(6666).

├── data
│   ├── waymo
│   │   │── testing
│   │   │── training
│   │   │── validation
│   │   │── ImageSets
│   │   │── raw_data
│   │   │   │── segment-xxxxxxxx.tfrecord
|   |   |   |── ...
|   |   |── waymo_processed_data_train_val_test
│   │   │   │── segment-xxxxxxxx/
│   │   │   │   │── FRONT
│   │   │   │      │── 0000.jpg
│   │   │   │      │── ...
│   │   │   │   │── FRONT_LEFT
│   │   │   │   │── FRONT_RIGHT
│   │   │   │   │── SIDE_LEFT
│   │   │   │   │── SIDE_RIGHT
│   │   │   │   │── 0000.npy
│   │   │   │   │── ...
|   |   |   |── ...
│   │   │   │── There are a total of 1150 files(include testing, training and validation).
│   │   │── pcdet_waymo_track_dbinfos_train_cp.pkl
│   │   │── waymo_infos_test.pkl
│   │   │── waymo_infos_train.pkl
│   │   │── waymo_infos_val.pkl

Installation

git clone https://github.com/hailanyi/CasA.git
cd YONTDMOT
pip install -r requirements.txt

# If you Install corresponding dependencies based on different detectors.
# If you want to use the detector of OpenPCDet, please run following command:
cd detector/pcdet
python3 setup.py develop

# If you want to use the detector of CasA, please run following command:
cd detector/CasA
python3 setup.py develop

# If you want to use the detector of TED, please run following command:
cd detector/TED
python3 setup.py develop

Quick start

KITTI Dataset

• Please modify the dataset path(data_path) in the config/kitti.yaml file to your own path.
• Then run python3 kitti_main.py
• The results are automatically saved to output/training/, and evaluated by HOTA metrics.

Waymo Dataset

• Please modify the dataset path(DATA_PATH) in the config/waymo.yaml file to your own path.
• Then run python3 waymo_main.py
• The results are automatically saved to output/result_dets.pkl.

Acknowledgementn

This repo is developed from OpenPCDet and CasA. A portion code is borrowed from CasTrack-waymo. Many thanks to their wonderful work!

Citation

If you find this work useful, please consider to cite our paper:

@ARTICLE{10777493,
  author={Wang, Xiyang and Fu, Chunyun and He, Jiawei and Huang, Mingguang and Meng, Ting and Zhang, Siyu and Zhou, Hangning and Xu, Ziyao and Zhang, Chi},
  journal={IEEE Robotics and Automation Letters}, 
  title={A Multi-Modal Fusion-Based 3D Multi-Object Tracking Framework With Joint Detection}, 
  year={2025},
  volume={10},
  number={1},
  pages={532-539},
  keywords={Three-dimensional displays;Trajectory;Detectors;Object detection;Proposals;Feature extraction;Solid modeling;Data models;Point cloud compression;Kalman filters;3D MOT;Camera and LiDAR fusion;object detection and tracking},
  doi={10.1109/LRA.2024.3511438}}