This repository provides the official implementation for PANet in the following paper. [arxiv paper]
- easydict
- numba
- numpy
- pyyaml
- python=3.7
- scikit-learn
- scipy
- spconv=1.1
- tensorboard=2.7.0
- torch=1.6
- torchvision=0.7.0
- torch-cluster=1.6.0
- torch-scatter=2.0.8
- tqdm
Please download the SemanticKITTI dataset to the folder data and the structure of the folder should look like:
./
├──
├── ...
└── data/
├──sequences
├── 00/
│ ├── velodyne/
| | ├── 000000.bin
| | ├── 000001.bin
| | └── ...
│ └── labels/
| ├── 000000.label
| ├── 000001.label
| └── ...
├── 08/ # for validation
├── 11/ # 11-21 for testing
└── 21/
└── ...
The training pipeline of our PANet consists of two steps: 1) semantic segmentation training following GASN; 2) instance aggregation training. The first step give us the semantic backbone. Then our learning-free sparse proposal module (SIP) can be directly used to group instances. The last step gives our instance aggregation module (IA), which impoves the segmentation performance of large objects. We provide the corresponding pretrained model of each step. The training and inferencing details are further explained in this section.
All the slurm and pytorch version of scripts are tested and should work well.
If you wish to use our pretrained models, remember to create a new folder weights and put all the downloaded models there.
| Step | Module | PQ | PQ (Truck) | mIoU | Download Link |
|---|---|---|---|---|---|
| 1 | +SIP | 61.5 | 61.2 | 68.1 | kitti_backbone_v2.pth |
| 2 | +SIP, +IA | 61.7 | 64.3 | 68.1 | kitti_panet.pth |
We provide inferencing scripts for our PANet.
Our semantic backbone combines the sparse 3D CNN and 2D CNN. You can train the semantic backbone using the code released by GASN.
Before using our SIP module for inference, please make sure you have downloaded the pretrained semantic backbone model (of step 1) or put the models trained by yourself (in step 1) to ./weights. For validaton, you can use the following command. Feel free to play around with different parameter settings in ./cfgs/sip.yaml.
$ python cfg_train.py --cfg cfgs/sip.yaml --onlyval
Before inferencing with our full model PANet, remember to download the pretrained model (of step 2) or put the model trained by yourself (in step 2) to ./weights and make sure you pass the right path to --pretrained_ckpt option. For validation, you can use the following command.
$ python cfg_train.py --cfg cfgs/panet.yaml --onlyval --pretrained_ckpt </path/to/kitti_panet.pth>
For testing on SemanticKITTI test set, we used the ones that performed best on the validation set during training in step 2. you can use the following command to obtain the predictions.
$ python cfg_train.py --cfg cfgs/panet.yaml --onlytest --pretrained_ckpt </path/to/kitti_panet.pth>
You can use --log_dir and --tag to pass the path for desired output folder.
The training code for the semantic backbone has not been incorporated yet. For now, please train the semantic backbone using the code released by GASN or download the step 1 pretrained model using the above link.
The training scripts of step 2 could be found in ./scripts/*_train.sh. Before using the training scripts of this part, please download the pretrained model (of step 1) to folder ./weights or put the model trained (in step 1) to ./weights. You could experiment with different parameter settings in ./cfgs/panet.yaml. For training with a single GPU, you can also use the following command.
$ python cfg_train.py --cfg cfgs/panet.yaml --log_dir </path/to/logs> --tag <panet>
Distributed under the MIT License. See LICENSE for more information.
In our implementation, we refer to the following open-source databases: