Integrate Multi sensor fusion model 'BEVFusion' into Autoware

[cyn-liu]

Description

BEVFusion (paper) (code), an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information.

Purpose

In the existing camera_lidar perception mode of Autoware, lidar and cameras use multiple individual models for object detection, and then the detection results of the two are fused through a rule based fusion module.

In contrast, BEVFusion unifies multi-modal features in the shared bird's-eye view (BEV) perspectives for 3D object detection. In another word, BEVFusion dose all the staff above in one single model.

Further more, this is not only simplify the perception pipeline of multi-sensor fusion mode, but also continuously improve the accuracy of 3D object detection through iterative model and algorithms.

Based on above fact, we plan to integrating BEVFusion into Autoware for 3D object detection based on multi-view images and LiDAR.

Possible approaches

We plan to integrate BEVFusion algorithm based on this repo TensorRT deployment solution of BEVFusion provide by NVIDIA.

To Do List

• Wrap the original TensorRT deployment solution of BEVFusion code as a vendor ROS2 package (library package) and maintain it outside Autoware with different CIs settings and code standards.
• Create autoware_tensorrt_bevfusion package in Autoware.universe and call the bevfusion_vendor library for 3D object detection inference.
• Retrain BEVFusion model with TIER4 dataset to verify its generalization performance, upload the retrain tutorial to AWF official document site at the same time

Others

The license for BEVFusion's training code is Apache-2.0, which is compatible with autoware.

What worry me is that the license of BEVFusion's deployment code is MIT, we plan to integrate nvidia's "Lidar_AI_Solution" into autoware's repo as a separate vendor package following MIT license. Is this acceptable to AWF? Will that bring legal risk to AWF? @kminoda

[knzo25]

@cyn-liu
As a hobby, as I already performed the integration of bevfusion in autoware. In the past we had also tried to do it, but got stopped due to how NVIDIA released their bevfusion implementation (shared library and copyright).

https://github.com/knzo25/bevfusion_ros2

My integration directly uses weights trained using mmdet3d or similar, and uses either mmcv's spconv or traveller59's one as TensorRT plugins. The before mentioned repo contains only mmcv's implementation since I am still cleaning travellers59's.

Regarding licenses, both are Apache, so there would be no problem:
https://github.com/open-mmlab/mmcv
https://github.com/traveller59/spconv

As to why I haven't sent a PR yet, I am waiting for autoware's TensorRT update, which should happen soon.

[liuXinGangChina]

Hi kenzo-san， thanks for your quick reply @knzo25

1. considerring tire4 side is already working on retrain， there are no reason that we do that again。After all ， our engineer have abundant experience on bev-series model， feel free to contact us if your side need help with retrain stuff
2. we will waiting for your pr to finish， by the meantime， do you think it is reasonable for me to implement a “nvidia optimized bev-fusion autoware version” as a alternative for those autoware developers who pursue extremely fast inference speed ？At last ， the developer can choose which one to use freely according to their deployment environment（commercial or academic research）in terms of license issue

happy jacking！

心刚

[knzo25]

Regarding 2), I don't really have a say in that, maybe autoware's architects have a more defined opinion. However, as explained previously, I don't feel particularly happy linking against something I can neither control (fix) nor know its actual implementation (black box).

I will post any updates from our side to this issue

[liuXinGangChina]

Got you point ， kenzo-san

BY “ autoware's architects ” ， do you mean mitsudomi-san？

happy hacking！

心刚

[knzo25]

Yes, I meant people like @mitsudome-r and @xmfcx !

[liuXinGangChina]

Thanks kenzo-san， i will discuss my proposal with them and get back to you soon

Have a nice day！

心刚