feat: always separate lidar preprocessing from pointcloud_container

[kminoda]

Description

Current Autoware has use_pointcloud_container flag, which is to define whether to gather necessary nodes into one composable node, namely /pointcloud_container. Currently, the flag is also applied to LiDAR preprocessing nodes, gathering all of those from multiple LiDARs into one composable node, which makes Autoware vulnerable to process failure in those nodes.

In order to improve redundancy towards the node failure in LiDAR preprocessing, we would like to separate LiDAR preprocessing nodes from /pointcloud_container regardless of the use_pointcloud_container flag. Specifically,

• Changes to sensing: Remove use_pointcloud_container from LiDAR preprocessing node launchers
• Changes to perception: Clarify the difference between pointcloud_container_name and individual_container_name. Former one is used when use_pointcloud_container is true.
• Changes to localization: We need some modification in localization nodes as well since it will no longer be attached to /pointcloud_container but rather to LiDAR container.

MUST BE MERGED WITH

• feat: always separate lidar preprocessing from pointcloud_container autoware_launch#796
• feat: always separate lidar preprocessing from pointcloud_container sample_sensor_kit_launch#85
• feat: always separate lidar preprocessing from pointcloud_container tier4/aip_launcher#197
• feat: always separate lidar preprocessing from pointcloud_container tier4/awsim_sensor_kit_launch#13

Note that the future work may include

• Removing use_pointcloud_container flag: Current Autoware effectively assumes that use_pointcloud_container is true due to the performance capability. It is reasonable to remove the flag at least for now to improve the readability of launch files.
• Reorganizing glog: With this modification, glog_component will no longer be applied to LiDAR preprocessing nodes. I will create another PR to organize glog so that it will be enabled for those nodes as well.

Related links

• feat: always separate lidar preprocessing from pointcloud_container autoware_launch#796

• feat: always separate lidar preprocessing from pointcloud_container sample_sensor_kit_launch#85

• feat: always separate lidar preprocessing from pointcloud_container tier4/aip_launcher#197

• feat: always separate lidar preprocessing from pointcloud_container tier4/awsim_sensor_kit_launch#13

• INTERNAL SLACK LINK FOR TIER IV

Tests performed

Performance evaluation (For XX1)

Since this PR will separate concatenation node from LiDAR preprocessing nodes, I have tested the PR with JPN TAXI to see if the performance regression is negligible or not.
The results are written here: TIER IV INTERNAL LINK. We observed a slight delay in the deriving topics from concatenated/pointcloud, which is around ~10ms in average.

Logging simulator w/ awf/autoware (sample_sensor_kit)

use_pointcloud_container = true

• Confirmed that all the nodes are in the intended containers: use_pointcloud_container_true.txt
• Confirmed that localization and perception works

use_pointcloud_container = false

• Confirmed that all the nodes are in the intended containers: use_pointcloud_container_false.txt
• Confirmed that localization and perception works

Logging simulator w/ tier4/pilot-auto (aip_xx1)

use_pointcloud_container = true

• Confirmed that all the nodes are in the intended containers: use_pointcloud_container_true.txt
• Confirmed that localization and perception works

use_pointcloud_container = false

• Confirmed that all the nodes are in the intended containers: use_pointcloud_container_false.txt
• Confirmed that localization and perception works

Notes for reviewers

None

Interface changes

This PR includes name change in composable node.
No topics nor node names are changed.

Effects on system behavior

Increase in CPU usage is expected since it will increase the inter-process communication (mostly in a single computer). However, we estimate this risk small enough given the above-mentioned delay estimation results.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.
• The PR has been properly tested.
• The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.
• The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

[yukkysaito]

LGTM

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a3f90f0) 14.63% compared to head (7bc683b) 14.63%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6091   +/-   ##
=======================================
  Coverage   14.63%   14.63%           
=======================================
  Files        1858     1858           
  Lines      126848   126848           
  Branches    37173    37173           
=======================================
  Hits        18570    18570           
  Misses      87390    87390           
  Partials    20888    20888           

Flag	Coverage Δ		*Carryforward flag
differential	27.56% <ø> (?)
total	14.63% <ø> (ø)		Carriedforward from a3f90f0

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[miursh]

When use_pointcloud_container is set to false, each node (ground_segmentation, concatenate, compare_map_filter, ) is placed in a separate container. Is this intended?"

$ ros2 component list
/perception/occupancy_grid_map/occupancy_grid_map_container
  1  /perception/occupancy_grid_map/occupancy_grid_map_node
/sensing/lidar/rear/pointcloud_preprocessor/pointcloud_container
  1  /sensing/lidar/rear/velodyne_driver_ros_wrapper_node
  2  /sensing/lidar/rear/crop_box_filter_self
  3  /sensing/lidar/rear/crop_box_filter_mirror
  4  /sensing/lidar/rear/distortion_corrector_node
  5  /sensing/lidar/rear/ring_outlier_filter
/perception/obstacle_segmentation/ground_segmentation_container
  1  /perception/obstacle_segmentation/crop_box_filter
  2  /perception/obstacle_segmentation/common_ground_filter
  3  /perception/obstacle_segmentation/occupancy_grid_map_outlier_filter
/sensing/lidar/top/pointcloud_preprocessor/pointcloud_container
  1  /localization/util/crop_box_filter_measurement_range
  2  /sensing/lidar/top/velodyne_driver_ros_wrapper_node
  3  /localization/util/voxel_grid_downsample_filter
  4  /sensing/lidar/top/crop_box_filter_self
  5  /localization/util/random_downsample_filter
  6  /sensing/lidar/top/crop_box_filter_mirror
  7  /sensing/lidar/top/distortion_corrector_node
  8  /sensing/lidar/top/ring_outlier_filter
/perception/object_recognition/detection/pointcloud_map_filter_container
  1  /perception/object_recognition/detection/voxel_grid_downsample_filter
  2  /perception/object_recognition/detection/voxel_based_compare_map_filter
/perception/object_recognition/detection/clustering/euclidean_cluster_container
  1  /perception/object_recognition/detection/clustering/low_height_crop_box_filter
  2  /perception/object_recognition/detection/clustering/euclidean_cluster
/map/map_container
  1  /map/lanelet2_map_loader
  2  /map/lanelet2_map_visualization
  3  /map/pointcloud_map_loader
  4  /map/vector_map_tf_generator
/sensing/lidar/concatenate_container
  1  /sensing/lidar/concatenate_data

[kminoda]

@miursh Yes, that is intended. From the performance perspective, it is better to gather as much as possible in one container even when use_pointcloud_container is false, but I didn't made that modification from the following reasons:

• Doing the modification would make this PR larger (e.g. launching /perception_container when use_pointcloud_container=false)
• use_pointcloud_container will be true in most of the cases, and thus the above implementation would make little difference for Autoware users

[KYabuuchi]

tier4_localization_launch looks good to me 🙆‍♂️

[mebasoglu]

Hello, could you share some metrics about this? How much performance do we lose with separating nodes? Maybe this PR could be merged after #6056 so that we can see delays.

[kminoda]

@mebasoglu Thank you for the comment! We indeed did a performance evaluation on our real vehicle, mostly with the same approach in the above PR (we've also got an advice from @.xmfcx about the accumulated time based evaluation last week). It's in description of this PR. I will just copy-and-paste it here as well for your convenience.

In short, we observed a negligible increase in topic delay, possibly because this PR only separate one single node (=concatenation node), so we estimate the degradation due to this PR fairly limited. We measured it in two approaches: ros2 topic delay and the accumulated time based approach (as in #6056). So I believe we can just merge this PR without #6056, although this is not a hard requirement and thus don't mind waiting for #6056 if you still see a concern in this PR.

BTW the difference between ros2 topic delay approach and the accumulated time based approach was very small in our case.

---

Performance evaluation (For XX1)

Since this PR will separate concatenation node from LiDAR preprocessing nodes, I have tested the PR with JPN TAXI to see if the performance regression is negligible or not.
The results are written here: TIER IV INTERNAL LINK. We observed a slight delay in the deriving topics from concatenated/pointcloud, which is around ~10ms in average.

[miursh]

LGTM

[kminoda]

Memo: Further investigation has been done
INTERNAL LINK FOR JIRA TICKET