Compatibility tests - sensors data publishers

[WJaworskiRobotec]

Description

Preparation of second batch of tests related to publication of sensor data by the simulator. Important to have a definition of topics, message types in the configuration file to easily change the number of published lidars/cameras

Purpose

To evaluate the sensor data published by the simulators

Definition of Done

• PR created
• Tests verified in AWSIM and MORAI SIM

[SoohyeokPark-MORAI]

I am planning to start this Friday.
I made a sample configuration file that includes sensor-related publishers and publishers and subscribers just in case.
sample configuration file link

[SoohyeokPark-MORAI]

camera, gps (gnss pose, pose with covariance), imu are under tests with MORAI SIM.

[WJaworskiRobotec]

@SoohyeokPark-MORAI Is the PR for these tests ready ?

[SoohyeokPark-MORAI]

@WJaworskiRobotec Here I share the link PR link

[WJaworskiRobotec]

@SoohyeokPark-MORAI thanks a lot, Have you tested it with AWSIM ?

[SoohyeokPark-MORAI]

@WJaworskiRobotec I tried it with AWSIM but found an issue related to LIDAR whose data is not received.

[WJaworskiRobotec]

OK I will check it with AWSIM with modified QoS

[WJaworskiRobotec]

I have managed to make it work with AWSIM with some changes that I will push to the repo.

*4 failed tests for lidars are related to left and right sensors not included in the binary that I've tested

[Hyeongseok-Jeon]

MORAI update

• I started to look the compatibility test code from @SoohyeokPark-MORAI
• I could not specify the sensor configuration file yet

[WJaworskiRobotec]

@Hyeongseok-Jeon any progress in this task ?

[Hyeongseok-Jeon]

@WJaworskiRobotec There is no prograss on this task in terms of importing the sensor configuration file in compatibility test code. Sorry for delaying and we will work on this issue from next week (after finishing regular version update of our product)

[Hyeongseok-Jeon]

@WJaworskiRobotec @SoohyeokPark-MORAI

How about build configuration file with the sensor category and corresponding topics?
Sample for the config file will be below

{
  "Target_sim": [
    "AWSIM"
  ],
  "Camera_list": [
    "/sensing/camera/traffic_light/image_raw",
	"/sensing/camera/traffic_light/camera_info"
  ],
  "GPS_list": [
	 "/sensing/gnss/pose",
	 "/sensing/gnss/pose_with_covariance"
  ],
  "Lidar_list": [
        "/sensing/lidar/left/pointcloud_raw",
	"/sensing/lidar/left/pointcloud_raw_ex",
	"/sensing/lidar/right/pointcloud_raw",
	"/sensing/lidar/right/pointcloud_raw_ex",
	"/sensing/lidar/top/pointcloud_raw",
	"/sensing/lidar/top/pointcloud_raw_ex"
  ]
}

Based on this structure, compatibility test will be processed only for the topic in the above configuration file on the Target simulator.
Please let me know your opinion.

[WJaworskiRobotec]

@Hyeongseok-Jeon
In general looks fine, but I think we should add msg type to each topic, right ?
For example "/sensing/gnss/pose" and
"/sensing/gnss/pose_with_covariance" have different message types, the same for image_raw and camera_info.

How about the structure like that:

{
    "Target_sim": [
      "AWSIM"
    ],
    "Camera_list": [
      {
        "topic":"/sensing/camera/traffic_light/image_raw",
        "msg": "..."
      },
      {
        "topic":"/sensing/camera/traffic_light/camera_info",
        "msg": "..."
      }
    ],
...
}

[Hyeongseok-Jeon]

MORAI started developing the compatibility test tool which will be done before Jan 18th.

[WJaworskiRobotec]

@SoohyeokPark-MORAI
I see you started to push some commits, let me know when It's ready for initial testing :)

[Soohyeok-Park]

@WJaworskiRobotec @Hyeongseok-Jeon
There have been some bug fixes and It's ready to be tested.
I have tested with the MORAI SIM.

[WJaworskiRobotec]

@SoohyeokPark-MORAI

Should I use humble or galactic ?

btw. Have you added some documentation of how to run the tests ? If not , it might be a good idea to add it a a separate page here : https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ under Digital Twin Simulations

[Hyeongseok-Jeon]

@WJaworskiRobotec @Soohyeok-Park

1. We test compatiblity testing tool based on humble
2. To run the test, please refer the followings

cd ~/${autoware installed directory}
source /opt/ros/humble/setup.bash
source install/setup.bash

cd ~/${autoware installed directory}/src/universe/autoware.universe/tools/simulator_test/simulator_compatibility_test/test_sim_common_manual_testing/
python3 -m pytest .

3. I will make PR for the compatilibty test manual soon.

[Hyeongseok-Jeon]

@WJaworskiRobotec @SoohyeokPark-MORAI

I created PR in the link
Sorry for delaying and please let me know if you have any comment on the PR

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[Hyeongseok-Jeon]

@WJaworskiRobotec

Does it have to be higher priority? Since Soohyeok is decided to leave MORAI, there will be some delay before hiring new developer.

[WJaworskiRobotec]

Sure, no problem. It's definitely not the highest priority

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.