From cd3f7b20df5b12f4f99386db459a7b847dfc7f9a Mon Sep 17 00:00:00 2001 From: Ryohsuke Mitsudome Date: Tue, 30 Apr 2024 17:47:19 +0900 Subject: [PATCH 1/6] feat: add TIER IV datasets Signed-off-by: Ryohsuke Mitsudome --- docs/datasets/.pages | 2 + docs/datasets/annotated-data/index.md | 78 ++++++++++++++++++++++++++ docs/datasets/index.md | 76 ++----------------------- docs/datasets/raw-sensor-data/index.md | 11 ++++ 4 files changed, 96 insertions(+), 71 deletions(-) create mode 100644 docs/datasets/annotated-data/index.md create mode 100644 docs/datasets/raw-sensor-data/index.md diff --git a/docs/datasets/.pages b/docs/datasets/.pages index 35fd5a113be..a964fbd7180 100644 --- a/docs/datasets/.pages +++ b/docs/datasets/.pages @@ -1,2 +1,4 @@ nav: - index.md + - Vehicle Sensor Data: vehicle-sensor-data + - Annotated Data: annotated-data diff --git a/docs/datasets/annotated-data/index.md b/docs/datasets/annotated-data/index.md new file mode 100644 index 00000000000..7efd29c4d6f --- /dev/null +++ b/docs/datasets/annotated-data/index.md @@ -0,0 +1,78 @@ +# Raw Sensor Datasets + +This page explains about the datsets collected from the sensors attached to a vehicle. + +## Bus-ODD (Operational Design Domain) datasets + +### Leo Drive - ISUZU sensor data + +This dataset contains data from the Isuzu bus used in the Bus ODD project. + +The data contains data from following sensors: + +- 1 x VLP16 +- 2 x VLP32C +- 1 x Applanix POS LV 120 GNSS/INS +- 3 x Lucid Vision Triton 5.4MP cameras (left, right, front) +- Vehicle status report + +It also contains `/tf` topic for static transformations between sensors. + +#### Required message types + +The GNSS data is available in `sensor_msgs/msg/NavSatFix` message type. + +But also the Applanix raw messages are also included in `applanix_msgs/msg/NavigationPerformanceGsof50` and `applanix_msgs/msg/NavigationSolutionGsof49` message types. +In order to be able to play back these messages, you need to build and source the `applanix_msgs` package. + +```bash +# Create a workspace and clone the repository +mkdir -p ~/applanix_ws/src && cd "$_" +git clone https://github.com/autowarefoundation/applanix.git +cd .. + +# Build the workspace +colcon build --symlink-install --packages-select applanix_msgs + +# Source the workspace +source ~/applanix_ws/install/setup.bash + +# Now you can play back the messages +``` + +Also make sure to source Autoware Universe workspace too. + +#### Download instructions + +```console +# Install awscli +$ sudo apt update && sudo apt install awscli -y + +# This will download the entire dataset to the current directory. +# (About 10.9GB of data) +$ aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ ./2022-08-22_leo_drive_isuzu_bags --no-sign-request + +# Optionally, +# If you instead want to download a single bag file, you can get a list of the available files with following: +$ aws s3 ls s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ --no-sign-request + PRE all-sensors-bag1_compressed/ + PRE all-sensors-bag2_compressed/ + PRE all-sensors-bag3_compressed/ + PRE all-sensors-bag4_compressed/ + PRE all-sensors-bag5_compressed/ + PRE all-sensors-bag6_compressed/ + PRE driving_20_kmh_2022_06_10-16_01_55_compressed/ + PRE driving_30_kmh_2022_06_10-15_47_42_compressed/ + +# Then you can download a single bag file with the following: +aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/all-sensors-bag1_compressed/ ./all-sensors-bag1_compressed --no-sign-request +``` + +### AutoCore.ai - lidar ROS 2 bag file and pcap + +This dataset contains pcap files and ros2 bag files from Ouster OS1-64 Lidar. +The pcap file and ros2 bag file is recorded in the same time with slight difference in duration. + +[Click here to download (~553MB)](https://autoware-files.s3.us-west-2.amazonaws.com/collected_data/2022-04-14_autocore-lidar-bag-pcap/Lidar_Data_220414_bag_pcap.zip) + +[Reference Issue](https://github.com/autowarefoundation/autoware.universe/issues/562#issuecomment-1102662448) diff --git a/docs/datasets/index.md b/docs/datasets/index.md index 475c999a76a..5228bb379c1 100644 --- a/docs/datasets/index.md +++ b/docs/datasets/index.md @@ -2,77 +2,11 @@ Autoware partners provide datasets for testing and development. These datasets are available for download here. -## Bus-ODD (Operational Design Domain) datasets +## Raw Sensor Datasets -### Leo Drive - ISUZU sensor data +We provide raw data from the sensors attached to a vehicle. +The data is mainly used to learn how data should be provided to Autoware or to test functions of Autoware. -This dataset contains data from the Isuzu bus used in the Bus ODD project. +## Annotated Datasets -The data contains data from following sensors: - -- 1 x VLP16 -- 2 x VLP32C -- 1 x Applanix POS LV 120 GNSS/INS -- 3 x Lucid Vision Triton 5.4MP cameras (left, right, front) -- Vehicle status report - -It also contains `/tf` topic for static transformations between sensors. - -#### Required message types - -The GNSS data is available in `sensor_msgs/msg/NavSatFix` message type. - -But also the Applanix raw messages are also included in `applanix_msgs/msg/NavigationPerformanceGsof50` and `applanix_msgs/msg/NavigationSolutionGsof49` message types. -In order to be able to play back these messages, you need to build and source the `applanix_msgs` package. - -```bash -# Create a workspace and clone the repository -mkdir -p ~/applanix_ws/src && cd "$_" -git clone https://github.com/autowarefoundation/applanix.git -cd .. - -# Build the workspace -colcon build --symlink-install --packages-select applanix_msgs - -# Source the workspace -source ~/applanix_ws/install/setup.bash - -# Now you can play back the messages -``` - -Also make sure to source Autoware Universe workspace too. - -#### Download instructions - -```console -# Install awscli -$ sudo apt update && sudo apt install awscli -y - -# This will download the entire dataset to the current directory. -# (About 10.9GB of data) -$ aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ ./2022-08-22_leo_drive_isuzu_bags --no-sign-request - -# Optionally, -# If you instead want to download a single bag file, you can get a list of the available files with following: -$ aws s3 ls s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ --no-sign-request - PRE all-sensors-bag1_compressed/ - PRE all-sensors-bag2_compressed/ - PRE all-sensors-bag3_compressed/ - PRE all-sensors-bag4_compressed/ - PRE all-sensors-bag5_compressed/ - PRE all-sensors-bag6_compressed/ - PRE driving_20_kmh_2022_06_10-16_01_55_compressed/ - PRE driving_30_kmh_2022_06_10-15_47_42_compressed/ - -# Then you can download a single bag file with the following: -aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/all-sensors-bag1_compressed/ ./all-sensors-bag1_compressed --no-sign-request -``` - -### AutoCore.ai - lidar ROS 2 bag file and pcap - -This dataset contains pcap files and ros2 bag files from Ouster OS1-64 Lidar. -The pcap file and ros2 bag file is recorded in the same time with slight difference in duration. - -[Click here to download (~553MB)](https://autoware-files.s3.us-west-2.amazonaws.com/collected_data/2022-04-14_autocore-lidar-bag-pcap/Lidar_Data_220414_bag_pcap.zip) - -[Reference Issue](https://github.com/autowarefoundation/autoware.universe/issues/562#issuecomment-1102662448) +We provide annoated datasets for training/evaluating machine learning based modules. diff --git a/docs/datasets/raw-sensor-data/index.md b/docs/datasets/raw-sensor-data/index.md new file mode 100644 index 00000000000..d89997b3bef --- /dev/null +++ b/docs/datasets/raw-sensor-data/index.md @@ -0,0 +1,11 @@ +# Annotaed Dataset + +## TIER IV Dataset + +[Download Link](https://drive.google.com/drive/folders/1VVUq48NOjYQ9zHNEg3_GTU_FpFkkquxZ?usp=drive_link) + +The data contains the following information with annotation: + - 63316 images from 6 camearas (All faces and license plates are blurred) + - 13758 frames of lidar points + +The data format is based on nuScenes and nuImages format. Form the details, please check this [link](https://github.com/tier4/tier4_perception_dataset/blob/main/docs/t4_format_3d_detailed.md). From 12bd704bc5201bf78518f7df3bb6fe7444cf3a31 Mon Sep 17 00:00:00 2001 From: Ryohsuke Mitsudome Date: Wed, 1 May 2024 00:04:41 +0900 Subject: [PATCH 2/6] fix: fix typo Signed-off-by: Ryohsuke Mitsudome --- docs/datasets/raw-sensor-data/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/datasets/raw-sensor-data/index.md b/docs/datasets/raw-sensor-data/index.md index d89997b3bef..98898e9d0c5 100644 --- a/docs/datasets/raw-sensor-data/index.md +++ b/docs/datasets/raw-sensor-data/index.md @@ -1,4 +1,4 @@ -# Annotaed Dataset +# Annotated Dataset ## TIER IV Dataset From ee89bfda2edc9cf50d5a4d1e7e07e830b820cab1 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 30 Apr 2024 15:06:08 +0000 Subject: [PATCH 3/6] style(pre-commit): autofix --- docs/datasets/raw-sensor-data/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/datasets/raw-sensor-data/index.md b/docs/datasets/raw-sensor-data/index.md index 98898e9d0c5..51dad3aaa75 100644 --- a/docs/datasets/raw-sensor-data/index.md +++ b/docs/datasets/raw-sensor-data/index.md @@ -5,7 +5,8 @@ [Download Link](https://drive.google.com/drive/folders/1VVUq48NOjYQ9zHNEg3_GTU_FpFkkquxZ?usp=drive_link) The data contains the following information with annotation: - - 63316 images from 6 camearas (All faces and license plates are blurred) - - 13758 frames of lidar points + +- 63316 images from 6 camearas (All faces and license plates are blurred) +- 13758 frames of lidar points The data format is based on nuScenes and nuImages format. Form the details, please check this [link](https://github.com/tier4/tier4_perception_dataset/blob/main/docs/t4_format_3d_detailed.md). From c6a9e1a5d527c079e10bf2ac97e208d057e95221 Mon Sep 17 00:00:00 2001 From: Ryohsuke Mitsudome Date: Tue, 7 May 2024 15:26:14 +0900 Subject: [PATCH 4/6] fix: fix links to raw-sensor-data Signed-off-by: Ryohsuke Mitsudome --- docs/datasets/.pages | 2 +- docs/datasets/annotated-data/index.md | 80 +++----------------------- docs/datasets/raw-sensor-data/index.md | 80 +++++++++++++++++++++++--- 3 files changed, 81 insertions(+), 81 deletions(-) diff --git a/docs/datasets/.pages b/docs/datasets/.pages index a964fbd7180..c597c4913a4 100644 --- a/docs/datasets/.pages +++ b/docs/datasets/.pages @@ -1,4 +1,4 @@ nav: - index.md - - Vehicle Sensor Data: vehicle-sensor-data + - Raw Sensor Data: raw-sensor-data - Annotated Data: annotated-data diff --git a/docs/datasets/annotated-data/index.md b/docs/datasets/annotated-data/index.md index 7efd29c4d6f..51dad3aaa75 100644 --- a/docs/datasets/annotated-data/index.md +++ b/docs/datasets/annotated-data/index.md @@ -1,78 +1,12 @@ -# Raw Sensor Datasets +# Annotated Dataset -This page explains about the datsets collected from the sensors attached to a vehicle. +## TIER IV Dataset -## Bus-ODD (Operational Design Domain) datasets +[Download Link](https://drive.google.com/drive/folders/1VVUq48NOjYQ9zHNEg3_GTU_FpFkkquxZ?usp=drive_link) -### Leo Drive - ISUZU sensor data +The data contains the following information with annotation: -This dataset contains data from the Isuzu bus used in the Bus ODD project. +- 63316 images from 6 camearas (All faces and license plates are blurred) +- 13758 frames of lidar points -The data contains data from following sensors: - -- 1 x VLP16 -- 2 x VLP32C -- 1 x Applanix POS LV 120 GNSS/INS -- 3 x Lucid Vision Triton 5.4MP cameras (left, right, front) -- Vehicle status report - -It also contains `/tf` topic for static transformations between sensors. - -#### Required message types - -The GNSS data is available in `sensor_msgs/msg/NavSatFix` message type. - -But also the Applanix raw messages are also included in `applanix_msgs/msg/NavigationPerformanceGsof50` and `applanix_msgs/msg/NavigationSolutionGsof49` message types. -In order to be able to play back these messages, you need to build and source the `applanix_msgs` package. - -```bash -# Create a workspace and clone the repository -mkdir -p ~/applanix_ws/src && cd "$_" -git clone https://github.com/autowarefoundation/applanix.git -cd .. - -# Build the workspace -colcon build --symlink-install --packages-select applanix_msgs - -# Source the workspace -source ~/applanix_ws/install/setup.bash - -# Now you can play back the messages -``` - -Also make sure to source Autoware Universe workspace too. - -#### Download instructions - -```console -# Install awscli -$ sudo apt update && sudo apt install awscli -y - -# This will download the entire dataset to the current directory. -# (About 10.9GB of data) -$ aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ ./2022-08-22_leo_drive_isuzu_bags --no-sign-request - -# Optionally, -# If you instead want to download a single bag file, you can get a list of the available files with following: -$ aws s3 ls s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ --no-sign-request - PRE all-sensors-bag1_compressed/ - PRE all-sensors-bag2_compressed/ - PRE all-sensors-bag3_compressed/ - PRE all-sensors-bag4_compressed/ - PRE all-sensors-bag5_compressed/ - PRE all-sensors-bag6_compressed/ - PRE driving_20_kmh_2022_06_10-16_01_55_compressed/ - PRE driving_30_kmh_2022_06_10-15_47_42_compressed/ - -# Then you can download a single bag file with the following: -aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/all-sensors-bag1_compressed/ ./all-sensors-bag1_compressed --no-sign-request -``` - -### AutoCore.ai - lidar ROS 2 bag file and pcap - -This dataset contains pcap files and ros2 bag files from Ouster OS1-64 Lidar. -The pcap file and ros2 bag file is recorded in the same time with slight difference in duration. - -[Click here to download (~553MB)](https://autoware-files.s3.us-west-2.amazonaws.com/collected_data/2022-04-14_autocore-lidar-bag-pcap/Lidar_Data_220414_bag_pcap.zip) - -[Reference Issue](https://github.com/autowarefoundation/autoware.universe/issues/562#issuecomment-1102662448) +The data format is based on nuScenes and nuImages format. Form the details, please check this [link](https://github.com/tier4/tier4_perception_dataset/blob/main/docs/t4_format_3d_detailed.md). diff --git a/docs/datasets/raw-sensor-data/index.md b/docs/datasets/raw-sensor-data/index.md index 51dad3aaa75..7efd29c4d6f 100644 --- a/docs/datasets/raw-sensor-data/index.md +++ b/docs/datasets/raw-sensor-data/index.md @@ -1,12 +1,78 @@ -# Annotated Dataset +# Raw Sensor Datasets -## TIER IV Dataset +This page explains about the datsets collected from the sensors attached to a vehicle. -[Download Link](https://drive.google.com/drive/folders/1VVUq48NOjYQ9zHNEg3_GTU_FpFkkquxZ?usp=drive_link) +## Bus-ODD (Operational Design Domain) datasets -The data contains the following information with annotation: +### Leo Drive - ISUZU sensor data -- 63316 images from 6 camearas (All faces and license plates are blurred) -- 13758 frames of lidar points +This dataset contains data from the Isuzu bus used in the Bus ODD project. -The data format is based on nuScenes and nuImages format. Form the details, please check this [link](https://github.com/tier4/tier4_perception_dataset/blob/main/docs/t4_format_3d_detailed.md). +The data contains data from following sensors: + +- 1 x VLP16 +- 2 x VLP32C +- 1 x Applanix POS LV 120 GNSS/INS +- 3 x Lucid Vision Triton 5.4MP cameras (left, right, front) +- Vehicle status report + +It also contains `/tf` topic for static transformations between sensors. + +#### Required message types + +The GNSS data is available in `sensor_msgs/msg/NavSatFix` message type. + +But also the Applanix raw messages are also included in `applanix_msgs/msg/NavigationPerformanceGsof50` and `applanix_msgs/msg/NavigationSolutionGsof49` message types. +In order to be able to play back these messages, you need to build and source the `applanix_msgs` package. + +```bash +# Create a workspace and clone the repository +mkdir -p ~/applanix_ws/src && cd "$_" +git clone https://github.com/autowarefoundation/applanix.git +cd .. + +# Build the workspace +colcon build --symlink-install --packages-select applanix_msgs + +# Source the workspace +source ~/applanix_ws/install/setup.bash + +# Now you can play back the messages +``` + +Also make sure to source Autoware Universe workspace too. + +#### Download instructions + +```console +# Install awscli +$ sudo apt update && sudo apt install awscli -y + +# This will download the entire dataset to the current directory. +# (About 10.9GB of data) +$ aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ ./2022-08-22_leo_drive_isuzu_bags --no-sign-request + +# Optionally, +# If you instead want to download a single bag file, you can get a list of the available files with following: +$ aws s3 ls s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/ --no-sign-request + PRE all-sensors-bag1_compressed/ + PRE all-sensors-bag2_compressed/ + PRE all-sensors-bag3_compressed/ + PRE all-sensors-bag4_compressed/ + PRE all-sensors-bag5_compressed/ + PRE all-sensors-bag6_compressed/ + PRE driving_20_kmh_2022_06_10-16_01_55_compressed/ + PRE driving_30_kmh_2022_06_10-15_47_42_compressed/ + +# Then you can download a single bag file with the following: +aws s3 sync s3://autoware-files/collected_data/2022-08-22_leo_drive_isuzu_bags/all-sensors-bag1_compressed/ ./all-sensors-bag1_compressed --no-sign-request +``` + +### AutoCore.ai - lidar ROS 2 bag file and pcap + +This dataset contains pcap files and ros2 bag files from Ouster OS1-64 Lidar. +The pcap file and ros2 bag file is recorded in the same time with slight difference in duration. + +[Click here to download (~553MB)](https://autoware-files.s3.us-west-2.amazonaws.com/collected_data/2022-04-14_autocore-lidar-bag-pcap/Lidar_Data_220414_bag_pcap.zip) + +[Reference Issue](https://github.com/autowarefoundation/autoware.universe/issues/562#issuecomment-1102662448) From 8a90a326e90d88574fc4ca54774f65a471bfd010 Mon Sep 17 00:00:00 2001 From: Ryohsuke Mitsudome Date: Tue, 14 May 2024 22:21:08 +0900 Subject: [PATCH 5/6] fix: update dataset top page Signed-off-by: Ryohsuke Mitsudome --- docs/datasets/index.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/datasets/index.md b/docs/datasets/index.md index 5228bb379c1..ad25a3cc630 100644 --- a/docs/datasets/index.md +++ b/docs/datasets/index.md @@ -2,11 +2,5 @@ Autoware partners provide datasets for testing and development. These datasets are available for download here. -## Raw Sensor Datasets - -We provide raw data from the sensors attached to a vehicle. -The data is mainly used to learn how data should be provided to Autoware or to test functions of Autoware. - -## Annotated Datasets - -We provide annoated datasets for training/evaluating machine learning based modules. + * [Raw Sensor Datasets](./raw-sensor-data/index.md): raw data from sensors attached to a vehicle. They are mainly used to learn how data should be provided to Autoware or test functions of Autoware. + * [Annotated Datasets](./annotated-data/index.md): Annotated datasets for training/evaluating machine learning based functions. From f6263b5aeb8b6708ca8efbe064746df19fd035cb Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 14 May 2024 13:22:24 +0000 Subject: [PATCH 6/6] style(pre-commit): autofix --- docs/datasets/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/datasets/index.md b/docs/datasets/index.md index ad25a3cc630..ebdd7cd472c 100644 --- a/docs/datasets/index.md +++ b/docs/datasets/index.md @@ -2,5 +2,5 @@ Autoware partners provide datasets for testing and development. These datasets are available for download here. - * [Raw Sensor Datasets](./raw-sensor-data/index.md): raw data from sensors attached to a vehicle. They are mainly used to learn how data should be provided to Autoware or test functions of Autoware. - * [Annotated Datasets](./annotated-data/index.md): Annotated datasets for training/evaluating machine learning based functions. +- [Raw Sensor Datasets](./raw-sensor-data/index.md): raw data from sensors attached to a vehicle. They are mainly used to learn how data should be provided to Autoware or test functions of Autoware. +- [Annotated Datasets](./annotated-data/index.md): Annotated datasets for training/evaluating machine learning based functions.