Merge pull request #33 from JulienStanguennec-Leddartech/feat/wod-dataparser

feat/wod dataparser

Author: georghess

.github/workflows/core_code_checks.yml

@@ -15,17 +15,18 @@ jobs:
 
     steps:
       - uses: actions/checkout@v3
-      - name: Set up Python 3.8.13
+      - name: Set up Python 3.10.14
         uses: actions/setup-python@v4
         with:
-          python-version: '3.8.13'
+          python-version: '3.10.14'
       - uses: actions/cache@v2
         with:
           path: ${{ env.pythonLocation }}
           key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
       - name: Install dependencies
         run: |
           pip install --upgrade --upgrade-strategy eager -e .[dev]
+          pip install waymo-open-dataset-tf-2-11-0==1.6.1
       - name: Check notebook cell metadata
         run: |
           python ./nerfstudio/scripts/docs/add_nb_tags.py --check

Dockerfile

@@ -138,6 +138,9 @@ RUN git clone --recursive https://github.com/cvg/pixel-perfect-sfm.git && \
     python3.10 -m pip install --no-cache-dir -e . && \
     cd ..
 
+# Install waymo-open-dataset
+RUN python3.10 -m pip install --no-cache-dir waymo-open-dataset-tf-2-11-0==1.6.1
+
 # Copy nerfstudio folder.
 ADD . /nerfstudio
 

README.md

@@ -82,10 +82,10 @@ Our installation steps largely follow Nerfstudio, with some added dataset-specif
 
 ### Create environment
 
-NeuRAD requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/miniconda.html) before proceeding.
+NeuRAD requires `python >= 3.10`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/miniconda.html) before proceeding.
 
 ```bash
-conda create --name neurad -y python=3.8
+conda create --name neurad -y python=3.10
 conda activate neurad
 pip install --upgrade pip
 ```
@@ -109,6 +109,11 @@ pip install --upgrade pip "setuptools<70.0"
 pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
 ```
 
+For support of Waymo-Open-Dataset v2 (requires python3.10, also dependencies from this package are very strict so cannot add it to pyproject.toml and need install first):
+```bash
+pip install waymo-open-dataset-tf-2-11-0==1.6.1
+```
+
 We refer to [Nerfstudio](https://github.com/nerfstudio-project/nerfstudio/blob/v1.0.3/docs/quickstart/installation.md) for more installation support.
 
 ### Installing NeuRAD
@@ -227,8 +232,9 @@ To add a dataset, create `nerfstudio/data/dataparsers/mydataset.py` containing o
 | 🚗 [Argoverse 2](https://www.argoverse.org/av2.html)   | 7 ring cameras + 2 stereo cameras            | 2 x 32-beam lidars                   |
 | 🚗 [PandaSet](https://pandaset.org/) ([huggingface download](https://huggingface.co/datasets/georghess/pandaset))         | 6 cameras | 64-beam lidar                                  |
 | 🚗 [KITTIMOT](https://www.cvlibs.net/datasets/kitti/eval_tracking.php) ([Timestamps](https://www.cvlibs.net/datasets/kitti/raw_data.php)) | 2 stereo cameras | 64-beam lidar
+| 🚗 [Waymo v2](https://waymo.com/open/)         | 5 cameras | 64-beam lidar   
 
-
+A brief introduction about Waymo dataparser for NeuRAD can be found in [waymo_dataparser.md](./nerfstudio/data//dataparsers/waymo_dataparser.md)
 
 ## Adding Methods
 

docs/_static/imgs/NeuRAD-RS-Waymo-Front.png

@@ -921,12 +921,22 @@ def _compute_rays_for_vr180(
 
         if self.metadata and "rolling_shutter_offsets" in self.metadata and "velocities" in self.metadata:
             cam_idx = camera_indices.squeeze(-1)
-            heights, rows = self.height[cam_idx], coords[..., 0:1]
-            duration = self.metadata["rolling_shutter_offsets"][cam_idx].diff()
-            time_offsets = rows / heights * duration + self.metadata["rolling_shutter_offsets"][cam_idx][..., 0:1]
+            offsets = self.metadata["rolling_shutter_offsets"][cam_idx]
+            duration = offsets.diff()
+            if "rs_direction" in metadata and metadata["rs_direction"] == "Horizontal":
+                # wod (LEFT_TO_RIGHT or RIGHT_TO_LEFT)
+                width, cols = self.width[cam_idx], coords[..., 1:2]
+                time_offsets = cols / width * duration + offsets[..., 0:1]
+            else:
+                # pandaset (TOP_TO_BOTTOM)
+                heights, rows = self.height[cam_idx], coords[..., 0:1]
+                time_offsets = rows / heights * duration + offsets[..., 0:1]
+
             origins = origins + self.metadata["velocities"][cam_idx] * time_offsets
             times = times + time_offsets
             del metadata["rolling_shutter_offsets"]  # it has served its purpose
+            if "rs_direction" in metadata:
+                del metadata["rs_direction"]  # it has served its purpose
 
         return RayBundle(
             origins=origins,

docs/_static/imgs/NeuRAD-RS-Waymo-Left.png

@@ -35,6 +35,7 @@
 from nerfstudio.utils.misc import strtobool, torch_compile
 from nerfstudio.utils.tensor_dataclass import TensorDataclass
 
+# torch._dynamo.config.suppress_errors = True
 TORCH_DEVICE = Union[torch.device, str]  # pylint: disable=invalid-name
 
 HORIZONTAL_BEAM_DIVERGENCE = 3.0e-3  # radians, or meters at a distance of 1m
@@ -50,6 +51,7 @@ class LidarType(Enum):
     VELODYNE64E = auto()
     VELODYNE128 = auto()
     PANDAR64 = auto()
+    WOD64 = auto()
 
 
 LIDAR_MODEL_TO_TYPE = {
@@ -59,6 +61,7 @@ class LidarType(Enum):
     "VELODYNE64E": LidarType.VELODYNE64E,
     "VELODYNE128": LidarType.VELODYNE128,
     "PANDAR64": LidarType.PANDAR64,
+    "WOD64": LidarType.WOD64,
 }
 
 

docs/_static/imgs/NeuRAD-RS-Waymo-Right.png

@@ -26,6 +26,7 @@
 from nerfstudio.data.dataparsers.kittimot_dataparser import KittiMotDataParserConfig
 from nerfstudio.data.dataparsers.nuscenes_dataparser import NuScenesDataParserConfig
 from nerfstudio.data.dataparsers.pandaset_dataparser import PandaSetDataParserConfig
+from nerfstudio.data.dataparsers.wod_dataparser import WoDParserConfig
 from nerfstudio.data.dataparsers.zod_dataparser import ZodDataParserConfig
 from nerfstudio.plugins.registry_dataparser import discover_dataparsers
 
@@ -35,6 +36,7 @@
     "argoverse2-data": Argoverse2DataParserConfig(),
     "zod-data": ZodDataParserConfig(),
     "pandaset-data": PandaSetDataParserConfig(),
+    "wod-data": WoDParserConfig(),
 }
 
 external_dataparsers, _ = discover_dataparsers()

nerfstudio/cameras/cameras.py

@@ -0,0 +1,52 @@
+# NeuRAD on Waymo open dataset
+
+## About
+Thanks to the excellent work of NeuRAD, we reproduce some results on the Waymo open dataset.
+
+Our goal in reproducing and open-sourcing this waymo dataparser for NeuRAD is to provide a basic reference for the self-driving community and to inspire more work.
+
+In the same folder, there is [wod_dataparser.py](./wod_dataparser.py) which followed the [README-Adding Datasets](https://github.com/georghess/neurad-studio?tab=readme-ov-file#adding-datasets) suggestions. In addition, we added also [wod_utils.py](./wod_utils.py) which did the main work for converting/exporting Waymo dataset.
+
+In addition, we have also added the rolling shutter support for Waymo dataset as the rolling shutter direction is horizontal instead of the vertical one in Pandaset. Here are some examples of the comparison results (on squence of 10588):
+![](./../../../docs/_static/imgs/NeuRAD-RS-Waymo-Front.png)
+![](./../../../docs/_static/imgs/NeuRAD-RS-Waymo-Left.png)
+![](./../../../docs/_static/imgs/NeuRAD-RS-Waymo-Right.png)
+
+
+### Benchmark between Pandaset & Waymo
+|  Dataset	| Sequence 	|  Frames | Cameras	|  PSNR	| SSIM 	| LIPS |
+|---	    |---	    |---	  |---      |---	|---	|---   |
+| Pandaset	| 006     	|80       | FC   	|25.1562​|0.8044​ |0.1575​|
+| Pandaset 	| 011  	    |80   	  | 360  	|26.3919​|0.8057​ |0.2029​|
+| Waymo  	| 10588771936253546636|  50	|   FC	| 27.5555|0.8547|0.121
+| Waymo  	| 473735159277431842  | 150| FC 	| 29.1758|0.8717|0.1592
+| Waymo  	| 4468278022208380281  | ALL| FC 	|30.5247​|0.8787​|0.1701​
+
+Notes: All above results were obtained with the same hyperparameters and configurations from NeuRAD paper (**Appendix A**)
+
+### Results
+#### Waymo RGB rendering - Sequence 10588 - 3 cameras (FC_LEFT, FC, FC_RIGHT)
+[![Sequence 10588 - 3 cameras](http://img.youtube.com/vi/eR1bHeh7p8A/0.jpg)](https://www.youtube.com/watch?v=eR1bHeh7p8A)
+> Up is ground truth, bottom is rendered.
+
+#### Actor removal - Sequence 20946​ - FC cameras
+[![Sequence 20946](http://img.youtube.com/vi/mkMdzAvTez4/0.jpg)](https://www.youtube.com/watch?v=mkMdzAvTez4)
+> Left is ground truth, right is rendered.
+
+#### Novel view synthesis - Sequence 20946​ - Ego vehicle 1m up
+[![Ego vehicle 1m up](http://img.youtube.com/vi/U8VRboWLj_c/0.jpg)](https://www.youtube.com/watch?v=U8VRboWLj_c)
+> Left is ground truth, right is rendered.
+
+#### Novel view synthesis - Sequence 20946​ - Ego vehicle 1m left
+[![Ego vehicle 1m left](http://img.youtube.com/vi/q_HFmc6JPzQ/0.jpg)](https://www.youtube.com/watch?v=q_HFmc6JPzQ)
+> Left is ground truth, right is rendered.
+
+## Links
+
+Results has been done with waymo open dataset [v2.0.0, gcloud link](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_2_0_0)
+
+## Contributors
+
+- Lei Lei, Leddartech
+- Julien Stanguennec, Leddartech
+- Pierre Merriaux, Leddartech