diff --git a/_publications/splatad/argoverse2_front_2_neurad.jpg b/_publications/splatad/argoverse2_front_2_neurad.jpg new file mode 100644 index 0000000..2c0fcc5 Binary files /dev/null and b/_publications/splatad/argoverse2_front_2_neurad.jpg differ diff --git a/_publications/splatad/argoverse2_front_2_splatad.jpg b/_publications/splatad/argoverse2_front_2_splatad.jpg new file mode 100644 index 0000000..3739b48 Binary files /dev/null and b/_publications/splatad/argoverse2_front_2_splatad.jpg differ diff --git a/_publications/splatad/argoverse2_front_splatad.jpg b/_publications/splatad/argoverse2_front_splatad.jpg new file mode 100644 index 0000000..c255050 Binary files /dev/null and b/_publications/splatad/argoverse2_front_splatad.jpg differ diff --git a/_publications/splatad/argoverse2_front_streetgs.jpg b/_publications/splatad/argoverse2_front_streetgs.jpg new file mode 100644 index 0000000..9101661 Binary files /dev/null and b/_publications/splatad/argoverse2_front_streetgs.jpg differ diff --git a/_publications/splatad/front-fig-stacked.jpg b/_publications/splatad/front-fig-stacked.jpg new file mode 100644 index 0000000..0767e2c Binary files /dev/null and b/_publications/splatad/front-fig-stacked.jpg differ diff --git a/_publications/splatad/frontfig.png b/_publications/splatad/frontfig.png new file mode 100644 index 0000000..c4f94c0 Binary files /dev/null and b/_publications/splatad/frontfig.png differ diff --git a/_publications/splatad/hessian.png b/_publications/splatad/hessian.png new file mode 100644 index 0000000..51dcf4f Binary files /dev/null and b/_publications/splatad/hessian.png differ diff --git a/_publications/splatad/methodoverview.png b/_publications/splatad/methodoverview.png new file mode 100644 index 0000000..5f3b644 Binary files /dev/null and b/_publications/splatad/methodoverview.png differ diff --git a/_publications/splatad/nuscenes_front_splatad.jpg b/_publications/splatad/nuscenes_front_splatad.jpg new file mode 100644 index 0000000..47fc81d Binary files /dev/null and b/_publications/splatad/nuscenes_front_splatad.jpg differ diff --git a/_publications/splatad/nuscenes_front_streetgs.jpg b/_publications/splatad/nuscenes_front_streetgs.jpg new file mode 100644 index 0000000..c2ad5d4 Binary files /dev/null and b/_publications/splatad/nuscenes_front_streetgs.jpg differ diff --git a/_publications/splatad/pandaset_016_front_neurad.jpg b/_publications/splatad/pandaset_016_front_neurad.jpg new file mode 100644 index 0000000..37a31af Binary files /dev/null and b/_publications/splatad/pandaset_016_front_neurad.jpg differ diff --git a/_publications/splatad/pandaset_016_front_splatad.jpg b/_publications/splatad/pandaset_016_front_splatad.jpg new file mode 100644 index 0000000..f39b9d3 Binary files /dev/null and b/_publications/splatad/pandaset_016_front_splatad.jpg differ diff --git a/_publications/splatad/qualitative-comparison.jpg b/_publications/splatad/qualitative-comparison.jpg new file mode 100644 index 0000000..6f66db1 Binary files /dev/null and b/_publications/splatad/qualitative-comparison.jpg differ diff --git a/_publications/splatad/references.bib b/_publications/splatad/references.bib new file mode 100644 index 0000000..e69de29 diff --git a/_publications/splatad/splatad.md b/_publications/splatad/splatad.md new file mode 100644 index 0000000..15238ec --- /dev/null +++ b/_publications/splatad/splatad.md @@ -0,0 +1,275 @@ +--- +layout: publication +permalink: /publications/splatad/ +title: "SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving" +venue: ARXIV +authors: + - Hess + - Lindstrom + - Fatemi + - Petersson + - Svensson +code: https://github.com/carlinds/splatad +date: 2024-11-30 00:00:00 +00:00 +arxiv: https://arxiv.org/abs/2411.XXXXX +n_equal_contrib: 2 +thumbnail-video: +--- +
+
+ +
Fig 1.: SplatAD is the first method capable of realistic camera and lidar rendering using 3D Gaussian Splatting. It enables real-time, high-quality rendering, and can reach competitive performance within minutes of training.
+
+
+ +--- + +# Abstract +Ensuring the safety of autonomous robots, such as self-driving vehicles, requires extensive testing across diverse driving scenarios. Simulation is a key ingredient for conducting such testing in a cost-effective and scalable way. Neural rendering methods have gained popularity, as they can build simulation environments from collected logs in a data-driven manner. However, existing neural radiance field (NeRF) methods for sensor-realistic rendering of camera and lidar data suffer from low rendering speeds, limiting their applicability for large-scale testing. While 3D Gaussian Splatting (3DGS) enables real-time rendering, current methods are limited to camera data and are unable to render lidar data essential for autonomous driving. To address these limitations, we propose SplatAD, the first 3DGS-based method for realistic, real-time rendering of dynamic scenes for both camera and lidar data. SplatAD accurately models key sensor-specific phenomena such as rolling shutter effects, lidar intensity, and lidar ray dropouts, using purpose-built algorithms to optimize rendering efficiency. Evaluation across three autonomous driving datasets demonstrates that SplatAD achieves state-of-the-art rendering quality with up to +2 PSNR for NVS and +3 PSNR for reconstruction while increasing rendering speed over NeRF-based methods by an order of magnitude. Code to be released upon publication. + + +--- + +# Videos + +## Image and lidar rendering +SplatAD can render both images and lidar point clouds. The videos demonstrate rendered images, bird's-eye views of the rendered point cloud, and point clouds overlaid on images, with points colored based on range and intensity. +

+Note: Videos have been compressed to reduce file size. +
+ + +
+
+ + +
+ +## Sequence browser +
+
+ + + +
+
+
+
+ +
+
+ +
+Note: Videos have been compressed to reduce file size. +
+ +
+ + + +## Trajectory editing +Note: Videos have been compressed to reduce file size. + + +--- + +# Method +
+
+ +
+
+ +SplatAD can render both sensor-realistic lidar and camera data from the same 3D Gaussian representation. For lidar rendering, we: +
    +
  1. Project the 3D Gaussians to spherical coordinates.
  2. +
  3. Intersect Gaussians with non-equdistant tiles, matching to the lidar beam distribution.
  4. +
  5. Rasterize depth and features for a non-linear grid of points, with rolling shutter compensation.
  6. +
  7. Decode features to lidar intensity and lidar ray drop probability.
  8. +
+For efficiency, we implement these operations using custom CUDA kernels. + +For the camera rendering, we modify the standard 3DGS pipeline: + + +--- + +# Comparisons + +
+ {% include two_image_slider.html + left_image="pandaset_016_front_splatad.jpg" + right_image="pandaset_016_front_neurad.jpg" + width="50" + id="2" + max-width="1920px" + max-height="1080px" + caption="SplatAD vs NeuRAD NVS rendering on the PandaSet dataset." + %} +
+ +
+ {% include two_image_slider.html + left_image="argoverse2_front_splatad.jpg" + right_image="argoverse2_front_streetgs.jpg" + width="25" + id="3" + linkid="4" + max-width="1550px" + max-height="1798px" + caption="SplatAD vs Street Gaussians NVS rendering on the Argoverse2 dataset." + %} + {% include two_image_slider.html + left_image="argoverse2_front_2_splatad.jpg" + right_image="argoverse2_front_2_neurad.jpg" + width="25" + id="4" + linkid="3" + max-width="1550px" + max-height="1798px" + caption="SplatAD vs NeuRAD NVS rendering on the Argoverse2 dataset." + %} +
+ +
+ {% include two_image_slider.html + left_image="nuscenes_front_splatad.jpg" + right_image="nuscenes_front_streetgs.jpg" + width="50" + id="5" + max-width="1600px" + max-height="900px" + caption="SplatAD vs Street Gaussians NVS rendering on the nuScenes dataset." + %} +
+ +
+
+ +
Fig 6.: SplatAD learns both sharp and accurate geometries, in constrast to NeuRADs oversmoothed results, and Steet Gaussians cutting through objects due to erroneous line-of-sight.
+
+
+ +--- + +# BibTeX +```bibtex +@article{hess2024splatad, + title = {SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving}, + author = {Hess, Georg and Lindstr{\"o}m, Carl and Fatemi, Maryam and Petersson, Christoffer and Svensson, Lennart}, + journal = {arXiv preprint arXiv:2411:XXXXX}, + year = {2024} +} +``` diff --git a/_publications/splatad/videos/Argoverse2/0b86f508.mp4 b/_publications/splatad/videos/Argoverse2/0b86f508.mp4 new file mode 100644 index 0000000..5db1ce5 Binary files /dev/null and b/_publications/splatad/videos/Argoverse2/0b86f508.mp4 differ diff --git a/_publications/splatad/videos/Argoverse2/280269f9.mp4 b/_publications/splatad/videos/Argoverse2/280269f9.mp4 new file mode 100644 index 0000000..e442e7e Binary files /dev/null and b/_publications/splatad/videos/Argoverse2/280269f9.mp4 differ diff --git a/_publications/splatad/videos/Argoverse2/2f2321d2.mp4 b/_publications/splatad/videos/Argoverse2/2f2321d2.mp4 new file mode 100644 index 0000000..2a9b73d Binary files /dev/null and b/_publications/splatad/videos/Argoverse2/2f2321d2.mp4 differ diff --git a/_publications/splatad/videos/Argoverse2/3bffdcff.mp4 b/_publications/splatad/videos/Argoverse2/3bffdcff.mp4 new file mode 100644 index 0000000..a2bf07e Binary files /dev/null and b/_publications/splatad/videos/Argoverse2/3bffdcff.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/001.mp4 b/_publications/splatad/videos/PandaSet/001.mp4 new file mode 100644 index 0000000..230dec9 Binary files /dev/null and b/_publications/splatad/videos/PandaSet/001.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/011.mp4 b/_publications/splatad/videos/PandaSet/011.mp4 new file mode 100644 index 0000000..6c2aa89 Binary files /dev/null and b/_publications/splatad/videos/PandaSet/011.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/016.mp4 b/_publications/splatad/videos/PandaSet/016.mp4 new file mode 100644 index 0000000..a4a4b5c Binary files /dev/null and b/_publications/splatad/videos/PandaSet/016.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/053.mp4 b/_publications/splatad/videos/PandaSet/053.mp4 new file mode 100644 index 0000000..32536b1 Binary files /dev/null and b/_publications/splatad/videos/PandaSet/053.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/106.mp4 b/_publications/splatad/videos/PandaSet/106.mp4 new file mode 100644 index 0000000..e899d3b Binary files /dev/null and b/_publications/splatad/videos/PandaSet/106.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/123.mp4 b/_publications/splatad/videos/PandaSet/123.mp4 new file mode 100644 index 0000000..1634ce3 Binary files /dev/null and b/_publications/splatad/videos/PandaSet/123.mp4 differ diff --git a/_publications/splatad/videos/PandaSet/158.mp4 b/_publications/splatad/videos/PandaSet/158.mp4 new file mode 100644 index 0000000..5fa02de Binary files /dev/null and b/_publications/splatad/videos/PandaSet/158.mp4 differ diff --git a/_publications/splatad/videos/edits/edits.mp4 b/_publications/splatad/videos/edits/edits.mp4 new file mode 100644 index 0000000..6e02769 Binary files /dev/null and b/_publications/splatad/videos/edits/edits.mp4 differ diff --git a/_publications/splatad/videos/edits/rots_106.mp4 b/_publications/splatad/videos/edits/rots_106.mp4 new file mode 100644 index 0000000..4b720c5 Binary files /dev/null and b/_publications/splatad/videos/edits/rots_106.mp4 differ diff --git a/_publications/splatad/videos/edits/shift_001.mp4 b/_publications/splatad/videos/edits/shift_001.mp4 new file mode 100644 index 0000000..45ca581 Binary files /dev/null and b/_publications/splatad/videos/edits/shift_001.mp4 differ diff --git a/_publications/splatad/videos/edits/spiral_011.mp4 b/_publications/splatad/videos/edits/spiral_011.mp4 new file mode 100644 index 0000000..6d65ce3 Binary files /dev/null and b/_publications/splatad/videos/edits/spiral_011.mp4 differ diff --git a/_publications/splatad/videos/multi/pc_bev.mp4 b/_publications/splatad/videos/multi/pc_bev.mp4 new file mode 100644 index 0000000..f88b9eb Binary files /dev/null and b/_publications/splatad/videos/multi/pc_bev.mp4 differ diff --git a/_publications/splatad/videos/multi/pc_overlay_intensity.mp4 b/_publications/splatad/videos/multi/pc_overlay_intensity.mp4 new file mode 100644 index 0000000..1aaa0b7 Binary files /dev/null and b/_publications/splatad/videos/multi/pc_overlay_intensity.mp4 differ diff --git a/_publications/splatad/videos/multi/pc_overlay_range.mp4 b/_publications/splatad/videos/multi/pc_overlay_range.mp4 new file mode 100644 index 0000000..f2c89cb Binary files /dev/null and b/_publications/splatad/videos/multi/pc_overlay_range.mp4 differ diff --git a/_publications/splatad/videos/multi/rgb.mp4 b/_publications/splatad/videos/multi/rgb.mp4 new file mode 100644 index 0000000..3ba1e92 Binary files /dev/null and b/_publications/splatad/videos/multi/rgb.mp4 differ diff --git a/_publications/splatad/videos/nuScenes/0039.mp4 b/_publications/splatad/videos/nuScenes/0039.mp4 new file mode 100644 index 0000000..b48d38f Binary files /dev/null and b/_publications/splatad/videos/nuScenes/0039.mp4 differ diff --git a/_publications/splatad/videos/nuScenes/0061.mp4 b/_publications/splatad/videos/nuScenes/0061.mp4 new file mode 100644 index 0000000..a69ad65 Binary files /dev/null and b/_publications/splatad/videos/nuScenes/0061.mp4 differ diff --git a/_publications/splatad/videos/nuScenes/0104.mp4 b/_publications/splatad/videos/nuScenes/0104.mp4 new file mode 100644 index 0000000..c8768eb Binary files /dev/null and b/_publications/splatad/videos/nuScenes/0104.mp4 differ diff --git a/_publications/splatad/videos/nuScenes/0193.mp4 b/_publications/splatad/videos/nuScenes/0193.mp4 new file mode 100644 index 0000000..3dd1b6e Binary files /dev/null and b/_publications/splatad/videos/nuScenes/0193.mp4 differ diff --git a/_sass/buttons.scss b/_sass/buttons.scss new file mode 100644 index 0000000..7f5a866 --- /dev/null +++ b/_sass/buttons.scss @@ -0,0 +1,46 @@ + +.button { + background-color: #e4e2e2; + border-color: #ffffff; + border-width: 1px; + color: #363636; + cursor: pointer; + justify-content: center; + padding-bottom: calc(.5em - 1px); + padding-left: 1em; + padding-right: 1em; + padding-top: calc(.5em - 1px); + text-align: center; + white-space: nowrap +} + +.button.is-rounded { + border-radius: 2px; + padding-left: calc(1em + .25em); + padding-right: calc(1em + .25em) +} + +.button.is-medium { + font-size: 1.0rem +} + +.button.is-hovered,.button:hover { + border-color: #b5b5b5; + color: #363636 +} + +.button.active { + border-color: #000000; + color: #363636 +} + +.buttons { + align-items: center; + display: flex; + flex-wrap: wrap; + justify-content: flex-start +} + +.buttons.is-centered { + justify-content: center +} diff --git a/assets/css/main.scss b/assets/css/main.scss index 2d02102..66f33ee 100644 --- a/assets/css/main.scss +++ b/assets/css/main.scss @@ -13,6 +13,7 @@ search: false @import "author_list"; @import "two_image_slider"; @import "publication_cards"; +@import "buttons"; .page .page__inner-wrap { margin-top: 0;