Skip to content

Commit

Permalink
correct
Browse files Browse the repository at this point in the history
  • Loading branch information
范丞德 committed Oct 19, 2024
1 parent fa67ad2 commit 938e14d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion spectromotion/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ <h3>
</h3>
<image src="img/overview_combined.png" width=100% style="display: block; margin: auto;"></image>
<p class="text-justify">
(a) We optimize a NeRF to minimize a reconstruction loss \(\mathcal{L}_\mathrm{recon}\) between renderings and a few input images, as well as a sample loss \(\mathcal{L}_\mathrm{sample}\) between a rendering from a random pose and an image predicted by a diffusion model for that pose. (b) To generate the sample image, we use a PixelNeRF-style model to fuse information from the input images and to render a predicted feature map corresponding to the sample view camera pose. (c) This feature map is concatenated with the noisy latent (computed from the current NeRF rendering at that pose) and is provided to a diffusion model, which additionally uses CLIP embeddings of the input images via cross-attention. The resulting decoded output sample is used to enforce an image-space loss on the corresponding NeRF rendering (\(\mathcal{L}_\mathrm{sample}\)).
Our method stabilizes the scene geometry through three stages. In the static stage, we stabilize the geometry of the static scene by minimizing photometric loss \(\mathcal{L}_{\text{color}}\) between vanilla 3DGS renders and ground truth images. The dynamic stage combines canonical 3D Gaussians \(\textbf{G}\) with a deformable Gaussian MLP to model dynamic scenes while simultaneously minimizing normal loss \(\mathcal{L}_{\text{normal}}\) between rendered normal map \(\mathbf{N}^t\) and gradient normal map from depth map (\{\mathbf{D}^t}\), thus further enhancing the overall scene geometry. Finally, the specular stage introduces a deformable reflection MLP to handle changing environment lighting, deforming reflection directions \(\omega^t_r\) to query a canonical environment map for specular color \(\mathbf{c}_s^t\). It is then combined with diffuse color \(\mathbf{c_d}\) (using zero-order spherical harmonics) and learnable specular tint \(\mathbf{s_\mathbf{tint}}\) per 3D Gaussian to obtain the final color \(\mathbf{c}_\mathbf{final}^t\). This approach enables the modeling of dynamic specular scenes and high-quality novel view rendering.
</p>
</div>
</div><br><br>
Expand Down

0 comments on commit 938e14d

Please sign in to comment.