We introduce StyleGaussian, a novel 3D style transfer technique that allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps). Leveraging 3D Gaussian Splatting (3DGS), StyleGaussian achieves style transfer without compromising its real-time rendering ability and multi-view consistency. It achieves instant style transfer with three steps: embedding, transfer, and decoding. Initially, 2D VGG scene features are embedded into reconstructed 3D Gaussians. Next, the embedded features are transformed according to a reference style image. Finally, the transformed features are decoded into the stylized RGB. StyleGaussian has two novel designs. The first is an efficient feature rendering strategy that first renders low-dimensional features and then maps them into high-dimensional features while embedding VGG features. It cuts the memory consumption significantly and enables 3DGS to render the high-dimensional memory-intensive features. The second is a K-nearest-neighbor-based 3D CNN. Working as the decoder for the stylized features, it eliminates the 2D CNN operations that compromise strict multi-view consistency. Extensive experiments show that StyleGaussian achieves instant 3D stylization with superior stylization quality while preserving real-time rendering and strict multi-view consistency.
我们介绍了 StyleGaussian，一种新颖的 3D 风格转换技术，它能够以每秒 10 帧（fps）的速度将任何图像的风格瞬间转移到 3D 场景中。利用 3D 高斯喷溅（3DGS），StyleGaussian 在不影响其实时渲染能力和多视图一致性的前提下实现了风格转换。它通过三个步骤实现瞬间风格转换：嵌入、转换和解码。最初，2D VGG 场景特征被嵌入到重构的 3D 高斯中。接下来，嵌入的特征根据参考风格图像进行变换。最后，变换后的特征被解码成风格化的 RGB。StyleGaussian 有两个新颖的设计。第一个是一个高效的特征渲染策略，它首先渲染低维特征，然后在嵌入 VGG 特征时将它们映射到高维特征。这显著减少了内存消耗，并使 3DGS 能够渲染高维的、内存密集型的特征。第二个是基于 K 最近邻的 3D CNN。作为风格化特征的解码器，它消除了破坏严格多视图一致性的 2D CNN 操作。广泛的实验表明，StyleGaussian 在保持实时渲染和严格多视图一致性的同时，实现了瞬间 3D 风格化，并具有优越的风格化质量。