A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision

Chensheng Peng, Ido Sobol, Masayoshi Tomizuka, Kurt Keutzer, Chenfeng Xu, Or Litany · 2024

Read A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision by Chensheng Peng, Ido Sobol, Masayoshi Tomizuka, Kurt Keutzer, Chenfeng Xu, Or Litany (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00623.txt.

Introduction

Building a 3D scene from a single image is guesswork, since many shapes fit the same picture. Generative models suit the task, but they usually demand 3D training data that barely exists. Chensheng Peng and co-authors present in 2024 a way to train a 3D diffusion model with only 2D supervision. Their trick decouples the two sides of training: the noisy samples being cleaned up live in 3D, while the supervision signal stays in 2D.

A deterministic image-to-3D model plays teacher, producing imperfect 3D guesses that become the noisy inputs the diffusion model learns to refine. Tested on object and scene datasets with two different Gaussian-splat teachers, the student consistently beat its teachers. The authors argue the approach makes 3D generative modeling scalable and sharper without any full 3D ground truth.

Catalog record

CREATORChensheng Peng, Ido Sobol, Masayoshi Tomizuka, Kurt Keutzer, Chenfeng Xu, Or Litany
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00623

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision Chensheng Peng, Ido Sobol, Masayoshi Tomizuka, Kurt Keutzer, Chenfeng Xu, Or Litany Abstract We present a novel framework for training 3D image-conditioned diffusion models using only 2D supervision. Recovering 3D structure from 2D images is inherently ill-posed due to the ambiguity of possible reconstructions, making generative models a natural choice. However, most existing 3D generative models rely on full 3D supervision, which is impractical due to the scarcity of large-scale 3D datasets. To address this, we propose leveraging sparse-view supervision as a scalable alternative. While recent reconstruction models use sparse-view supervision with differentiable rendering to lift 2D images to 3D, they are predominantly deterministic, failing to capture the diverse set of plausible solutions and producing blurry predictions in uncertain regions. A key challenge in training 3D diffusion models with 2D supervision is that the standard training paradigm requires both the denoising process and supervision to be in the same modality. We address this by decoupling the noisy samples being denoised from the supervision signal, allowing the former to remain in 3D while the latter is provided in 2D. Our approach leverages suboptimal predictions from a deterministic image-to-3D model—acting as a "teacher"—to generate noisy 3D inputs, enabling effective 3D diffusion training without requiring full 3D ground truth. We validate our framework on both object-level and scene-level datasets, using two different 3D Gaussian Splat (3DGS) teachers. Our results show that our approach consistently improves upon these deterministic teachers, demonstrating its effectiveness in scalable and high-fidelity 3D generative modeling. See our project page https://lesson-in-splats.github.io/ . Figure 1 : (Left) Standard diffusion training is constrained to same-modality supervision. We break this barrier by decoupling the sources of noised samples and supervision. Leveraging imperfect predictions of a feedforward 3D reconstruction module, our method offers a fully image-based 3D diffusion training scheme. (Right) When paired with two different noisy teachers, our diffusion model enhances reconstruction quality and 3D geometry across both objects and scenes. Notably, our model is trained on the same data as the teachers, and uses a smaller model size. "Medium" and "Large" denote the model size, see Sec. 4.1 . 1 Introduction 3D Reconstruction is essential for computer vision applications, such as augmented reality, robotics, and autonomous driving [ 30 , 13 , 8 , 42 ] , which rely on inferring 3D structures from limited viewpoints. However, reconstructing 3D objects or scenes from 2D images is challenging. First, it is an ill-posed problem because different 3D shapes can produce identical 2D projections. Second, 3D datasets are scarce, especially in comparison to their image dataset counterparts, limiting the ability to directly train on 3D data. Current approaches for 3D reconstruction from single images can be categorized into two main types: deterministic predictions and generative models, each with distinct limitations. A prevalent approach in 3D reconstruction is to use deterministic feedforward neural networks to map input images to 3D representations, such as Neural Radiance Fields (NeRF) [ 37 , 19 ] and 3D Gaussian Splats (3DGS) [ 23 , 55 , 54 , 70 ] . Leveraging differentiable rendering techniques, these methods can be trained directly from sprase 2D views , circumventing the need for large volumes of 3D data. This is advantageous because 3D data is often difficult or impractical to obtain, especially for real-world applications. However, despite ongoing performance improvements, deterministic models remain inherently limited by the ambiguity in the 2D-to-3D mapping. These models cannot fully capture the range of possible 3D structures that correspond to a source image, leading to overly smooth or blurred outputs when supervised by appearance-based losses. In contrast, diffusion models [ 17 , 10 ] , have recently shown a strong potential in generating 3D data. 3D Diffusion models are trained to progressively denoise corrupted versions of 3D data to generate 3D outputs that are likely under the training set distribution, either by directly operating in the 3D space [ 1 , 33 , 38 , 34 , 72 ] or in a higher-dimensional latent space [ 47 , 58 , 46 ] . However, diffusion models for 3D generation face a fundamental limitation due to their training process, in which the denoiser – which operates in 3D – is trained on noisy samples using their clean counterparts as supervision. This requirement demands a substantial amount of 3D data, making these models difficult to scale to real-world applications where 3D data is limited. Some attempts have been made to bypass these limitations by training 3D generative models using multi-view images [ 53 , 66 ] . These models aggregate information across multiple views, structuring predictions in 3D space. However, such methods rely on the bijectivity of multi-view and 3D representations, which only holds for a substantial number of images limiting their applicability. When the number of views is limited, they often fall short in generation quality. Thus, although both deterministic and generative models have made strides in 3D reconstruction, the field lacks scalable, high-performance solutions that can infer 3D structures from single 2D images. Research into training 3D diffusion models using only 2D supervision remains underexplored, highlighting an important gap that our work aims to address. In this work, we propose a novel training strategy that fundamentally revises the principles of diffusion model training by decoupling the denoised modality (3D) from the supervision modality (2D). This stands in contrast to traditional diffusion training, which requires the noisy and clean signals to remain in the same modality—here, in 3D. Our solution leverages deterministic 3D reconstruction methods as “noisy teachers”. While deterministic 3D predictions are imperfect and exhibit artifacts, we show that they nonetheless can generate useful 3D samples as input to the denoiser. Specifically, by introducing noise beyond a critical timestep t^{*} , the noisy 3D signal provided by the deterministic model nearly matches that of the (unavailable) true 3D structure. This “sweet spot” in noise level is inspired by techniques like SDEdit [ 36 ] . Once denoised, the predicted clean 3D structure can be rendered and supervised with reference images, alleviating the need for 3D supervision. However, this alone is not sufficient because if the denoiser only learns from timesteps t>t^{*} , it is bound to produce blurry outputs, thus not able to fully exploit fine-grained details in images. To overcome this, we introduce a second key innovation: a multi-step denoising strategy that replaces the traditional single-step denoising framework. Specifically, starting from a noise level t>t^{*} , our model performs iterative denoising, akin to its behavior during inference, progressively reducing noise over multiple steps until reaching its sharpest 3D estimate at t=0 . Rendering this output, enables supervising the model with ground truth images, effectively propagating gradients across lower time steps t\leq t^{*} to adapt the denoiser to generate high quality reconstructions. In summary, by leveraging 2D supervised deterministic teachers, and multi-step denoising, our method offers a fully image-based 3D diffusion training scheme. Notably, this strategy is flexible and can utilize various teacher models. In our experiments, we demonstrate this flexibility using two types of deterministic models: Splatter Image [ 55 ] and Flash3D [ 54 ] . With these models, we train on single object and scene data, respectively. In both

Read the complete text