Coherent 3D Scene Diffusion From a Single RGB Image

Manuel Dahnert, Angela Dai, Norman Müller, Matthias Nießner · 2024

Read Coherent 3D Scene Diffusion From a Single RGB Image by Manuel Dahnert, Angela Dai, Norman Müller, Matthias Nießner (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10294.txt.

Introduction

Manuel Dahnert, Angela Dai, and their co-authors rebuild a whole 3D room from a single photograph. Their 2024 paper uses a diffusion model trained on scenes, not single objects. Rebuilding a scene from one view is ill posed: depth, hidden sides, and object layout all must be guessed, and the guesses must agree with each other.

Their model denoises the poses and shapes of every object at once, guided by the photo. Learning from all objects together captures how items relate, a chair near a table, so the result stays coherent. A surface alignment loss helps training when full ground truth is missing, using a shape code that allows direct point sampling. Scores improved by about twelve percent on one benchmark and thirteen on another.

Catalog record

CREATORManuel Dahnert, Angela Dai, Norman Müller, Matthias Nießner
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Coherent 3D Scene Diffusion From a Single RGB Image Manuel Dahnert, Angela Dai, Norman Müller, Matthias Nießner Abstract We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene. Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process. We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions. By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04% improvement in AP 3D \text{AP}_{\text{3D}} on SUN RGB-D and a 13.43% increase in F-Score on Pix3D. 1 Introduction Figure 1: Given a single RGB image of an indoor scene, our model reconstructs the 3D scene by jointly estimating object arrangements and shapes in a globally consistent manner. Our novel diffusion-based 3D scene reconstruction approach achieves highly accurate predictions by utilizing a novel generative scene prior that captures scene context and inter-object relationships, and by employing an efficient surface alignment loss formulation for joint pose- and shape-synthesis. Holistic 3D scene understanding is crucial for various fields and lays the foundation for many downstream tasks in robotics, 3D content creation, and mixed reality. It bridges the gap between 2D perception and 3D understanding. Despite impressive advancements in 2D perception and 3D reconstruction of individual objects [ 56 , 5 , 12 , 38 ] , 3D scene reconstruction from a single RGB observation remains a challenging problem due to its ill-posed nature, heavy occlusions, and the complex multi-object arrangements found in real-world environments. While previous works [ 15 , 32 , 33 ] have shown promising results, they often recover 3D shapes independently and thus do not leverage the scene context nor inter-object relationships. This leads to unrealistic and intersecting object arrangements. Additionally, common feed-forward reconstruction methods [ 48 , 77 , 37 ] struggle with heavy occlusions and weak shape priors, resulting in noisy or incomplete 3D shapes, which hinders immersion and hence limits the applicability in downstream tasks. To address these challenges and to advance 3D scene understanding, we propose a novel generative approach for coherent 3D scene reconstruction from a single RGB image. Specifically, we introduce a new diffusion model that learns a generative scene prior capturing the relationships between objects in terms of arrangement and shapes. When conditioned on a single image, this model simultaneously reconstructs poses and 3D geometries of all scene objects. By framing the reconstruction task as a conditional synthesis process, we achieve significantly more accurate object poses and sharper geometries. Publicly available 3D datasets [ 47 , 62 ] typically only provide partial ground-truth annotations, which complicates joint training of shape and pose. To overcome this, we propose a novel and efficient surface alignment loss formulation \mathcal{L}_{\text{align}} that enables joint training of shape and pose even under the lack of full ground-truth supervision. Unlike previous methods [ 48 , 77 ] that involve costly shape decoding and point sampling on the reconstructed surface, our approach employs an expressive intermediate shape representation that enables direct point sampling from the conditional shape prior. This provides additional supervision and results in more globally consistent 3D scene reconstructions. Our method not only outperforms current state-of-the-art methods by 12.04% in \text{AP}^{\text{15}}_{\text{3D}} on SUN RGB-D [ 62 ] and by 13.43% in F-Score on Pix3D [ 64 ] but also generalizes to other indoor datasets without further fine-tuning. In summary, our contributions include: • A novel diffusion-based 3D scene reconstruction approach that jointly predicts poses and shapes of all visible objects within a scene. • A novel way for modeling a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and inter-object relationships. • An efficient surface alignment loss formulation \mathcal{L}_{\text{align}} that leverages an expressive intermediate shape representation for additional supervision, even in the absence of full ground-truth annotation. 2 Related Works The task of 3D scene reconstruction from a single view combines the fundamental domains of 2D perception and 3D modeling into a unified challenge of holistic 3D understanding. Given the multi-faceted nature of the task, we are providing a comprehensive overview of the relevant research directions and contextualizing our contributions. 2.1 Single-View 3D Reconstruction Object Reconstruction. Since the foundational work by Roberts [ 54 ] , numerous methods have been developed to learn cues for deriving 3D object structures, thereby bridging the gap between 2D perception and the 3D world. These methods typically involve an image encoder network that processes the input image of a single object, capturing its features. The extracted features are either correlated with an encoded shape database to retrieve a suitable shape [ 32 , 33 , 17 ] , or used by a 3D decoder to reconstruct the object in a specific 3D representation, such as voxel grids [ 8 , 72 ] , point clouds [ 14 , 43 ] , meshes [ 70 , 66 ] , or neural fields [ 73 , 27 ] . [ 19 ] uses a message-passing graph network between geometric primitves to reason about the structure of the shape. Scene Reconstruction. Early works formulated single-view scene reconstruction as 3D scene completion from given or estimated depth information [ 63 , 10 , 78 , 9 ] in a volumetric grid. While these methods have produced promising results, their representational power to model fine details is limited by the spatial resolution of the 3D grid. Multi-object reconstruction and scene parsing methods represented objects using primitives [ 13 , 23 ] , voxel grids [ 68 , 35 , 52 ] , or CAD models [ 26 , 24 ] , while also considering the relation between the objects [ 31 ] . The approach presented by Nie et al let@tokeneonedot [ 48 ] is particularly relevant, proposing a holistic method for joint pose and shape estimation from a single image. Zhang et al let@tokeneonedot [ 77 ] extended this idea by incorporating an implicit shape representation and an additional pose refinement using a graph neural network. Although these methods provided significant advances in holistic scene understanding, they struggled with accurate pose estimation and produced noisy scene objects, leading to intersecting or incomplete objects. In contrast to these previous works, we are proposing a generative method to obtain a strong scene prior and formulate the reconstruction task as a conditional synthesis task. This allows for more robust reconstruction that is less prone to object insections or implausible object geometries. 2.2 3D Diffusion Models In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as a versatile class of generative models, demonstrating impressive results in image and video generation. Unlike other classes of generative models such as auto-regressive models [ 46 , 75

Read the complete text