3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework
Tobias Sautter, Jan-Niklas Dihlmann, Hendrik Lensch · 2025
Read 3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework by Tobias Sautter, Jan-Niklas Dihlmann, Hendrik Lensch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17459.txt.
Introduction
Tobias Sautter, Jan-Niklas Dihlmann, and Hendrik P A Lensch present 3D-RE-GEN, a system that turns one photo of a room into an editable 3D scene. Film and game artists need scenes they can move and relight, yet past single-photo methods fused objects wrongly, misplaced them, or dropped the background entirely.
Their pipeline chains specialist models: one spots each object, one rebuilds it in 3D, and one sets it back in place. Hidden parts of objects are filled in by an image editor with scene-wide reasoning, and a generated backdrop locks in lighting and scale. A fine alignment step seats each object on the floor plane. The result tops prior scores and stays fully editable, the trait artists asked for.
Catalog record
| CREATOR | Tobias Sautter, Jan-Niklas Dihlmann, Hendrik Lensch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework Tobias Sautter, Jan-Niklas Dihlmann, Hendrik Lensch Abstract Recent advances in 3D scene generation produce visually appealing output, but current representations hinder artists’ workflows that require modifiable 3D textured mesh scenes for visual effects and game development. Despite significant advances, current textured mesh scene reconstruction methods are far from artist ready, suffering from incorrect object decomposition, inaccurate spatial relationships, and missing backgrounds. We present 3D-RE-GEN, a compositional framework that reconstructs a single image into textured 3D objects and a background. We show that combining state of the art models from specific domains achieves state of the art scene reconstruction performance, addressing artists’ requirements. Our reconstruction pipeline integrates models for asset detection, reconstruction, and placement, pushing certain models beyond their originally intended domains. Obtaining occluded objects is treated as an image editing task with generative models to infer and reconstruct with scene level reasoning under consistent lighting and geometry. Unlike current methods, 3D-RE-GEN generates a comprehensive background that spatially constrains objects during optimization and provides a foundation for realistic lighting and simulation tasks in visual effects and games. To obtain physically realistic layouts, we employ a novel 4-DoF differentiable optimization that aligns reconstructed objects with the estimated ground plane. 3D-RE-GEN achieves state of the art performance in single image 3D scene reconstruction, producing coherent, modifiable scenes through compositional generation guided by precise camera recovery and spatial optimization. https://3dregen.jdihlmann.com/ Figure 1 : 3D-RE-GEN in action : A single image is decomposed into a clean background and a complete 3D scene with individual 3D objects, creating a production-ready asset for immediate use in VFX and games. Project page: https://3dregen.jdihlmann.com/ 1 Introduction The demand for immersive 3D content is a cornerstone of modern creative industries, most notably in Visual Effects (VFX) and game development. Traditionally, the creation of 3D scenes is a significant production bottleneck, requiring highly specialized artists with extensive experience. To construct a single scene, an environment artist must first model and texture the background. Then, every individual asset within that scene, from chairs to tables to small props, must be independently modeled and textured. Finally, a layout artist must meticulously assemble all these components into a coherent whole. Even for highly skilled professionals, this workflow is exceptionally time consuming and costly, scaling exponentially with scene complexity. Recent generative models for single image to 3D (e.g., 6 ; 46 ) offer the potential to accelerate asset creation and democratize 3D modeling. However, while these models are effective at generating isolated objects, they generally fail when applied to complex, multi object scenes. Reconstructing a 3D scene from a single 2D image is an inherently ill-posed problem. A 2D projection lacks true depth information, and objects mutually occlude one another, resulting in incomplete structural data. Furthermore, accurately inferring the spatial positioning of objects to produce a credible layout remains a significant challenge, as the 2D image provides no explicit information about object specific distances, scale, or contact points. Previous research has addressed this issue through various approaches. Early holistic methods attempted to reconstruct entire scenes in a single pass 19 ; 44 , while retrieval based methods matched image objects to databases of 3D models 15 ; 22 . More recently, compositional and diffusion based methods have become prevalent 20 ; 45 ; 3 ; 32 ; 31 . Despite a strong emphasis on spatial relationships between objects, these methods share common critical gaps: the background environment is frequently neglected, and there is often no mechanism to enforce physical plausibility, such as aligning objects to a common ground plane. This paper introduces 3D-RE-GEN, a framework designed to address this gap and generate production ready 3D scenes from a single image. The modular pipeline leverages large scale pretrained models and incorporates a novel four degree of freedom (4-DoF) constrained optimization with differentiable rendering. Our method estimates precise camera poses and generates an accurate textured background mesh; creates high quality 3D assets using a novel context aware inpainting technique; and aligns objects to the background through a robust optimization strategy, alternating between a five degree of freedom (5-DoF) model for free floating objects and the new 4-DoF model for ground aligned objects. Although 3D-RE-GEN is developed with artists in mind, it demonstrates competitive performance and achieves state of the art results on comparative academic benchmarks. By automating the most laborious parts of scene reconstruction, 3D-RE-GEN allows artists to move from a single concept image to a fully editable 3D environment in minutes, not days, drastically accelerating the creative workflow and minimizing needed experience. 2 Related Works Figure 2 : 3D-RE-GEN Framework Overview. Our framework converts a single image into a complete 3D scene. First, we segment the image: these masks provide the 2D silhouette loss, while our novel Application-Querying (A-Q) model generates clean, inpainted object images for 3D meshing. In parallel, the input and a generated ”empty room” image are used to extract the camera and scene point cloud, which is masked to create the 3D geometric loss. Finally, our Scene Positioning model assembles the 3D assets and background by minimizing both losses, using a novel 4-DOF constrained workflow to ensure all ground based objects are physically aligned to the floor. Reconstructing 3D scenes from single 2D images is a fundamental challenge in computer vision, as it requires synthesizing complex geometric structures and spatial relationships from limited visual data 10 ; 25 ; 43 ; 29 ; 42 . In recent years, this task has attracted considerable research interest, with various approaches proposed to address the ambiguities and occlusions inherent in single-view observations 30 ; 16 ; 8 ; 33 ; 40 ; 41 ; 18 ; 20 . The field has progressed from early methods that utilized explicit geometric priors to contemporary approaches that employ deep learning and generative models 3 ; 20 ; 38 . This section reviews major developments, with particular emphasis on object level reconstruction 2.1 , scene composition strategies 2.2 , and diffusion-based techniques 2.3 that have emerged as promising solutions. 2.1 Monocular Object-Level 3D Reconstruction Recent advances in 3D representation learning have facilitated substantial progress in reconstructing individual objects from single images 6 ; 27 ; 46 . To extract the correct object from an image, models often leverage pretrained 2D object detectors or segmentation networks 28 ; 36 . Early approaches primarily utilized supervised learning frameworks that mapped image features to 3D geometry, often employing explicit representations such as point clouds or meshes 16 ; 19 . These methods generally required extensive 3D supervision and exhibited limited generalization to novel object categories. The introduction of diffusion models has transformed the field, enabling the generation of high fidelity 3D shapes with enhanced generalization 39 ; 47 . Diffusion based methods denoise latent representations of 3D geometry conditioned on image inputs, capturing complex shape priors without explicit 3D supervision. The availability of large scale 3D datasets has further accelerated progress, supporting the training of models capable of generating diverse and detailed 3D objects from single view observations 14 ; 13 ; 12 . 2.2 Scene Composition Strategies Scene reconstruction introduces challenges beyond those encountered in object level reconstruction, as it necessitates modeling spatial relationships among multiple objects. Existing approaches are generally categorized