ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar

Clément Jambon, Changwoon Choi, Dongsu Zhang, Olga Sorkine-Hornung, Young Min Kim · 2024

Read ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar by Clément Jambon, Changwoon Choi, Dongsu Zhang, Olga Sorkine-Hornung, Young Min Kim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16253.txt.

Introduction

Clement Jambon and co-authors present ExCellGen in a 2024 paper on generating realistic three dimensional scenes from a single real example. Large, high quality three dimensional data are scarce, and manual modeling needs expert skill. Slow cycles of revision can stifle creative work.

Their pipeline first rebuilds the example, such as handheld or drone footage, as a high quality appearance model. A generative cellular model then produces a sparse scene volume, and a patch step fills appearance from the original. Training takes minutes per example and generation seconds. An interactive interface gives users full control.

Catalog record

CREATORClément Jambon, Changwoon Choi, Dongsu Zhang, Olga Sorkine-Hornung, Young Min Kim
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar Clément Jambon, Changwoon Choi, Dongsu Zhang, Olga Sorkine-Hornung, Young Min Kim Abstract Photorealistic 3D scene generation is challenging due to the scarcity of large-scale, high-quality real-world 3D datasets and complex workflows requiring specialized expertise for manual modeling. These constraints often result in slow iteration cycles, where each modification demands substantial effort, ultimately stifling creativity. We propose a fast, exemplar-driven framework for generating 3D scenes from a single casual input, such as handheld video or drone footage. Our method first leverages 3D Gaussian Splatting (3DGS) to robustly reconstruct input scenes with a high-quality 3D appearance model. We then train a per-scene Generative Cellular Automaton (GCA) to produce a sparse volume of featurized voxels, effectively amortizing scene generation while enabling controllability. A subsequent patch-based remapping step composites the complete scene from the exemplar’s initial 3D Gaussian splats, successfully recovering the appearance statistics of the input scene. The entire pipeline can be trained in less than 10 minutes for each exemplar and generates scenes in 0.5-2 seconds. Our method enables interactive creation with full user control, and we showcase complex 3D generation results from real-world exemplars within a self-contained interactive GUI. Figure 1: We propose ExCellGen, a framework for fast, controllable, and photorealistic 3D scene generation from real-world exemplars. We convert casual video inputs (e.g., drone or mobile footage) into a high-quality 3D representation using 3D Gaussian splats. Within our GUI editor, users can extract regions from which they wish to generate variations. Our two-stage amortized generation strategy first uses a per-scene Generative Cellular Automaton (GCA) [ 75 ] to produce a sparse volume of featurized voxels, then composites the final appearance via sparse patch-based remapping from the original scene. Given coarse conditioning voxels, users can synthesize new variations in just seconds in a fully interactive way. Keywords: Interactive Method, Generative Cellular Automata, 3D Gaussian Splatting, Scene Generation, Patch-based Synthesis 1 Introduction Generating realistic 3D scenes is key to many applications in 3D content creation, visual effects, robotics [ 72 ] , and autonomous driving [ 43 ] . Generative modeling has recently demonstrated remarkable progress in creating photorealistic images, largely owing to the tremendous data and development of deep learning network architectures. In contrast, achieving realism in 3D scene generation faces several fundamental challenges. Unlike images or photos, 3D models are limited in scale and quality. Available 3D datasets are often synthetic, and composed of object-centric meshes with simple textured appearance [ 13 ] . In contrast, existing real-world datasets are mostly confined to specific distributions, including indoor scenes [ 8 ] , urban footage [ 33 ] or object-centric captures [ 52 , 36 , 17 ] . This falls short of the diversity found in real-world scenes, rich in intricate geometry and natural elements like vegetation, which are notoriously difficult to capture with meshes or represent using conventional 3D assets. In this work, we propose to formulate the task of 3D scene generation as an exemplar-based synthesis task, building on a rich line of research in texture synthesis [ 15 , 18 ] . We target real-world captures in the form of casual video footage. All of our input exemplars are derived from footage recorded using smartphones or drones, demonstrating the practicality of our approach. To reconstruct 3D scenes, we build on the recent success of 3D Gaussian Splatting (3DGS) [ 28 ] . Gaussian splats enable fast retrieval and rendering and lead to a photorealistic appearance. They accurately reflect the user’s intent by faithfully reproducing the images captured by the user, providing unambiguous control in selecting the ingredients of scene generation. Moreover, 3D Gaussians are explicit representations that can be directly manipulated, transported, and composited. Previous approaches for exemplar-based synthesis demonstrate results on clean synthetic or pre-processed scenes, and typically support only mesh-like or SDF-based exemplars [ 38 , 67 , 66 ] . Even when supporting radiance fields, methods such as Sin3DGen [ 32 ] still require a mesh to guide generation. This requirement limits applicability to real-world scenes, where appearance may be fuzzy in some regions. Moreover, casually captured scenes often contain complex backgrounds and surrounding objects with unclear segmentation boundaries. We thus design intuitive means to select a desired region from the input exemplar scene by distilling and clustering semantic feature maps from large vision models [ 7 ] . This provides both control and additional guidance during generation. Synthesizing 3D scenes is often memory and computationally intensive, as it requires inferring 3D geometry, leading to slow, iterative processes (e.g., 1-3 minutes for Sin3DGen). We propose a fast and controllable two-stage generation strategy that enables interactive feedback within seconds. First, we generate a sparse voxel volume that captures the mesoscale structure of the scene, abstracting away fine-grained details and amortizing the generation task. Then, we reconstruct high-frequency appearance by compositing 3D Gaussians from the exemplar, guided by the generated voxels. More precisely, in the first stage, we adapt Generative Cellular Automata [ 75 ] (GCA) to controllable scene generation to efficiently generate a sparse 3D volume of voxels. The generative kernels of GCA are trained from a single exemplar and are highly efficient in both training (less than 10 minutes) and generation (0.5-2 s). This neural approach offers two additional advantages. First, contrary to cascaded generation strategies [ 32 ] , our generative network is naturally trained to reflect conditional signals, which we materialize as coarse voxels that describe the location and shape of the desired output scene. Second, neural approaches generalize better across various distributions without the need for cumbersome parameter tuning. In the second stage, we recover the final appearance of the scene by introducing a novel sparse and efficient patch-based consistency step, which results in smooth transitions between voxels. The final scene is converted into 3DGS with high-frequency details by transporting and compositing 3D Gaussians from the exemplar scene according to the generated voxels. We demonstrate the practicality and capabilities of our method on a variety of real-world scenes captured using smartphones or drones, and manipulated directly within our interactive GUI editor. In summary, our contributions are as follows: (a) We propose an efficient 3D scene generation technique in which exemplar scenes are directly captured from casual real-world videos; (b) Our method is controllable, inviting user controls in capturing, selecting, and compositing the scene; (c) We formulate a lightweight generative model from a single exemplar, allowing scene synthesis at interactive rates (approximately 0.5-2 s); (d) We provide a fully self-contained GUI to demonstrate the practicality of the method. The source code is available at https://github.com/clementjambon/excellgen . Figure 2: Overview of our pipeline. Training phase. (left) From a casual video, an exemplar scene is reconstructed as 3D Gaussians augmented with DINO features. After selecting the region of interest within an interactive editor, the 3D Gaussians are converted into a sparse volume of voxels. A per-scene Generative Cellular Automaton (GCA) is then efficiently trained on random crops of the scenes in under 10 minutes. Generation phase. (right) A set of coarse conditioning voxels, either copied from parts of the exemplar or manually edited, is provided to the pre-trained GCA. The GCA generates a novel sparse volume of featurized voxels. These voxels are then

Read the complete text