DecompDreamer: Advancing Structured 3D Asset Generation with Multi-Object Decomposition and Gaussian Splatting
Utkarsh Nath, Rajeev Goel, Rahul Khurana, Kyle Min, Mark Ollila, Pavan Turaga, Varun Jampani, Tejaswi Gowda · 2025
Read DecompDreamer: Advancing Structured 3D Asset Generation with Multi-Object Decomposition and Gaussian Splatting by Utkarsh Nath, Rajeev Goel, Rahul Khurana, Kyle Min, Mark Ollila, Pavan Turaga, Varun Jampani, Tejaswi Gowda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11981.txt.
Introduction
Making a 3D scene from text works for one object but tangles when several must sit together, and a 2025 paper by Utkarsh Nath and co-authors rethinks the training order. Under the head's title, the system is framed as a composition-aware curriculum for structured 3D asset generation.
Their system, DecompDreamer, first builds a scaffold that fixes how the objects relate in space. Only then does it polish each object's fine detail. Splitting the two goals in time avoids the clash of gradients that wrecks other methods, where shapes melt together or the run falls apart. Tests on varied multi-object prompts showed cleaner separation, better placement, and higher fidelity than leading methods.
Catalog record
| CREATOR | Utkarsh Nath, Rajeev Goel, Rahul Khurana, Kyle Min, Mark Ollila, Pavan Turaga, Varun Jampani, Tejaswi Gowda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DecompDreamer: Advancing Structured 3D Asset Generation with Multi-Object Decomposition and Gaussian Splatting Utkarsh Nath, Rajeev Goel, Rahul Khurana, Kyle Min, Mark Ollila, Pavan Turaga, Varun Jampani, Tejaswi Gowda Abstract Text-to-3D generation saw dramatic advances in recent years by leveraging Text-to-Image models. However, most existing techniques struggle with compositional prompts, which describe multiple objects and their spatial relationships. They often fail to capture fine-grained inter-object interactions. We introduce DecompDreamer, a Gaussian splatting-based training routine designed to generate high-quality 3D compositions from such complex prompts. DecompDreamer leverages Vision-Language Models (VLMs) to decompose scenes into structured components and their relationships. We propose a progressive optimization strategy that first prioritizes joint relationship modeling before gradually shifting toward targeted object refinement. Our qualitative and quantitative evaluations against state-of-the-art text-to-3D models demonstrate that DecompDreamer effectively generates intricate 3D compositions with superior object disentanglement, offering enhanced control and flexibility in 3D generation. Project page : https://decompdreamer3d.github.io Figure 1 : Illustration of high-quality, complex 3D scenes generated by DecompDreamer for prompts containing 6-11 objects. Existing methods struggle to generate all objects accurately and fail to maintain correct spatial relationships. † † footnotetext: * Equal Contribution. † † footnotetext: \dagger Corresponding author: [email protected] 1 Introduction Figure 2 : (a) Illustration of feature-object mismatch. The image generated by StableDiffusion 2.1 using the prompt ”A blue bird sitting on a black branch of a white tree” (b) 3D representations generated by LucidDreamer [ 20 ] and DecompDreamer. 3D representations play a crucial role in many applications including perception and autonomy, as well as in the media industries, enabling the creation of immersive and realistic digital environments. These industries strive to replicate real-world complexity by incorporating high-fidelity 3D objects that interact dynamically within virtual spaces. Real-world scenes consist of objects and agents with complex spatial relationships and potential interactions, with their surroundings, that can only be described by intricate semantic relationships. Recent methods employ 2D lifting techniques [ 6 , 21 , 24 , 32 , 40 ] to generate 3D models from text using pre-trained diffusion models [ 34 , 29 , 35 ] and Score Distillation Sampling (SDS) [ 32 ] . While effective for single objects with simple text prompts, these methods struggle with compositional prompts involving multiple objects, with quality significantly degrading as inter-object relationships increase. This decline is primarily due to (1) feature-object mismatch and (2) poor modeling of 3D spatial relationships. Feature-object mismatch arises as a fundamental limitation of text-image embeddings [ 33 ] . As text complexity increases, matching attributes to their corresponding objects becomes increasingly challenging. For instance, in the prompt “A blue bird sitting on a black branch of a white tree” (Figure 2 (a)), the text-to-image model confuses the attributes “blue” and “white,” resulting in a bird with blue-white coloration instead of the intended output. Furthermore, as the number of inter-object relationships grows, existing methods fail to maintain spatial coherence, leading to incorrect placements, overlapping artifacts, or missing contextual dependencies. For example, in Figure 2 (b), the generated 3D representation by LucidDreamer [ 20 ] deteriorates as the complexity and number of inter-object relationships increase. While generating a high-fidelity 3D representation corresponding to compositional text prompts is crucial, it is equally important to break the scene down into smaller, well-defined components. A monolithic 3D output limits flexibility, making it difficult to refine individual objects or modify their attributes without affecting the entire scene. A decomposed representation provides greater control, allowing artists and developers to tweak specific parts while preserving overall scene coherence. In this paper, our goal is to generate decomposable 3D representations corresponding to complex text prompts involving intricate inter-object relationships. Let us consider how an artist might approach this problem [ 9 ] . An expert 3D modeler tasked with designing a 3D scene consisting of multiple objects would likely begin by identifying the number of simple objects, estimating their approximate relative positions, and then creating meshes for each object while refining their spatial relationships. Following this simple yet effective strategy, we propose DecompDreamer, a Gaussian-splatting-based optimization routine that decomposes complex 3D representations into smaller components and progressively refines them. Our approach is built on three core design choices: Structured 3D Decomposition and Initialization: Given a text prompt, we use a Vision-Language Model (VLM) to decompose the scene into individual objects, infer their spatial arrangement, and establish inter-object relationships. Each object is represented using 3D Gaussians, enabling structured parameter initialization via coarse 3D point clouds generated by Point-E [ 28 ] . Unlike implicit representations such as NeRF [ 25 ] , which lacks object-level control, this approach ensures a spatially consistent and well-structured starting point. Progressive Optimization: Existing SDS-based methods optimize entire scenes at once [ 6 , 21 , 24 , 32 , 40 , 43 ] or iteratively refine objects, relationships and full scene [ 13 , 14 ] ; often leading to entangled representations and inconsistencies. While effective for simple objects or those with few inter-object relationships, their performance degrades significantly as the number of relationships increases. In this paper, we propose a progressive optimization strategy inspired by artistic workflows. Unlike existing methods, we do not primarily focus on optimizing the entire scene at once. Instead, we emphasize modeling inter-object relationships and refining individual objects, with occasional scene-level optimization. We start with joint optimization of objects with respect to their relationships, then gradually shift focus toward individual object refinement. Optimizing objects in a compositional environment is significantly more challenging than in isolation as it requires accounting for spatial dependencies and relative viewing direction. To address this, we incorporate object-view-aware optimization, improving object disentanglement and ensuring a coherent 3D structure. Once the scene is well-formed, we apply targeted optimization within the relational context to refine fine-grained object details while preserving spatial coherence. Flow Matching Distillation: To effectively model objects and their relationships, we extend diffusion-based SDS to flow-based SDS, leveraging rectified flow models [ 1 , 2 ] for smoother and more controlled optimization. Flow matching enables stable and efficient optimization by learning a direct vector field, eliminating stochastic noise and improving convergence. With these advancements, DecompDreamer is capable of generating high-quality compositional 3D objects (Figure 1 , 4 ) while achieving enhanced disentanglement for individual objects (Figure 5 ). We evaluate DecompDreamer through both qualitative and quantitative analyses. A user study indicates that DecompDreamer outperforms existing models, including Gala3D [ 43 ] , GraphDreamer [ 13 ] , LucidDreamer [ 20 ] , MVDream [ 36 ] , and HiFA [ 45 ] , in terms of image quality and semantic alignment. Additionally, we perform quantitative evaluations using CLIP score [ 33 ] , and Pick-A-Pic [ 19 ] on 30 compositional text prompts containing 2 to 11 objects, demonstrating DecompDreamer’s superior performance in compositional 3D generation. Our contributions can be summarized as: 1. We introduce DecompDreamer, a 3D Gaussian-based training routine that decomposes complex 3D representations into smaller, manageable parts and applies progressive optimization to refine relationships and individual objects, using a novel flow matching distillation approach. 2. DecompDreamer achieves strong disentanglement across individual object components using view-aware optimization, enhancing interpretability and controllability. 3. We extensively validate our approach using a diverse set of complex text prompts and benchmark it both qualitatively and quantitatively against