OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion
Yunhan Yang, Yufan Zhou, Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying-Tian Liu, Hao Xu, Ding Liang, Yan-Pei Cao, Xihui Liu · 2025
Read OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion by Yunhan Yang, Yufan Zhou, Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying-Tian Liu, Hao Xu, Ding Liang, Yan-Pei Cao, Xihui Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06165.txt.
Introduction
Yunhan Yang and nine co-authors generate 3D objects that come apart into named, editable parts, in a 2025 paper. Most generators output one solid lump, which game and design tools cannot easily reshape.
Their framework, OmniPart, splits the job in two. A planner first lays out a row of 3D boxes, one per part, steered by flat masks a user draws. A second model then builds all parts at once inside that layout, keeping them joined into one object. Users can choose how fine the split runs. Tests report state-of-the-art quality for part-aware generation. The parts stay semantically separate yet structurally sound. Editable output matters for animation, manufacturing, and virtual worlds alike.
Catalog record
| CREATOR | Yunhan Yang, Yufan Zhou, Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying-Tian Liu, Hao Xu, Ding Liang, Yan-Pei Cao, Xihui Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06165
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion Yunhan Yang, Yufan Zhou, Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying-Tian Liu, Hao Xu, Ding Liang, Yan-Pei Cao, Xihui Liu Abstract . The creation of 3D assets with explicit, editable part structures is crucial for advancing interactive applications, yet most generative methods produce only monolithic shapes, limiting their utility. We introduce OmniPart, a novel framework for part-aware 3D object generation designed to achieve high semantic decoupling among components while maintaining robust structural cohesion. OmniPart uniquely decouples this complex task into two synergistic stages: (1) an autoregressive structure planning module generates a controllable, variable-length sequence of 3D part bounding boxes, critically guided by flexible 2D part masks that allow for intuitive control over part decomposition without requiring direct correspondences or semantic labels; and (2) a spatially-conditioned rectified flow model, efficiently adapted from a pre-trained holistic 3D generator, synthesizes all 3D parts simultaneously and consistently within the planned layout. Our approach supports user-defined part granularity, precise localization, and enables diverse downstream applications. Extensive experiments demonstrate that OmniPart achieves state-of-the-art performance, paving the way for more interpretable, editable, and versatile 3D content. Project page: https://omnipart.github.io/ . Figure 1. OmniPart: Generating Complex 3D Objects as Compositions of Controllable Parts. From simple 2D images (shown framed) and mask inputs, OmniPart first plans a 3D part structure using an autoregressive model, then synthesizes all high-quality, textured parts simultaneously (individual components within transparent displays). These seamlessly merge into a coherent object, offering explicit part control for enhanced editing, customization, and animation. 1. Introduction The creation of rich, interactive 3D worlds is fundamental to modern visual computing, driving applications from immersive gaming and virtual reality to digital twins and robotic interaction. A persistent challenge is the generation of 3D assets that are not merely static, monolithic forms. While recent generative models produce impressive holistic 3D shapes ( Poole et al. 2023 ; Zhang et al. 2023 ; Zhang et al. 2024b ; Li et al. 2025 ; Xiang et al. 2024 ) , these often lack the intrinsic part-based structure inherent to real-world objects. This structural opaqueness limits their direct use for essential tasks such as compositional editing, procedural animation, material assignment, and semantic understanding—capabilities crucial for artists, developers, and downstream systems. Part-aware 3D generation ( Chen et al. 2024b ; Yang et al. 2025b ; Liu et al. 2024a ) , which explicitly models objects as assemblies of semantically meaningful components, offers a path to overcome these limitations. However, creating such structured assets requires a careful balance: achieving low semantic coupling (parts are distinct and independently addressable) while ensuring high structural cohesion (parts form a plausible, integrated whole). Prior work has often struggled here. Approaches reconstructing from 2D part segmentations ( Liu et al. 2024a ; Chen et al. 2024b ) suffer from geometric inconsistencies and insufficient detail fidelity, primarily due to challenges in view aggregation and the inherent limitations of 2D models in capturing complete 3D part geometry. Conversely, existing methods for direct 3D part generation ( Li et al. 2024b ) may lack robust, fine-grained control over part decomposition or rely on extensive, scarce 3D part-level annotations, hindering their scalability and practical use. We emphasize that robust and versatile part-aware 3D generation hinges on a principled decoupling of high-level structural planning from detailed part synthesis, unified by strong conditioning mechanisms . We introduce OmniPart , a novel framework that implements this principle through a synergistic two-stage pipeline. Initially, a Controllable Structure Planning module addresses the crucial task of defining the part-level spatial layout. It autoregressively generates a variable-length sequence of 3D bounding boxes, each representing a distinct part’s 3D location. A key challenge here is compositional ambiguity (e.g., limbs vs. a composite torso), which can lead to unpredictability. OmniPart resolves this by conditioning the planning on intuitive, flexible 2D part masks (obtainable from user input or 2D models like SAM ( Ravi et al. 2024 ) ). These masks, which delineate desired regions for part decomposition without imposing strict one-to-one correspondences or requiring explicit semantic labels , guide the transformer-based autoregressive model to produce 3D bounding boxes reflecting the intended part structure and accommodate varying part counts. To further enhance the accuracy of this structural plan, we introduce a novel Part Coverage Loss , ensuring that each predicted bounding box comprehensively encloses its corresponding object part. Subsequently, building on this planned structure, a Spatially-Conditioned Part Synthesis module efficiently generates consistent and high-quality 3D representations for all object parts. Given the scarcity of extensive part-level 3D annotations, our approach effectively adapts a powerful, pre-trained holistic 3D generator (TRELLIS ( Xiang et al. 2024 ) ) based on rectified flow into a part-aware 3D generation model. The predicted bounding boxes from the first stage define spatial regions within TRELLIS’s voxel representation, serving as initializations for part-wise latent codes. To promote semantic awareness and overall structural coherence, each part’s voxel representation is considered alongside the full-object context, and distinctive part-aware embeddings are injected. All part latents are then jointly refined via a denoising process that operates on both global and local information, ensuring consistency. Since voxels only approximate the coarse geometry of objects and parts, overlaps frequently occur at the boundaries between adjacent parts. The voxel initialization from the bounding boxes of the first stage may introduce noise. To address this, we introduce a novel voxel discarding mechanism , enabling precise indication of whether a voxel actually belongs to its assigned part, which aids in creating clean interfaces and allows for the efficient, simultaneous generation of all parts . These fine-tuning strategies yield detailed, coherent, and generalizable part-level 3D outputs despite limited part-specific supervision. OmniPart enables the generation of 3D objects with explicit, controllable, and semantically meaningful part structures. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in part-aware 3D generation and facilitates a diverse range of downstream applications, including fine-grained compositional editing, material assignment, and animation support. Collectively, OmniPart significantly advances the creation of more interpretable and editable 3D content. Our core contributions are: • A novel two-stage generative formulation and framework that strategically decouples part structure planning from part geometry synthesis, achieving superior controllability, part coherence, and overall quality. • An autoregressive controllable structure planning module guided by flexible 2D part masks, providing effective control over variable part granularity and 3D decomposition. • A spatially-conditioned part synthesis module that generates all 3D parts simultaneously and consistently, conditioned on the planned spatial layout, efficiently leveraging powerful pre-trained holistic models for high-fidelity part-aware generation with limited part-specific supervision. 2. Related Work 2.1. 3D Shape Generation DreamFusion ( Poole et al. 2023 ) introduces score distillation sampling to optimize 3D scenes from 2D diffusion models ( Rombach et al. 2022 ) , enabling text-driven 3D generation without 3D supervision. A complementary line of work ( Liu et al. 2024b ; Long et al. 2024 ; Shi et al. 2023 ; Liu et al. 2024c ; Yang et al. 2024b ; Yang et al. 2025a ; Xu et al. 2024 ; Qi et al. 2024 ; Zou et al. 2024 ) generates multi-view images using 2D diffusion models and reconstructs 3D geometry via multi-view consistency algorithms. However, these 2D-driven pipelines often suffer from geometric artifacts due to inherent view