Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion CCS: Computing methodologies Point-based models CCS: Computing methodologies Shape analysis CCS: Computing methodologies Mesh geometry models
Wang Zhao, Yan-Pei Cao, Jiale Xu, Yuejiang Dong, Ying Shan · 2025
Read Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion CCS: Computing methodologies Point-based models CCS: Computing methodologies Shape analysis CCS: Computing methodologies Mesh geometry models by Wang Zhao, Yan-Pei Cao, Jiale Xu, Yuejiang Dong, Ying Shan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17074.txt.
Introduction
Wang Zhao and co-authors present a system that assembles 3D objects from parts, in a 2025 paper. Given part meshes and a reference image, the system must place each part to rebuild the object. Older methods predict one pose per part and train per category, which breaks on real objects with differing part counts and shapes.
Their system, Assembler, treats assembly as generation: a diffusion model samples arrangements, which handles symmetry, repeated parts, and cases with several valid answers. Parts are represented as sparse clouds of anchor points, so generation happens in space rather than in pose space, and scales better. Training uses a new data set of over three hundred and twenty thousand part-object assemblies, built and filtered from existing shape collections. The system tops prior results on a standard benchmark and is the first to assemble complex real-world objects well, the authors report.
Catalog record
| CREATOR | Wang Zhao, Yan-Pei Cao, Jiale Xu, Yuejiang Dong, Ying Shan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion CCS: Computing methodologies Point-based models CCS: Computing methodologies Shape analysis CCS: Computing methodologies Mesh geometry models Wang Zhao, Yan-Pei Cao, Jiale Xu, Yuejiang Dong, Ying Shan Abstract . We present Assembler, a scalable and generalizable framework for 3D part assembly that reconstructs complete objects from input part meshes and a reference image. Unlike prior approaches that mostly rely on deterministic part pose prediction and category-specific training, Assembler is designed to handle diverse, in-the-wild objects with varying part counts, geometries, and structures. It addresses the core challenges of scaling to general 3D part assembly through innovations in task formulation, representation, and data. First, Assembler casts part assembly as a generative problem and employs diffusion models to sample plausible configurations, effectively capturing ambiguities arising from symmetry, repeated parts, and multiple valid assemblies. Second, we introduce a novel shape-centric representation based on sparse anchor point clouds, enabling scalable generation in Euclidean space rather than SE(3) pose prediction. Third, we construct a large-scale dataset of over 320K diverse part-object assemblies using a synthesis and filtering pipeline built on existing 3D shape repositories. Assembler achieves state-of-the-art performance on PartNet and is the first to demonstrate high-quality assembly for complex, real-world objects. Based on Assembler, we further introduce an interesting part-aware 3D modeling system that generates high-resolution, editable objects from images, demonstrating potential for interactive and compositional design. Project page: https://assembler3d.github.io Figure 1. 3D part assemblies of general objects by Assembler. Parts are labeled in different colors. 1. Introduction 3D part assembly is a fundamental task in computer vision and graphics, aiming at composing a complete object from a set of modular parts. This capability is increasingly critical across a wide range of applications, including 3D content creation, computer-aided design (CAD), manufacturing, and robotics. A robust, generalizable assembly system could greatly enhance 3D modeling workflows and unlock new possibilities for interactive and automated design. Despite its importance, automatic 3D part assembly remains a highly challenging problem. It requires a comprehensive understanding of part geometry and semantics, the ability to reason about inter-part relationships, and the capacity to imagine plausible complete object shapes. Some works ( Huang et al. 2006 ; Sellán et al. 2022 ; Lu et al. 2023 ; Xu et al. 2025a ) have explored the 3D fracture assembly problem, which focus primarily on low-level geometric cues—e.g., boundary curves and local correspondences—to reassemble broken objects. In contrast, 3D part assembly typically assumes that each part is a complete, semantically meaningful unit and often permits duplicated components. These characteristics introduce unique challenges, demanding higher-level reasoning about object structure, part functionality, and overall semantic coherence. Over the past decades, researchers have explored a variety of approaches to tackle the 3D part assembly problem. Early works addressed part assembly by developing efficient retrieval and registration based on curated part libraries ( Funkhouser et al. 2004 ) , or constructing graphical models ( Chaudhuri et al. 2011 ; Kalogerakis et al. 2012 ) to capture part semantics and relational constraints. While significantly reducing the manual efforts, they often relied on pre-defined part taxonomies or databases, limiting their scalability and generalization to novel or unstructured settings. With the rise of deep learning, recent works ( Zhan et al. 2020 ) have proposed to encode part features using neural networks, learning to predict the 6-DoF poses of parts to assemble. Following this direction, subsequent methods have introduced more powerful architectures ( Zhang et al. 2022 ; Zhang et al. 2024a ; Xu et al. 2025b ) , incorporated richer input information ( Narayan et al. 2022 ; Li et al. 2020 ) , and adopted hierarchical structures ( Du et al. 2024 ) to improve assembly accuracy. Although these models have achieved promising results—particularly on canonical object categories such as chairs, tables, and lamps from the PartNet dataset ( Mo et al. 2019 ) —they remain largely constrained to category-specific scenarios with relatively simple, well-aligned parts. How to scale these methods to handle general, in-the-wild objects with diverse geometries, part counts, and structural variations remains an open challenge in the field. In this work, we present Assembler, a scalable and effective framework for 3D part assembly of general objects. Given a set of part meshes and a reference image, Assembler produces accurate, high-fidelity assemblies that generalize across varying numbers of parts, intricate shapes, and diverse object categories. It comprehensively addresses the above scalability challenges of general 3D part assembly through three key innovations. First, we formulate 3D part assembly as a generative task, and leverage diffusion models to learn the distribution of plausible assemblies. A specially-designed diffusion model is introduced to effectively extract rich, aligned features from input conditions. By modeling as a generation task rather than the deterministic prediction, Assembler naturally handles the inherent ambiguities of part assembly—such as symmetry, duplicate components, and multiple valid configurations—thus better supporting generalization to in-the-wild objects. This shares similar motivations with recent works, which explored diffusion models for 2D ( Scarpellini et al. 2024 ) and fracture ( Sellán et al. 2022 ) assembly, and a score-based generative model for category-specific part assembly ( Cheng et al. 2023 ) . Second, we propose a novel representation for part assemblies based on sparse anchor point clouds. Existing methods typically predict the 6-DoF poses of parts in SE(3). However, this parametric representation is highly abstract and geometric that lacks explicit assembled shape information. Furthermore, the probabilistic distribution of poses is difficult for generative modeling due to its discontinuities and multimodal distribution, as observed in ( Leach et al. 2022 ; Yao et al. 2025 ) . In constrast, in Assembler, we innovatively introduce sparse anchor point clouds as the assembly representation, and formulate the part assembly as a point cloud generation problem. Each input part is sampled as a set of sparse anchor points, and the model generates an assembled anchor point cloud representing the final object shape. Part poses can be easily recovered by simple least-squares fitting. This shape-centric formulation enables the diffusion model to learn smooth and scalable distributions of the assembled object point clouds in Euclidean space, whose effectiveness is well validated in prior 3D point cloud generation methods ( Nichol et al. 2022 ; Lan et al. 2025 ) . Third, we address the lack of large-scale data for general 3D part assembly by proposing a data synthesis and filtering pipeline. Leveraging the disconnected-component property of existing large-scale 3D shape datasets, we generate diverse part-object pairs through segmentation and filtering. We collect and construct a dataset of over 320,000 high-quality, diverse object assemblies, providing the necessary scale and diversity to train a generalizable assembly model. With these innovations, Assembler achieves, for the first time, reasonable 3D part assembly of general objects conditioned on a single image. Our method achieves state-of-the-art performance on the PartNet benchmark and demonstrates strong generalization to novel, complex objects. Figure 1 shows some 3D part assembly examples of general objects. Furthermore, based on Assembler, we explore a new interesting 3D modeling system, which features part-aware high-resolution 3D object generation from a single image, and may inspire future research on high-quality part-based modeling. Our contributions are summarized as follows: 1) We formulate 3D part assembly as a generative task