Native 3D Editing with Full Attention
Weiwei Cai, Shuangkang Fang, Weicai Ye, Xin Dong, Yunhan Yang, Xuanyang Zhang, Wei Cheng, Yanpei Cao, Gang Yu, Tao Chen · 2025
Read Native 3D Editing with Full Attention by Weiwei Cai, Shuangkang Fang, Weicai Ye, Xin Dong, Yunhan Yang, Xuanyang Zhang, Wei Cheng, Yanpei Cao, Gang Yu, Tao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17501.txt.
Introduction
Weiwei Cai and nine co-authors present a framework for editing three-dimensional objects by instruction, working natively in 3D rather than through pictures. Editing 3D content by text command could widen access to 3D creation, but optimization-based methods are prohibitively slow. Methods that edit multiple 2D views and lift the results back often produce inconsistent geometry. This system manipulates the 3D representation directly in a single feed-forward pass.
To train it, the team built a large multimodal dataset of addition, deletion, and modification edits, curated so changes follow the instruction while untouched regions stay faithful to the source. They compared two ways of conditioning the model and found a 3D token concatenation strategy both cheaper in parameters and better in performance. Evaluations show the method beating 2D-lifting approaches on quality, consistency, and instruction fidelity.
Catalog record
| CREATOR | Weiwei Cai, Shuangkang Fang, Weicai Ye, Xin Dong, Yunhan Yang, Xuanyang Zhang, Wei Cheng, Yanpei Cao, Gang Yu, Tao Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17501
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Native 3D Editing with Full Attention Weiwei Cai, Shuangkang Fang, Weicai Ye, Xin Dong, Yunhan Yang, Xuanyang Zhang, Wei Cheng, Yanpei Cao, Gang Yu, Tao Chen Abstract Instruction-guided 3D editing is a rapidly emerging field with the potential to broaden access to 3D content creation. However, existing methods face critical limitations: optimization-based approaches are prohibitively slow, while feed-forward approaches relying on multi-view 2D editing often suffer from inconsistent geometry and degraded visual quality. To address these issues, we propose a novel native 3D editing framework that directly manipulates 3D representations in a single, efficient feed-forward pass. Specifically, we create a large-scale, multi-modal dataset for instruction-guided 3D editing, covering diverse addition, deletion, and modification tasks. This dataset is meticulously curated to ensure that edited objects faithfully adhere to the instructional changes while preserving the consistency of unedited regions with the source object. Building upon this dataset, we explore two distinct conditioning strategies for our model: a conventional cross-attention mechanism and a novel 3D token concatenation approach. Our results demonstrate that token concatenation is more parameter-efficient and achieves superior performance. Extensive evaluations show that our method outperforms existing 2D-lifting approaches, setting a new benchmark in generation quality, 3D consistency, and instruction fidelity. 2 2 footnotetext: ∗ Project Leader. 🖂 Corresponding Authors. 1 Introduction 3D creative editing has great potential in numerous fields, including film production, entertainment industry, and digital gaming. Recent advancements ( Erkoç et al. 2025 ; Barda et al. 2025 ; Li et al. 2025a ; Fang et al. 2024 ) , such as text-to-3D editing and image-to-3D editing, have significantly reduced the manual effort required by professional 3D artists while making 3D asset editing accessible to non-professionals. These approaches typically focus on score distillation sampling (SDS) ( Poole et al. 2022 ) to edit 2D images and lift them into 3D assets. While generating high-quality 3D objects, these optimization-based approaches are a lengthy and computationally intensive process. To address this limitation, recent studies ( Erkoç et al. 2025 ; Barda et al. 2025 ) reconstruct the edited 3D assets via a feed-forward 3D reconstruction model from single or sparse images. These methods edit the images using a multi-view diffusion model to keep the consistency of edited images. However, the paradigm of multi-view editing and then restructuring the 3D model also falls short in terms of either visual quality or 3D consistency due to its reliance on 2D space for editing. To address these challenges, we propose a native 3D editing paradigm that directly manipulates 3D objects based on textual instructions. A primary obstacle to this goal is the absence of a large-scale, high-quality dataset for instruction-guided 3D editing. To overcome this, we introduce a systematic data construction pipeline to create a comprehensive benchmark, as illustrated in Fig. 1 . Specifically, for the deletion task, we curate part-level 3D assets from the Objaverse dataset ( Deitke et al. 2023 ) to serve as source objects. By programmatically removing distinct parts from each asset, we generate the corresponding target objects, forming a large collection of source-target pairs for deletion edits. To obtain the guiding instructions, we leverage a powerful multimodal large language model (MLLM), Gemini 2.5 ( Comanici et al. 2025 ) , to generate descriptive text that accurately reflects the transformation from the source to the target object. For addition and modification tasks, we build upon existing open-source 2D editing datasets ( Ye et al. 2025 ) that provide paired source/target images, along with corresponding instructions. We employ the image-to-3D generation model, Hunyuan3D 2.1 ( Hunyuan3D et al. 2025 ) , to lift these 2D image pairs into high-fidelity 3D source and target objects, while retaining their original instructional texts. However, we observe that the generated 3D source and target objects often exhibit inconsistencies in geometry and appearance due to the inherent limitations of current image-to-3D generation models. To ensure data quality, we implement a rigorous manual curation process, selectively retaining only those pairs where the generated source and target objects demonstrate strong geometric and visual consistency. This two-pronged approach, combining procedural generation with careful curation, allows us to construct a novel and diverse 3D editing dataset, paving the way for data-driven native 3D editing. Building on this dataset, we further explore effective architectural designs for instruction-guided native 3D editing. Our investigation focuses on two distinct strategies for conditioning the model on the source 3D object. The first strategy employs a classic adapter-based approach ( Ye et al. 2023 ) , integrating the source object’s features into the network via cross-attention mechanisms. The second involves a more direct and parameter-efficient method: 3D token concatenation. Specifically, our model learns the editing transformation during training by processing a concatenated sequence of the source object’s and noised target object’s 3D tokens, guided by the corresponding instruction. This allows the full-attention network to directly model the relationship between the source, target, and instruction with minimal additional parameters. Once trained, the model can perform editing at inference time by taking only the source object and the editing instruction as input to generate the desired 3D assets within 20 seconds. We are the first to explore token concatenation as a conditioning method in the 3D domain, and our experimental results demonstrate that this strategy significantly outperforms the cross-attention approach, yielding superior results in terms of both generation quality and fidelity to the instruction. In summary, our contributions are as follows. • We introduce the first large-scale, multi-modal benchmark dataset specifically designed for instruction-guided native 3D editing. Our comprehensive dataset, encompassing a wide range of addition, deletion, and modification tasks, is systematically constructed by leveraging both part-level 3D assets and by lifting existing 2D edit datasets into the 3D domain. • We propose a novel and parameter-efficient architectural design for native 3D editing. We are the first to demonstrate in the 3D domain that conditioning via direct token concatenation significantly outperforms traditional cross-attention mechanisms, achieving higher fidelity to instructions and superior generation quality with minimal additional parameters. • We present an effective and efficient feed-forward framework for native 3D editing. Our approach directly manipulates 3D representations, bypassing the quality and consistency issues of prior multi-view editing pipelines. 2 Related Work 3D Reconstruction and Generation with Large Models. Stable Diffusion ( Poole et al. 2022 ) pioneered 3D generation work based on Score Distillation Sampling (SDS), and subsequently, numerous outstanding studies ( Xu et al. 2023 ; Lin et al. 2023 ; Melas-Kyriazi et al. 2023 ) have leveraged SDS optimization and its variant ( Chung et al. 2023 ; Hertz et al. 2023 ) for 3D reconstruction and generation. These methods yielded high-quality 3D generation but were often slow and impractical due to their reliance on per-case optimization. Meanwhile, the generative results are limited by the ability of the pre-trained 2D generative models. To mitigate these issues, the Large Reconstruction Model (LRM) ( Hong et al. 2023 ) proposes a feed-forward model trained on large-scale datasets to generate a NeRF from single images within 5 seconds rapidly. Subsequently, LGM ( Tang et al. 2024 ) trained a Large Multi-view Gaussian Model to reconstruct 3D Gaussians from multiview images. GRM ( Xu et al. 2024b ) introduces a feedforward transformer-based model to generate 3d assets from sparse-view images.