Beyond Voxel 3D Editing : Learning from 3D Masks and Self-Constructed Data

Yizhao Xu, Hongyuan Zhu, Caiyun Liu, Tianfu Wang, Keyu Chen, Sicheng Xu, Jiaolong Yang, Nicholas Jing Yuan, Qi Zhang, [0.5em] The Peking University HKUST(GZ) Microsoft AI Microsoft Research · 2026

Read Beyond Voxel 3D Editing : Learning from 3D Masks and Self-Constructed Data by Yizhao Xu, Hongyuan Zhu, Caiyun Liu, Tianfu Wang, Keyu Chen, Sicheng Xu, Jiaolong Yang, Nicholas Jing Yuan, Qi Zhang, [0.5em] The Peking University HKUST(GZ) Microsoft AI Microsoft Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13688.txt.

Introduction

Xu and eight colleagues move 3D editing beyond voxels. Their BVE framework learns from 3D masks and a self-constructed large dataset built for editing. Lightweight trainable modules inject textual meaning into a base image-to-3D model without expensive full retraining. An annotation-free masking strategy preserves local invariance, so untouched regions stay faithful to the original. Tests show high-quality, text-aligned assets that retain the input's visual character.

Catalog record

CREATORYizhao Xu, Hongyuan Zhu, Caiyun Liu, Tianfu Wang, Keyu Chen, Sicheng Xu, Jiaolong Yang, Nicholas Jing Yuan, Qi Zhang, [0.5em] The Peking University HKUST(GZ) Microsoft AI Microsoft Research
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41528
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Voxel 3D Editing : Learning from 3D Masks and Self-Constructed Data Yizhao Xu, Hongyuan Zhu, Caiyun Liu, Tianfu Wang, Keyu Chen, Sicheng Xu, Jiaolong Yang, Nicholas Jing Yuan, Qi Zhang, [0.5em] The Peking University HKUST(GZ) Microsoft AI Microsoft Research Abstract 3D editing refers to the ability to apply local or global modifications to 3D assets. Effective 3D editing requires maintaining semantic consistency by performing localized changes according to prompts, while also preserving local invariance so that unchanged regions remain consistent with the original. However, existing approaches have significant limitations: multi-view editing methods incur losses when projecting back to 3D, while voxel-based editing is constrained in both the regions that can be modified and the scale of modifications. Moreover, the lack of sufficiently large editing datasets for training and evaluation remains a challenge. To address these challenges, we propose a Beyond Voxel 3D Editing (BVE) framework with a self-constructed large-scale dataset specifically tailored for 3D editing. Building upon this dataset, our model enhances a foundational image-to-3D generative architecture with lightweight, trainable modules, enabling efficient injection of textual semantics without the need for expensive full-model retraining. Furthermore, we introduce an annotation-free 3D masking strategy to preserve local invariance, maintaining the integrity of unchanged regions during editing. Extensive experiments demonstrate that BVE achieves superior performance in generating high-quality, text-aligned 3D assets, while faithfully retaining the visual characteristics of the original input. Figure 1 : Fast and versatile 3D editing results. Our method supports both global (e.g., add operation, left) and local (e.g., replacement or style change operation, right) edits in seconds. It yields high-quality assets with detailed meshes for geometry and vivid 3D Gaussians/Radiance Fields for appearance. $\star$ $\star$ footnotetext: Work done during internship at Microsoft AI. $\dagger$ $\dagger$ footnotetext: Corresponding author. 1 Introduction Generative artificial intelligence is revolutionizing 3D content creation, fundamentally reshaping workflows across various domains. 3D generation is widely applied across creative domains such as 3D printing 63 , gaming 35 , virtual reality (VR), and augmented reality (AR). Recently, a surge of research 21 ; 10 ; 31 ; 81 ; 92 ; 39 and development in 3D generation has made it possible to produce high-resolution 3D assets with physically based rendering (PBR) 70 ; 23 ; 83 ; 11 materials. Users typically create 3D assets from image or text prompts using generative tools. However, it is still challenging to generate high-quality assets in a single attempt — the results often require refinement in professional modeling software such as Blender or 3Ds Max to meet production standards. To better address user needs and enhance the flexibility of 3D generation pipelines, recent studies have begun exploring 3D editing capabilities, enabling more efficient and controllable asset creation workflows. Current text-driven 3D editing approaches primarily fall into two categories, each with inherent trade-offs. The first, leveraging per-instance optimization like Score Distillation Sampling (SDS) 54 ; 8 ; 15 , iteratively refines a 3D asset to match the target prompt. While effective, this process is computationally prohibitive, rendering it impractical for interactive use. To enhance efficiency, a second category of methods pursues an indirect strategy: editing 2D rendered views and then lifting these changes back into 3D space 1 ; 6 ; 5 . However, this approach often struggles with cross-view consistency, leading to geometric artifacts and textural inconsistencies when fusing the edits into a coherent 3D model 43 ; 51 ; 7 . More recently, direct manipulation of 3D-native representations, such as voxel-level features in VoxHammer 36 , has emerged as a promising direction. Yet, these methods still face the critical challenge of achieving precise local edits while preserving the global structure and semantic integrity of unchanged regions. Indeed, 3D editing still faces significant challenges: • Limited datasets: There is a lack of sufficient 3D editing datasets for both training and evaluation. • Semantic consistency: Effective 3D editing requires making localized changes according to prompts while preserving the semantic and structural integrity of unchanged regions. • Flexible editing: Methods need to support both global and local editing of 3D assets. To this end, we propose Beyond Voxel 3D Editing (BVE), an efficient framework that achieves high-quality, text-driven 3D editing while faithfully preserving the original asset’s identity. Our approach integrates lightweight modules into a pretrained generative model and is validated on a large-scale benchmark we developed specifically for this task. In summary, our contributions are as follows: • We construct and will release Edit-3DVerse, the first large-scale, high-quality dataset for text-driven 3D editing, purpose-built to benchmark research in this domain. • We propose the BVE framework, which enables high-fidelity 3D editing via lightweight, trainable modules, eliminating the need for costly full-model retraining. • We design a novel annotation-free 3D masking strategy that ensures semantic and structural consistency by precisely preserving unedited regions. • We demonstrate through extensive experiments that our method significantly outperforms state-of-the-art approaches in both editing quality and identity preservation. Figure 2 : Dataset Construction pipeline and Sub-Task Distribution 2 Related Work 2.1 3D Generative Models Recent advancements in diffusion models 25 ; 64 and the availability of high-quality 3D datasets 13 ; 12 have significantly accelerated the development of 3D generative modeling 9 ; 5 ; 14 ; 19 ; 22 ; 27 ; 32 ; 38 ; 39 ; 44 ; 45 ; 46 ; 47 ; 50 ; 60 ; 66 ; 69 ; 71 ; 72 ; 74 ; 76 ; 77 ; 78 ; 80 ; 84 ; 85 ; 87 ; 90 . These methods can generally be categorized into two major paradigms. One approach involves generating 3D models by first synthesizing multi-view images and then reconstructing the 3D from these views 28 ; 30 ; 46 ; 47 ; 56 ; 57 . However, inconsistent multi-view synthesis may lower the quality of the final 3D model. Alternatively, a series of methods focuses on training native 3D generative models 38 ; 41 ; 40 ; 42 ; 77 ; 78 ; 79 ; 91 ; 93 . These often comprise a variational autoencoder 34 and a diffusion transformer (DiT) 53 for denoising in latent space. This approach effectively unifies 3D generation with high fidelity and consistency, laying the foundation for downstream inversion and editing. 2.2 3D Editing Compared to traditional image editing tasks 89 ; 33 ; 62 , 3D editing remains relatively underexplored. This is partly due to the limited availability of high-quality base models and 3D-related datasets. Recently, with advancements in 3D generation models, several works on 3D editing have begun to emerge. For example, work based on Score Distillation Sampling (SDS) 54 often requires substantial computational resources to achieve effective editing. In contrast, more recent research 1 ; 6 ; 5 ; 2 ; 4 ; 16 ; 37 ; 94 explores multi-view image editing as an indirect approach to modifying 3D assets. This strategy edits 3D content by altering its associated images, but it can introduce noise due to inconsistencies when projecting multiple views back onto the 3D asset. Recently, TRELLIS proposes a two-stage framework using SS and Slat structures for 3D asset generation. Additionally, several training-free methods 86 ; 36 have been introduced that edit voxel-level features directly. For instance, Voxhammer 36 merges the inverted sampling of the unmasked

Read the complete text