MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs

Xianglong He, Junyi Chen, Di Huang, Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, Yangguang Li · 2025

Read MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs by Xianglong He, Junyi Chen, Di Huang, Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, Yangguang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23022.txt.

Introduction

Xianglong He and co-authors introduce in 2025 a faster, more controllable way to generate 3D meshes with AI. Artists want meshes with clean topology, ready to use. Past methods grow a mesh token by token, which is slow and leaves the face count to chance.

Their system instead diffuses points in continuous space and turns them into triangle faces, using a flow based transformer design. Continuous diffusion sidesteps the token bottleneck, so generation speeds up. Artists also gain a dial over how many faces the mesh gets. A tiger with too many faces lags in games, while too few looks crude. Controllable counts let one model serve both film and game pipelines. The team reports quality on par with slower rivals.

Catalog record

CREATORXianglong He, Junyi Chen, Di Huang, Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, Yangguang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs Xianglong He, Junyi Chen, Di Huang, Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, Yangguang Li Abstract In the domain of 3D content creation, achieving optimal mesh topology through AI models has long been a pursuit for 3D artists. Previous methods, such as MeshGPT, have explored the generation of ready-to-use 3D objects via mesh auto-regressive techniques. While these methods produce visually impressive results, their reliance on token-by-token predictions in the auto-regressive process leads to several significant limitations. These include extremely slow generation speeds and an uncontrollable number of mesh faces. In this paper, we introduce MeshCraft, a novel framework for efficient and controllable mesh generation, which leverages continuous spatial diffusion to generate discrete triangle faces. Specifically, MeshCraft consists of two core components: 1) a transformer-based VAE that encodes raw meshes into continuous face-level tokens and decodes them back to the original meshes, and 2) a flow-based diffusion transformer conditioned on the number of faces, enabling the generation of high-quality 3D meshes with a predefined number of faces. By utilizing the diffusion model for the simultaneous generation of the entire mesh topology, MeshCraft achieves high-fidelity mesh generation at significantly faster speeds compared to auto-regressive methods. Specifically, MeshCraft can generate an 800-face mesh in just 3.2 seconds—35 × \times faster than existing baselines. Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art techniques in both qualitative and quantitative evaluations on ShapeNet dataset and demonstrates superior performance on Objaverse dataset. Moreover, it integrates seamlessly with existing conditional guidance strategies, showcasing its potential to relieve artists from the time-consuming manual work involved in mesh creation. Figure 1 : \mathbf{Overview} of generated meshes, speed and token numbers of MeshCraft. † † footnotetext: {\dagger} Corresponding authors. 1 Introduction With advances in fields such as gaming and 3D printing, the creation of high-quality, topologically sound 3D meshes has become increasingly important. However, generating well-structured 3D meshes from simple inputs—such as textual descriptions or a single 2D image—requires significant artistic expertise and often entails labor-intensive manual processes. Therefore, the ability to rapidly generate 3D meshes with high-quality topology that meet the needs of artists and professionals using AI models is a key goal. Significant efforts have been made to automate the generation of 3D meshes. Many works Poole et al. 2022 ; Xu et al. 2024 are modeling with dense triangle meshes, which are extracted from neural fields using iso-surfacing methods Mildenhall et al. 2021 ; Park et al. 2019 ; Kerbl et al. 2023 . Specifically, these methods first generate intermediate 3D representations and then post-process them into final meshes. Although these approaches yield impressive visualizations in producing neural representations, the resulting meshes often suffer from excessive face counts and artifacts due to misalignment that occur during the conversion of intermediate representations into final meshes via re-meshing techniques Lorensen and Cline 1998 . The efficiency of these processes is also limited by the post-processing step. To address these issues, another group of works Nash et al. 2020 ; Siddiqui et al. 2024 ; Alliegro et al. 2023 ; Hao et al. 2024 , modeling with triangle meshes that accurately reflect the compactness of artists’ design, offers great flexibility for manipulation and efficiency for storage. This approach, referred to as ”native mesh generation,” focuses on explicitly modeling mesh distributions. It first transforms the mesh into latent sequences by leveraging the spatial relationships among faces, vertices, and coordinates. These methods then directly generate the sequences, and even shows superior long sequence scalability. However, these approaches are often limited by generation effectiveness or are validated only on small datasets, such as ShapeNet Chang et al. 2015 . Furthermore, most of the methods lack user controllability, which is essential for the 3D creation process. To address these challenges, we introduce MeshCraft, an efficient and controllable approach for high-fidelity mesh generation method that leverages the strengths of continuous diffusion transformers with rectified flow (see Fig. 1 (b) and (c) for comparisons with prior works). Unlike auto-regressive methods such as Meshtron Hao et al. 2024 , which prioritize scalability, we explore the possibilities of different routes starting from the underlying principles of implementing native mesh generation, with the goal of demonstrating the potential of diffusion models for the practical usage of controllable and efficient mesh generation, rather than immediate scalability. MeshCraft saves token numbers for at most 9 times, and speed up for 35 times. It is designed as a two-stage pipeline that consists of directly modeling the mesh distribution in a latent space via a Variational Auto-Encoder(VAE) Kingma 2013 and then generating meshes with a diffusion transformer. Unlike previous works that compress meshes into discrete tokens, we explore the modeling of meshes in a low-dimensional and continuous latent space. Face features of the mesh are sent into the VAE encoder, regularized with KL divergence, and are subsequently decoded into vertex coordinates of each face. For the generative model, the diffusion transformer is modified to generate the continuous tokens with varying lengths with inspirations from recent advances in image generation Esser et al. 2024 ; Lu et al. 2024 . This approach enables MeshCraft to generate meshes at high speed. The guidance and masking strategy also allow users to control the number of faces of generated meshes in a user-friendly manner. We validate the effectiveness of MeshCraft on the ShapeNet dataset Chang et al. 2015 and demonstrate its great potential as a generic mesh generator on Objaverse Deitke et al. 2023 , which includes a diverse array of 3D objects across various categories. Extensive experiments illustrate the effectiveness, and controllability of MeshCraft from both qualitative and quantitative aspects. In summary, our contributions are as follows: • We propose a transformer-based VAE that encodes discrete triangle meshes into a continuous latent space and decodes them back to the original mesh, achieving competitive reconstruction performance compared to state-of-the-art mesh auto-encoders based on vector quantization. • We introduce a flow-based transformer diffusion model conditioned on the number of faces, integrating classifier-free guidance for both image inputs and face number to enable effective control over the mesh generation process. • MeshCraft significantly outperforms existing methods, achieving new state-of-the-art results on the ShapeNet dataset while being 35 \times faster than MeshGPT, and demonstrating the effectiveness of face number control on the large-scale Objaverse dataset. 2 Related works Recent advances Nash et al. 2020 ; Alliegro et al. 2023 ; Siddiqui et al. 2024 ; Weng et al. 2024 ; Chen et al. 2024b ; Hao et al. 2024 have seen pioneering efforts to generate meshes directly, with many employing auto-regressive models for this task. For instance, PolyGen Nash et al. 2020 utilizes two transformers to separately learn vertex and face distributions. MeshGPT Siddiqui et al. 2024 first encodes meshes into face-level quantized tokens via GNN-based Vector Quantization Variational Auto-Encoder(VQ-VAE) Van Den Oord et al. 2017 , followed by the application of GPT-style transformers for auto-regressive generation. MeshXL Chen et al. 2024b introduces another sequential mesh representation for one-stage auto-regressive generation. Even though Meshtron Hao et al. 2024 shows the scalability of auto-regressive generations, however, the generative capabilities of these models are restricted by the rapidly increasing number of tokens, leading to slow inference speeds. Additionally, users cannot precisely assign an

Read the complete text