Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation
Jiantao Lin, Xin Yang, Meixi Chen, Yingjie Xu, Dongyu Yan, Leyi Wu, Xinli Xu, Lie Xu, Shunsi Zhang, Ying-Cong Chen · 2025
Read Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation by Jiantao Lin, Xin Yang, Meixi Chen, Yingjie Xu, Dongyu Yan, Leyi Wu, Xinli Xu, Lie Xu, Shunsi Zhang, Ying-Cong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01370.txt.
Introduction
Jiantao Lin and co-authors turn image generators into 3D asset makers with a deliberately simple trick in a 2025 paper. Three-dimensional generation lags behind images because large 3D training collections are hard to gather. Their framework fine-tunes a 2D diffusion model instead of training a 3D model from scratch.
The model learns to produce a bundle image: a tile of multi-view renders plus matching normal maps. Normals rebuild the mesh, and the views supply its texture. The same model supports 3D editing and mesh or texture enhancement, borrowing techniques from the image world. Extensive tests show high-quality assets made efficiently. The authors frame the lesson as keeping 3D generation simple by reusing what image models already know.
Catalog record
| CREATOR | Jiantao Lin, Xin Yang, Meixi Chen, Yingjie Xu, Dongyu Yan, Leyi Wu, Xinli Xu, Lie Xu, Shunsi Zhang, Ying-Cong Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation Jiantao Lin, Xin Yang, Meixi Chen, Yingjie Xu, Dongyu Yan, Leyi Wu, Xinli Xu, Lie Xu, Shunsi Zhang, Ying-Cong Chen Abstract † † footnotetext: * Equal contribution † \dagger Corresponding author Diffusion models have achieved great success in generating 2D images. However, the quality and generalizability of 3D content generation remain limited. State-of-the-art methods often require large-scale 3D assets for training, which are challenging to collect. In this work, we introduce Kiss3DGen ( K eep I t S imple and S traightforward in 3D Gen eration), an efficient framework for generating, editing, and enhancing 3D objects by repurposing a well-trained 2D image diffusion model for 3D generation. Specifically, we fine-tune a diffusion model to generate ”3D Bundle Image”, a tiled representation composed of multi-view images and their corresponding normal maps. The normal maps are then used to reconstruct a 3D mesh, and the multi-view images provide texture mapping, resulting in a complete 3D model. This simple method effectively transforms the 3D generation problem into a 2D image generation task, maximizing the utilization of knowledge in pretrained diffusion models. Furthermore, we demonstrate that our Kiss3DGen model is compatible with various diffusion model techniques, enabling advanced features such as 3D editing, mesh and texture enhancement, etc. Through extensive experiments, we demonstrate the effectiveness of our approach, showcasing its ability to produce high-quality 3D models efficiently. Project page: https://ltt-o.github.io/Kiss3dgen.github.io . Figure 1 : A 3D Harry Potter scene built with Kiss3DGen. Our proposed framework, KISS3DGen, is a unified 3D generation framework that facilitates various 3D generation tasks, including text-to-3D, image-to-3D, 3D enhancement, editing and more. Specifically, most of the assets in the figure is generated from text (captioned with abbreviated text prompts ) or image (marked by dash lines ) conditions, while the main characters ( Hermoine , Ron and Potter ) are created using a hybrid pipeline that combines image-to-3D and text-guided mesh editing. Please zoom in for details and refer to our main paper for a more introduction. 1 Introduction † † footnotetext: This project is supported by Quwan Technology. In recent years, generative models have significantly advanced the field of 3D object generation, transforming computer vision and graphics. The rapid progress of diffusion models led to remarkable improvements in synthesizing highly realistic 2D images. These developments have opened new possibilities for creating complex 3D objects, essential for applications such as virtual reality, gaming, and scientific simulations. However, extending these advancements from 2D to 3D remains challenging due to the inherent complexity of 3D geometry, the scarcity of high-quality datasets, and the high computational costs involved. Existing 3D generation methods generally fall into two categories: optimization-based approaches and direct generation approaches. Optimization-based methods, such as DreamFusion [ 37 ] , ProlificDreamer [ 52 ] and LucidDreamer [ 23 ] , utilize pre-trained 2D diffusion models to create 3D content. While these methods have shown promising results, they are often time-consuming during inference time because of the intensive iterative optimization required to update the 3D representation. In contrast, direct 3D generation aims to create 3D models directly, without extensive optimization. Typical approaches include InstantMesh [ 59 ] , Unique3D [ 55 ] , PRM [ 10 ] , Clay [ 63 ] , Craftsman [ 21 ] , and Direct3D [ 56 ] . These methods are favored in practical 3D generation systems due to their fast generation speed. However, despite their potential, these techniques rely heavily on large-scale 3D datasets, which often suffer from limitations in quality and availability. For example, Objaverse-XL [ 7 ] , the largest 3D dataset available, contains approximately 10 million samples, but around 70% of the data is compromised by missing textures, low resolution, or poor aesthetics. This lack of high-quality data significantly hampers the effectiveness of these direct 3D generation methods. In contrast, 2D datasets such as LAION-5B [ 43 ] contain billions of high-quality images, underscoring the disparity in data availability between 2D and 3D content. Recent research has indicated that 2D diffusion models inherently possess powerful 3D priors that can be utilized for 3D understanding and generation [ 12 , 17 ] . However, these methods focus on producing 2.5D representations like depth or normal maps, which provide a partial view of the 3D structure and insufficient for full 3D generation. This raises an important question: Can we leverage the expressive priors learned by 2D diffusion models to create complete 3D representations? In this work, we introduce Kiss3DGen, a simple yet effective framework for 3D asset generation. Our approach involves fine-tuning a powerful diffusion transformer model (DiT), such as Flux [ 1 ] , for 3D generation tasks. Specifically, given a set of 3D objects for training, we first render each object into four distinct views along with their corresponding normal maps, forming a comprehensive representation we call the “3D Bundle Image”, which captures both geometry and texture. Note that 3D Bundle Image is essentially an image, making it highly compatible with the existing knowledge of pretrained diffusion models, thus ensuring easy training and strong generalization capabilities. Concretely, we use GPT-4V(ision) to generate descriptive captions for these 3D Bundle Images based on their RGB portion. These caption-image pairs are then used to fine-tune the Flux Model with Low-Rank Adaptation (LoRA) [ 15 ] , resulting in our core model: Kiss3DGen-Base. During testing, given a user-provided caption, Kiss3DGen generates a 3D Bundle Image, which is subsequently optimized using existing mesh reconstruction approaches (e.g., NeuS [ 29 ] , MeshFormer [ 25 ] , ISOMER [ 55 ] ) produce a final 3D mesh. Thanks to the generative power of the Flux model, Kiss3DGen is capable of generating 3D content beyond the training distribution. Since Kiss3DGen is fundamentally a diffusion model, it is naturally compatible with various diffusion-based techniques. In this work, we demonstrate this by incorporating ControlNet [ 62 ] to extend Kiss3DGen’s capabilities, which is termed Kiss3DGen-ControlNet. Given a 3D mesh, Kiss3DGen first renders a 3D Bundle Image, which serves as a condition for ControlNet to perform tasks such as mesh enhancement or editing tasks. Leveraging ControlNet, Kiss3DGen efficiently handles Image-to-3D generation (e.g., enhancing a coarse mesh), 3D editing (e.g., via ControlNet-Canny), and quality enhancement via ControlNet-Tile, which rebuilds low-quality 3D meshes to higher-quality. Our contributions can be summarized as follows: • We propose Kiss3DGen, a simple yet effective approach that retargets 2D diffusion models for 3D asset generation tasks. • We show that Kiss3DGen can be seamlessly integrated with ControlNet, enabling diverse functionalities such as text-to-3D generation, image-to-3D generation, 3D editing, and 3D asset enhancement. • Extensive experiments demonstrate that the Kiss3DGen model achieves state-of-the-art performance across various tasks. 2 Related Works 2.1 3D Generation Distillation-based 3D generation. The rapid advancement of large-scale generative models, particularly the remarkable success of 2D diffusion models [ 40 , 41 , 42 ] , has driven significant progress in 3D reconstruction. Pioneering methods like DreamFusion [ 37 ] and SJC [ 51 ] try to distill a 3D representation like NeRF [ 34 ] or Gaussian Splatting [ 18 ] from a 2D image diffusion by a Score Distillation Sampling (SDS) loss and its variants. Follow-up distillation-based methods [ 52 , 24 , 4 , 32