Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation
Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu · 2025
Read Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation by Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05427.txt.
Introduction
Xuyi Meng and co-authors turn a trained 2D image generator into a direct 3D object maker. Three-dimensional data are scarce and rough compared with the huge image sets behind today's diffusion models, so 3D generation lags. Their insight is that a popular 3D format, Gaussian splats, can be split into several flat views, each carrying a different attribute of the object. Generating those views is a 2D job, so a pre-trained image model can do it.
New attention layers tie the views and attributes together, keeping the result consistent as one solid object rather than several clashing pictures. Tests on synthetic and real-world photos showed stronger 3D generation than rivals, including on object types never seen in training. The authors call the system Zero-1-to-G. It offers a shortcut around the 3D data shortage: borrow the visual knowledge already inside 2D models.
Catalog record
| CREATOR | Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu Abstract Recent advances in 2D image generation have achieved remarkable quality, largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G , a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation. 1 1 footnotetext: Equal contribution. Figure 1: Zero-1-to-G tackles direct Gaussian splat generation from single images. By using pretrained 2D diffusion models, we are able to generalize to in-the-wild objects. 1 Introduction Figure 2: The pipeline of Zero-1-to-G. During training, we fine-tune both the VAE decoder (Sec. 3.3 ) and the denoising UNet (Sec. 3.2 ) of Stable Diffusion. At inference time, given a single view input of the target object, each component in the splatter image is generated by conditioning the camera view and attribute switcher. The generated set of splatter image components can be directly fused into Gaussian splats (Sec. 3.1 ). Here we show splatter images of 3 views for better illustration, while our main experiments are conducted with 6 views. Single image to 3D generation is a pivotal challenge in computer vision and graphics, supporting various downstream applications such as virtual reality and gaming technologies. A primary difficulty lies in managing the uncertainty of unseen regions, as these areas represent a conditional distribution based on the visible portions of a 3D object. Recent advancements in diffusion models ( Ho et al., 2020 ; Rombach et al., 2022 ) have demonstrated significant efficacy in capturing complex data distributions within images and videos, prompting researchers to harness these models for single image to 3D generation. Early efforts distilled 3D neural fields from pretrained 2D diffusion models via score distillation ( Poole et al., 2022 ; Wang et al., 2023 ) . However, these approaches necessitate per-scene optimization, which is time-consuming and susceptible to multi-faced Janus problems. Subsequent research achieved feed-forward generation by fine-tuning pretrained models to generate multi-view images of the same object ( Liu et al., 2023b ; Shi et al., 2023a ; Long et al., 2023 ; Liu et al., 2023c ) and enabling indirect 3D generation through sparse-view reconstruction models ( Li et al., 2023 ; Tang et al., 2024 ; Xu et al., 2024 ) , and Cat3D ( Gao et al., 2024 ) further extends sparse view generation to dense view generation for better reconstruction. Although these two-stage methods enhance quality and efficiency, they often yield poor geometric fidelity and blurry renderings due to inconsistencies in multi-view images. To circumvent these limitations, recent methodologies have trained diffusion models directly on 3D representations ( Liu et al., 2023d ; Chen et al., 2023a ; Zhang et al., 2024a ; He et al., 2024 ; Nichol et al., 2022 ) , thereby eliminating the reliance on multi-view images. However, direct 3D generation techniques necessitate training from scratch, requiring substantial computational resources and large 3D datasets, which remain scarce—three orders of magnitude less prevalent than 2D data. In this paper, we propose a novel approach for direct 3D generation that unites the strengths of both worlds: it leverages the expressive power of 2D diffusion networks while maintaining the 3D structural consistency required for accurate 3D generation. Our key contribution is bridging the gap between Gaussian splats and natural images typically used in 2D generation tasks. While the original Gaussian splats consist of 14-channel images encoding various attributes, we decompose each of them into multiple 3-channel attribute images while preserving its 3D information (Sec. 3.1 ) This decomposition also enables efficient latent diffusion training by projecting the splatter images into the latent space of a pretrained VAE, making our method directly generate 3D structures within a pretrained 2D diffusion framework. To ensure strong 3D consistency, we introduce cross-view attention layers in Stable Diffusion to enable information exchange between different viewpoints and cross-attribute attention mechanisms to maintain coherence across Gaussian attributes within a splatter image (Sec. 3.2 ). Additionally, we fine-tune the VAE decoder to address the domain gap between splatter images and natural images, as we observed that splatter image quality is highly sensitive to pixel-level variations (Sec. 3.3 ). By leveraging pretrained priors, our method not only generalizes better to unseen objects but also improves training efficiency compared to existing 3D generation methods. It is important to note that, although we generate multiview splatter images, our method produces more structurally consistent and higher-quality results compared to traditional two-stage multiview 3D generation approaches. In two-stage methods ( Xu et al., 2024 ; Tang et al., 2024 ) , strong pixel-level consistency is required in the first stage to ensure accurate reconstruction in the second stage, which is often difficult to achieve. Moreover, the first stage operates independently of the second, lacking coordination between the two, thereby exacerbating inconsistencies. In contrast, our approach directly generates the final 3D representation in a single stage, eliminating the need for pixel-level correspondence between multiview splatter images. Since splatter images can contain redundant information, and their spatial positions do not necessarily map to the final 3D positions, this single-stage process offers greater flexibility and robustness, ensuring a more consistent final 3D structure. Overall, our contributions can be summarized as below: • We present Zero-1-to-G , a novel direct 3D generative model for Gaussian splats that achieves excellent 3D consistency and superior rendering quality. • We observe that Gaussian splats, as a 3D representation, can be decomposed into a set of 2D images of different views and attributes, making them inherently compatible with 2D image generation frameworks. • Through decomposition and transformation of splatter images, we use 2D diffusion models for direct 3D generation with proper fine-tuning, unleash the power of pretrained 2D diffusion for training efficiently and better generalization towards in-the-wild data. 2 Related Works Optimization-based 3D Generation Dreamfusion ( Poole et al., 2022 ) and subsequent works ( Lin et al., 2023 ; Chen et al., 2023b ; Wang et al., 2023 ) utilize a pretrained text-to-image diffusion model to optimize a 3D representation through score distillation. DreamGaussian ( Tang et al., 2023 ) significantly reduces training time by optimizing Gaussian splats. However, score distillation-based methods still require minutes of optimization per scene, as they must compare renderings with diffusion outputs from various viewpoints, which limits their generation speed. Additionally, these methods lack a clear understanding