Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generation

Tiange Xiang, Kai Li, Chengjiang Long, Christian Häne, Peihong Guo, Scott Delp, Ehsan Adeli, Li Fei-Fei · 2025

Read Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generation by Tiange Xiang, Kai Li, Chengjiang Long, Christian Häne, Peihong Guo, Scott Delp, Ehsan Adeli, Li Fei-Fei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15877.txt.

Introduction

Tiange Xiang and seven colleagues borrow the power of two-dimensional image generators to make three-dimensional objects. Text-to-image diffusion models thrive on plentiful paired data. Three-dimensional data is scarce, so 3D generators lag behind.

Their trick is a format called Gaussian Atlas: a dense two-dimensional grid that holds the parts of a 3D Gaussian model in flattened form. A pretrained image model can then be fine-tuned to draw these atlases, which unfold back into 3D objects. To train it, the team built GaussianVerse, a set of about 205,000 fitted 3D objects. Tests show the adapted image models produce credible 3D content from text prompts.

Catalog record

CREATORTiange Xiang, Kai Li, Chengjiang Long, Christian Häne, Peihong Guo, Scott Delp, Ehsan Adeli, Li Fei-Fei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generation Tiange Xiang, Kai Li, Chengjiang Long, Christian Häne, Peihong Guo, Scott Delp, Ehsan Adeli, Li Fei-Fei Abstract Recent advances in text-to-image diffusion models have been driven by the increasing availability of paired 2D data. However, the development of 3D diffusion models has been hindered by the scarcity of high-quality 3D data, resulting in less competitive performance compared to their 2D counterparts. To address this challenge, we propose repurposing pre-trained 2D diffusion models for 3D object generation. We introduce Gaussian Atlas , a novel representation that utilizes dense 2D grids, enabling the fine-tuning of 2D diffusion models to generate 3D Gaussians. Our approach demonstrates successful transfer learning from a pre-trained 2D diffusion model to a 2D manifold flattened from 3D structures. To support model training, we compile GaussianVerse , a large-scale dataset comprising 205K high-quality 3D Gaussian fittings of various 3D objects. Our experimental results show that text-to-image diffusion models can be effectively adapted for 3D content generation, bridging the gap between 2D and 3D modeling. 1 Introduction Understanding the 3D world is crucial for numerous real-world applications. In this work, we focus on generative 3D modeling with the primary objective of generating high-quality 3D assets from given textural descriptions [ 13 , 33 ] . Figure 1 : Previous 3D generators are either end-to-end 3D models [ 57 , 10 ] or a combination of a multi-view 2D generator followed by a 2D-to-3D lifting model [ 54 , 49 ] . In this work, we achieve 3D object generation by directly fine-tuning 2D generation models. With advances in 3D content representation, diffusion models [ 11 , 44 ] begin to play a predominant role in 3D generation [ 25 , 31 , 12 ] . Among different 3D representations, 3D Gaussians [ 15 ] have gained popularity for modeling 3D scenes and objects due to their high efficiency, explicit structure, and superior rendering quality. Notably, training diffusion models to generate 3D Gaussians requires high-quality “ground truth” 3D Gaussians, which must be pre-fitted to the 3D objects [ 10 , 46 , 27 ] . To advance the community, we constructed a large-scale dataset named GaussianVerse , comprising high-quality 3D Gaussian fittings for a diverse range of objects. Unlike previous studies that also pre-compute 3D Gaussian references, GaussianVerse provides higher-quality fittings with the minimum required number of Gaussians, through a novel pruning strategy and a more effective, but compute-intensive, fitting process. Designing standalone diffusion models for 3D generation is straightforward. However, such models have significant limitations especially when trained solely on 3D data, as high-quality 3D data is relatively scarce compared to 2D images. To achieve higher rendering fidelity in 3D generation, it is desirable to leverage the learned prior knowledge from well-pretrained 2D diffusion models [ 33 , 5 , 21 , 28 ] . Some methods have been proposed to incorporate pretrained 2D diffusion models with frozen weights into the 3D generation pipeline through complex designs, such as score-distilled sampling [ 33 ] and collaborative controls [ 5 ] . In this paper, we propose a fresh perspective that repurposes 2D diffusion models for 3D generation through direct fine-tuning. To fully harness the capabilities of these 2D diffusion models, we introduce Gaussian Atlas , a novel 2D representation of 3D Gaussians. This representation enables direct fine-tuning by first transporting unorganized 3D Gaussians into a standard 3D sphere and then applying equirectangular projection to map them into a square 2D grid, creating a Gaussian Atlas. We show that these Gaussian atlases facilitate transfer of the prior knowledge captured in 2D diffusion models to the 3D generation task. By doing so, our approach provides a means to leverage the learned 2D priors for 3D generation, unlocking new possibilities for efficient and effective 3D content creation. To summarize, our major contributions are three-fold: ( i ) We present a large-scale dataset, GaussianVerse , consisting of 205,737 high-quality 3D Gaussian fittings for diverse objects sampled from Sketchfab [ 43 ] ; ( ii ) We propose a novel 2D representation of 3D Gaussians, Gaussian Atlas , facilitating 3D generation tasks through a new perspective by fine-tuning pretrained 2D diffusion models; ( iii ) We demonstrate that the proposed approaches surpass state-of-the-art 3D Gaussian generators in terms of both generation-prompt alignment and user preferences. 2 Related Work Repurposing 2D diffusion models. Diffusion models [ 11 , 37 , 6 ] , are originally designed for text-to-image generation. When trained on large-scale internet images, prior knowledge learned from diverse domains can be transferred to different tasks. Marigold [ 14 ] is an example of how fine-tuning pretrained Latent Diffusion (LD) models [ 37 ] can benefit depth prediction. Marigold first re-parameterizes single-channel depth maps as RGB images and uses the pretrained VAE encoder to compress both the input image as well as its paired depth map into a shared latent space. The two latents are then concatenated to enable conditional generation of depth latents, which are subsequently decoded back into RGB space through the VAE decoder. More recent work, such as GeoWizard [ 7 ] and GenPercept [ 52 ] , explored the repurposing of LD models for more geometric prediction tasks and beyond. As more and more studies have shown the potential of finetuning LD for different purposes, this work is among the first to explore how a LD model can be adapted for 3D generation, addressing the significant disparity between 3D objects and 2D images. Figure 2 : (a) GaussianVerse offers high-quality 3DGS fittings for diverse 3D objects. (b) A word cloud and (c) a list of the most frequently occurring words generated from the captions [ 26 ] of the fitted objects. (d) The distribution of the number of Gaussians per fitting shows that our method generates fewer Gaussians for simple objects and more for complex ones. Red dotted lines indicate the 25%, 50%, and 75% percentile of the distribution. 3D generation with Gaussian splatting. Unlike 2D images, 3D objects are more difficult to generate due to the additional dimension and geometric constraints [ 29 , 15 ] . Among all 3D generation methods [ 48 , 35 , 16 , 4 , 49 , 53 , 18 ] , there is a line of work that utilizes diffusion models to generate 3D Gaussians, which is closely related to this paper. GSD [ 30 ] introduced rendering guidance to constrain the sampling of 3D Gaussians with 2D observations. L3DG [ 36 ] is another example, which embeds 3D Gaussians onto a dense latent grid and learns a diffusion model in the latent space for generation. Moreover, GVGen [ 10 ] and GaussianCube [ 57 ] achieve 3D Gaussian diffusion by transforming sparsely located Gaussians into more structured 3D volumes. GVGen fits 3D Gaussians directly on a volume with offsets, while GaussianCube applies Optimal Transport to move 3D Gaussians to vertices of a predefined 3D grid. Gaussian Anything [ 17 ] adopts a two-step approach that generates Gaussians by auto-encoding point cloud latents. Recently, TRELLIS [ 51 ] designed a structured representation that supports decoding to various 3D formats. In this work, we propose to transform 3D Gaussians onto 2D atlases, to leverage

Read the complete text