Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models
Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, Jeong Joon Park · 2025
Read Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models by Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, Jeong Joon Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00651.txt.
Introduction
Wizadwongsa, Zhou, Li, and Park show that models built to reconstruct 3D shapes from images can double as encoders for 3D generators, in this 2024 paper. The field had split into reconstruction tools and generative tools trained separately. Reusing the reconstructors skips the heavy cost of training a fresh encoder.
Raw reconstruction features, though, are too messy for generation. The team standardized them and weighted the important regions, then trained with an added image-based quality loss. A multi-stream transformer flow model learned on the cleaned features. The pipeline reached top-tier text-to-3D quality while scaling cleanly. The authors frame reuse as the cheap road forward.
Catalog record
| CREATOR | Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, Jeong Joon Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00651
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, Jeong Joon Park Abstract “a golden yellow toilet” “a purple car with modern futuristic design” “a large pumpkin” “a green suv car” “an old-fashioned oil lamp” “a red office chair with wheels” “a pizza with cheese and pepperoni topping” “a blue living room chair” Figure 1 : Text-to-3D generation. Our TriFlow model is trained on triplanes from a pretrained feed-forward reconstruction model Xu et al. 2024 and can generate a high-quality 3D model in a few seconds. Left column: samples of a model trained on Objaverse Deitke et al. 2024 LVIS. Right: samples of models fine-tuned on ShapeNet Chang et al. 2015 chairs and cars. † † footnotetext: * Equal Contribution Abstract Recent AI-based 3D content creation has largely evolved along two paths: feed-forward image-to-3D reconstruction approaches and 3D generative models trained with 2D or 3D supervision. In this work, we show that existing feed-forward reconstruction methods can serve as effective latent encoders for training 3D generative models, thereby bridging these two paradigms. By reusing powerful pre-trained reconstruction models, we avoid computationally expensive encoder network training and obtain rich 3D latent features for generative modeling for free. However, the latent spaces of reconstruction models are not well-suited for generative modeling due to their unstructured nature. To enable flow-based model training on these latent features, we develop post-processing pipelines, including protocols to standardize the features and spatial weighting to concentrate on important regions. We further incorporate a 2D image space perceptual rendering loss to handle the high-dimensional latent spaces. Finally, we propose a multi-stream transformer-based rectified flow architecture to achieve linear scaling and high-quality text-conditioned 3D generation. Our framework leverages the advancements of feed-forward reconstruction models to enhance the scalability of 3D generative modeling, achieving both high computational efficiency and state-of-the-art performance in text-to-3D generation. 1 Introduction The demand for digital content across gaming, AR/VR, and interactive visualization continues to drive innovation in automatic 3D asset generation. Recent advances in AI-driven 3D content creation have evolved along two primary trajectories: 3D generative models trained with 2D or 3D supervision Poole et al. 2022 ; Lan et al. 2025 ; Vahdat et al. 2022 , and feed-forward image-to-3D reconstruction approaches Hong et al. 2023 ; Xu et al. 2024 ; Boss et al. 2024 . While both paths have demonstrated compelling results, they have largely evolved independently. We propose bridging these paradigms by repurposing pre-trained reconstruction models as latent encoders for training scalable 3D generative models. Diffusion models have revolutionized 2D image synthesis by leveraging VAE-based latent representations for efficient training on massive datasets. However, 3D generation faces unique challenges due to limited data availability and the complexity of multi-view representations. The absence of effective 3D latent encoders has led researchers to pursue either computationally intensive optimization-based approaches for obtaining 3D latents Shue et al. 2023 ; Chen et al. 2023 ; Liu et al. 2024 ; Cao et al. 2023 or methods that generate 3D content directly from 2D priors Poole et al. 2022 ; Bahmani et al. 2024 . Recent approaches like Lan et al. 2025 have demonstrated the potential of feed-forward VAEs for generating triplane latents, offering processing speeds comparable to 2D VAEs. However, these methods require training dataset-specific image-to-triplane VAEs with specialized loss functions—a computationally intensive process that rivals the training time of the diffusion model itself, typically requiring 7 days with several powerful GPUs. Instead, We propose leveraging existing state-of-the-art reconstruction models as alternatives to specialized VAE training. Recent advances in feed-forward methods Hong et al. 2023 ; Xu et al. 2024 have produced highly accurate triplane and Gaussian Splatting representations from single or multi-view images. These models, whose training requires significant computational resources (100s of GPUs over multiple days), can handle diverse scenes effectively. Rather than duplicating this computational effort with specialized encoders, we show that flow-based models can be trained directly in the latent spaces of these pre-trained reconstruction models. However, these feed-forward reconstruction models were not designed to serve as encoders for generative modeling, presenting several challenges in their direct application. First, unlike VAEs which are regularized towards producing standardized isotropic Gaussian distributions, their latent features lack consistent statistical properties. Second, their unregularized latent spaces contain high-frequency artifacts that don’t meaningfully impact the final rendered output but degrade training convergence. Third, the high dimensionality of these latent spaces creates a significant discrepancy between latent space and rendering-space losses, complicating diffusion-like training approaches. Finally, training flow-based models on high-dimensional triplane representations is more computationally demanding than conventional 2D image generation. To address these challenges, we develop a comprehensive pipeline for generative training from reconstructed latents. First, we introduce a feature standardization protocol that transforms the latent distributions to match VAE-like properties. Second, we implement spatial importance weighting that focuses computational resources on regions most relevant to the final rendering, effectively suppressing noise in less significant areas. To bridge the gap between latent and rendering spaces, we incorporate a perceptual rendering loss that ensures generated features produce visually realistic images. Finally, we propose a multi-stream transformer architecture, dubbed TriFlow, that processes these high-dimensional features efficiently through parallel attention mechanisms, achieving linear computational scaling with respect to the number of triplane tokens. This combination of techniques enables effective training of flow-based generative models on the reconstruction latent features. Our extensive experiments demonstrate that our approach matches or exceeds state-of-the-art 3D generation methods in unconditional generation and outperforms in text-conditional generation, despite relying on off-the-shelf reconstruction models rather than specially trained encoders. These results validate that existing reconstruction models can serve as effective latent encoders for high-quality 3D generation, presenting a practical path toward scalable 3D content creation. 2 Related Work 3D Generative Models. With the advancement generative models including VAE Kingma 2013 , generative adversarial networks (GANs) Goodfellow et al. 2014 ; Kang et al. 2023 , and diffusion models Ho et al. 2020 ; Song et al. 2020 ; Karras et al. 2022 , we have seen substantial progress in content generation. Similarly, the rapid evolution of 3D scene modeling Mildenhall et al. 2021 ; Kerbl et al. 2023 ; Park et al. 2019 has driven significant growth in 3D generative models. Early efforts utilizing 3D-aware GANs Or-El et al. 2022 ; Chan et al. 2022 ; Schwarz et al. 2020 ; Chan et al. 2021 ; Gao et al. 2022 successfully generate high-quality 3D models but still suffered from the intrinsic issues of GANs such as mode collapse, training instability, and limited scalability Brock et al. 1809 ; Kang et al. 2023 . As diffusion models Ho et al. 2020 ; Song et al. 2020 ; Karras et al. 2022 have demonstrated superior performance in 2D image generation Dhariwal and Nichol 2021 , the 3D generation community has also begun exploring 3D generative models with diffusion models Shue et al. 2023 ; Müller et al. 2023 ; Chen et al. 2023 ; Liu et al. 2024 ; Cao et al. 2023 ; Lan et al. 2025 ; Poole et al. 2022 ; Wang et al. 2023 ; Wang et al. 2024 ; Gupta et al. 2023 . One line of work Poole et al. 2022 ; Wang et