Text–Image Conditioned 3D Generation
Jiazhong Cen, Jiemin Fang, Sikuang Li, Guanjun Wu, Chen Yang, Taoran Yi, Zanwei Zhou, Zhikuan Bao, Lingxi Xie, Wei Shen, Qi Tian · 2026
Read Text–Image Conditioned 3D Generation by Jiazhong Cen, Jiemin Fang, Sikuang Li, Guanjun Wu, Chen Yang, Taoran Yi, Zanwei Zhou, Zhikuan Bao, Lingxi Xie, Wei Shen, Qi Tian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21295.txt.
Introduction
Cen and co-authors formalize text-image conditioned 3D generation and build a baseline called TIGON. Image conditioned generators give high fidelity but suffer viewpoint bias. Text conditioned ones give broad guidance but lack visual detail. A diagnostic study shows even simple late fusion of the two beats either alone. TIGON uses separate image and text backbones with light cross-modal fusion. Across experiments, joint conditioning improves over single modality methods throughout. The authors present vision-language guidance as a promising direction for 3D generation.
Catalog record
| CREATOR | Jiazhong Cen, Jiemin Fang, Sikuang Li, Guanjun Wu, Chen Yang, Taoran Yi, Zanwei Zhou, Zhikuan Bao, Lingxi Xie, Wei Shen, Qi Tian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44377 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text–Image Conditioned 3D Generation Jiazhong Cen, Jiemin Fang, Sikuang Li, Guanjun Wu, Chen Yang, Taoran Yi, Zanwei Zhou, Zhikuan Bao, Lingxi Xie, Wei Shen, Qi Tian Abstract High-quality 3D assets are essential for VR/AR, industrial design, and entertainment, motivating growing interest in generative models that create 3D content from user prompts. Most existing 3D generators, however, rely on a single conditioning modality: image-conditioned models achieve high visual fidelity by exploiting pixel-aligned cues bhaodeut suffer from viewpoint bias when the input view is limited or ambiguous, while text-conditioned models provide broad semantic guidance yet lack low-level visual detail. This limits how users can express intent and raises a natural question: can these two modalities be combined for more flexible and faithful 3D generation? Our diagnostic study shows that even simple late fusion of text- and image-conditioned predictions outperforms single-modality models, revealing strong cross-modal complementarity. We therefore formalize Text–Image Conditioned 3D Generation , which requires joint reasoning over a visual exemplar and a textual specification. To address this task, we introduce TIGON , a minimalist dual-branch baseline with separate image- and text-conditioned backbones and lightweight cross-modal fusion. Extensive experiments show that text–image conditioning consistently improves over single-modality methods, highlighting complementary vision–language guidance as a promising direction for future 3D generation research. Project page: https://jumpat.github.io/tigon-page 0 0 footnotetext: {}^{\text{\Letter}} Corresponding authors. 1 Introduction Generating high-quality 3D assets has attracted increasing attention due to its importance for downstream applications such as virtual reality, industrial design, and embodied AI. A particularly promising line of work 69 ; 22 ; 67 ; 58 ; 31 ; 81 learns to generate 3D objects from either an input image or a text description. However, relying on a single conditioning modality can limit the flexibility of 3D generation. Figure 1 : Single-modality conditioning has limitations in satisfying user intent. Image-only conditioning captures local appearance but omits unobserved regions; text-only conveys semantics but lacks visual fidelity. In contrast, joint text–image conditioning produces 3D assets that are both semantically aligned with the description and faithful to the reference appearance. As shown in Fig. 1 , when a user specifies a 3D object with an example image, image-conditioned 3D generation can preserve local appearance but is highly sensitive to viewpoint informativeness: occlusions, atypical views, or incomplete object coverage force the model to hallucinate under-constrained regions, causing the generated 3D asset to deviate from the intended semantics. In contrast, text-conditioned 3D generation is semantically reasonable but lacks concrete visual constraints, so the output may roughly match the prompt while exhibiting poor visual quality. These observations raise a natural question of whether image and text conditions can provide complementary information for more flexible 3D generation. Intuitively, images anchor the result to the actual observed view, supplying reliable geometry and appearance cues, while text can specify additional semantics to disambiguate unobserved regions ( e.g. , “with a long coiled tiger tail”). In a diagnostic study, we find that conditioning on a low-information view degrades performance, but adding a textual description and fusing the image- and text-conditioned predictions noticeably recovers quality. This motivates us to move beyond single-modality settings and introduce Text--Image Conditioned 3D Generation 1 1 1 Here we focus on the native 3D generation setting, where the model directly generates a 3D representation under joint text–image conditioning. , which requires the 3D generator to jointly reason over the visual exemplar and the textual specification, and to generate a consistent 3D asset that is simultaneously faithful to the image-conditioned appearance/geometry and aligned with the text-defined semantics. To address this task, we propose a strong yet minimalist baseline named TIGON . It adopts a dual-branch design that retains two modality-specialized DiT backbones and couples them via two lightweight fusion mechanisms: (i) cross-modal linear bridges for bidirectional feature sharing (early fusion), and (ii) step-wise prediction averaging along the denoising trajectory (late fusion). This design prevents either branch from shouldering the cross-modal domain gap, preserves their original single-modality ability, and enables free-form conditioning. Extensive experiments show that TIGON delivers more flexible 3D generation. Our contributions are summarized as follows: (1) We identify and empirically diagnose the limitations of existing single-modality 3D generation methods. (2) We show that image and text provide complementary conditioning signals and, motivated by this, introduce the task of text–image conditioned 3D generation. (3) We propose TIGON, a simple yet effective baseline method that leverages modality-specific backbones with lightweight cross-modal fusion. (4) We conduct extensive experiments to demonstrate that TIGON achieves more robust and flexible 3D generation. 2 Related Work 3D Generation with 2D Generative Models. SDS-based text-to-3D methods optimize differentiable 3D representations ( e.g. , NeRFs 44 ) using frozen 2D diffusion priors, as pioneered by DreamFusion 50 and SJC 61 . Later works improve resolution, geometry–appearance disentanglement, optimization stability, and multi-view consistency, while recent hybrids with 3D Gaussian Splatting or native 3D diffusion further boost efficiency and fidelity 68 ; 35 ; 4 ; 62 ; 33 ; 27 ; 54 ; 79 ; 56 ; 34 ; 30 . A parallel line studies image-conditioned 3D generation and reconstruction with 2D generative priors, either by directly optimizing 3D representations or by first generating consistent multi-view images and then reconstructing 3D content 43 ; 71 ; 57 ; 40 ; 63 ; 38 ; 39 ; 42 ; 41 ; 72 ; 30 . Despite strong progress, reliance on 2D priors often limits 3D consistency, motivating native 3D generative models and cross-modal latent modeling 83 . Among prior works, TICD 18 and FlexGen 74 are most related to ours. TICD augments SDS-based text-to-3D with an image diffusion prior, while FlexGen jointly conditions on text and image but focuses on 2D multi-view generation rather than native 3D synthesis. In contrast, we study native 3D generation under joint text-image conditioning. Native 3D Generative Models. Unlike SDS-based pipelines that rely on 2D generators at test time, native 3D generative models operate directly on 3D representations such as point clouds, meshes, voxels, 3D Gaussians 25 ; 55 , and neural fields 17 ; 44 ; 2 ; 29 . Early works 46 ; 59 introduce fast point-cloud synthesis via latent diffusion. Later models such as 3DShape2VecSet 80 improve geometry-aware latents for diffusion training. Moreover, scalable voxel-based methods 53 , octree-based models 70 and 3DGS-based models 76 improve generation resolution and efficiency. More recent studies scale data and model capacity for high-quality asset synthesis. Some focus on geometry-only generation for detailed meshes 6 ; 66 ; 67 ; 32 ; 77 , while others target fully textured 3D assets 81 ; 7 ; 23 ; 28 ; 31 ; 85 . Among the latter, TRELLIS 69 introduces a sparse, structured latent representation that enables high-fidelity 3D generation via a two-stage pipeline, and UniLat3D 65 provides a unified-latent variant with a simplified sampling process. Together, these advances establish native 3D diffusion models as strong alternatives to 2D diffusion-based pipelines. However, most existing approaches assume single-modality conditioning (either image or text), which limits the flexibility and expressiveness of user instructions. In this work, we study this limitation and introduce the task of text–image conditioned 3D generation. Multimodal-Conditioned Generation. Multimodal conditioning has proven effective in text, image, and video generation. Multimodal LLMs that ingest both language and visual (or even 3D) inputs