Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation
Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d · 2025
Read Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation by Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14271.txt.
Introduction
Weimin Bai, Yubo Li, Weijian Luo, and four co-authors let a vision-language model judge 3D generations while they are being made. Text-to-3D systems still miss fine prompt details and botch spatial relations, assembling parts in impossible ways. Their framework, VLM3D, repurposes the big model as a differentiable critic of both meaning and geometry.
The critic signal comes from the model's yes-or-no log-odds, scored along semantic and spatial axes. Plugged into optimization-based pipelines as a reward, it beat existing methods on standard benchmarks. Used as test-time guidance for feed-forward generators, it steered sampling to correct severe spatial errors, showing one signal can serve both paradigms.
Catalog record
| CREATOR | Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14271
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d Abstract Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM’s "Yes/No" log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM’s rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines. † † footnotetext: † Equal Contributions. ‡ Corresponding authors. 1 Introduction Creating 3D content from natural language descriptions has drawn significant attention 12 ; 51 ; 50 ; 26 ; 34 . Recent models, encompassing both optimization-based paradigms 26 and high-speed feed-forward architectures 12 ; 51 ; 50 , have achieved impressive results. However, state-of-the-art models across both paradigms still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details or complex interactions. For instance, given a detailed description of the “Embracing Peace” monument, even the powerful MVDream baseline omits one of the two figures entirely (see Figure 1 top). Second, they lack robust 3D spatial understanding. For instance, SOTA feed-forward models 12 ; 51 ; 50 can fail in part assembly and spatial relationships, generating nonsensical, floating collections of parts instead of coherent objects (see Figure 1 bottom). These failures arise from inadequate priors and limited reasoning: optimization-based methods inherit weak semantic grounding and poor spatial awareness from 2D diffusion models, while feed-forward methods are constrained by the limited complexity of 3D training data (often single objects rather than multi-object scenes), leading to poor modeling of 3D structure and compositional semantics. Both ultimately lack a deep understanding of spatial relationships and language. Figure 1 : Reproducing the “Embracing Peace” Statue with VLM3D. We challenge VLM3D’s dual paradigms with San Diego’s iconic monument. Top (Optimization-based): Given a long text description, baseline MVDream 34 suffers a catastrophic semantic failure, omitting the nurse entirely. Our VLM3D critic successfully recovers both figures and their signature pose. Key details are highlighted in red. Bottom (Feed-forward model-based): Comparing to the statue’s reference image, baseline Hunyuan3D 12 generates a distorted, spatially incoherent mesh. Our VLM3D substantially corrects these spatial failures and produces a more geometrically plausible 3D asset. To address these challenges across paradigms, we propose VLM3D, a 3D generation framework that integrates large vision-language models (VLMs) 1 ; 3 as powerful, differentiable semantic and spatial critics. Instead of relying on the weak semantic grounding of CLIP-style encoders 29 ; 48 or the 2D-centric priors of diffusion models 32 ; 34 , we leverage the VLM’s rich, language-grounded understanding of both fine-grained semantics and complex 3D spatial relationships 8 ; 6 ; 49 . Our core contribution is a dual-query VLM-based critic signal that simultaneously optimizes the 3D representation for both semantic fidelity and spatial/geometric coherence. Specifically, our framework tasks the VLM to act as a dual-objective reward model. It evaluates the multi-view renderings against a structured query comprising two distinct criteria: (1) Content Match: assessing correspondence to the semantic description, and (2) Geometric Quality: assessing whether the object is geometrically sound and consistent across views. The VLM is constrained to output only Yes or No ; we then design our differentiable VLM reward based on the extracted log-odds. Backpropagating this signal provides a direct semantically-grounded and spatially-aware gradient for optimizing the 3D representation. We demonstrate the generality of the proposed VLM-based critic by applying it across two distinct paradigms: • Score Distillation Sampling: We integrate VLM3D into an optimization-based pipeline SDS 26 ; 34 . On standard benchmarks like GPTEval3D 44 , VLM3D significantly outperforms existing methods in semantic accuracy and 3D plausibility. It successfully captures complex semantics (Fig. 1 ) and resolves geometric inconsistencies (Fig. 6 ). • Feed-forward Models: We show that VLM3D can be deeply integrated into the iterative sampling loop of SOTA native 3D feed-forward models 12 ; 51 . As shown in Fig. 4 , our test-time guidance approach effectively corrects severe geometric and part-assembly errors, producing coherent 3D assets where the original models failed. To the best of our knowledge, VLM3D makes the first attempt to establish a principled and generalizable path to inject the VLM’s rich, language-grounded understanding of both semantics and space into both optimization-based and feed-forward-based 3D generative pipelines. 2 Related Works Our work builds upon the intersection of two major paradigms in text-to-3D generation and the rapid advancements in vision-language models. 2.1 Optimization-based Text-to-3D Generation The first dominant paradigm leverages 2D diffusion priors 9 ; 32 to guide the per-scene optimization of a 3D representation. SDS 26 pioneered this by using the gradients from a pre-trained 2D diffusion model to optimize a Neural Radiance Field (NeRF) 22 . This distillation process enables plausible 3D asset creation from text without 3D data 26 . Subsequent methods improved this foundation. Magic3D introduced a coarse-to-fine optimization strategy 19 . MVDream trained a diffusion model on multi-view data to produce more 3D-consistent priors 34 , while ProlificDreamer framed the process as variational inference to improve diversity 42 . Other works have focused on integrating explicit reward models, such as DreamReward, which uses a 3D reward model trained on human feedback 46 . Despite their success, these methods inherit the limitations of their 2D priors. Their guidance signals often lack fine-grained semantic understanding, relying on CLIP-style encoders 28 ; 30 that struggle with complex prompts. Furthermore, being 2D-native, these priors lack explicit 3D spatial reasoning, leading to view-inconsistencies like the Janus problem 41 ; 23 ; 10 ; 33 . Our work addresses these semantic and spatial gaps with a VLM-based reward. 2.2 Feed-Forward Text-to-3D Generation The second major paradigm is feed-forward models, developed to overcome the slow optimization of SDS. These methods aim to generate 3D assets in a single forward pass, enabling near real-time creation. Early approaches focused on generating explicit representations like point clouds or voxels. More recently, methods like Instant3D 16 , LRM 11 and GRM 45 first generate consistent multi-view images and then use a fast reconstruction model to produce NeRF or 3D Gaussian Splatting 14 assets. Note, recent native 3D diffusion models 12 ; 51 ; 50 , which we also categorize as feed-forward, require an iterative sampling process to generate the final asset. These SOTA feed-forward models are trained on large-scale 3D datasets to directly output 3D representations from text