Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation

Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d · 2025

Read Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation by Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14271.txt.

Introduction

Weimin Bai, Yubo Li, Weijian Luo, and four co-authors let a vision-language model judge 3D generations while they are being made. Text-to-3D systems still miss fine prompt details and botch spatial relations, assembling parts in impossible ways. Their framework, VLM3D, repurposes the big model as a differentiable critic of both meaning and geometry.

The critic signal comes from the model's yes-or-no log-odds, scored along semantic and spatial axes. Plugged into optimization-based pipelines as a reward, it beat existing methods on standard benchmarks. Used as test-time guidance for feed-forward generators, it steered sampling to correct severe spatial errors, showing one signal can serve both paradigms.

Catalog record

CREATORWeimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14271

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai, Yequan Wang, Wenzheng Chen, He Sun[2pt] 1. Peking University2. Xiaohongshu Inc3. MMLab, CUHK4. BAAI, Beijing[2pt] https://ai4scientificimaging.org/vlm3d Abstract Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM’s "Yes/No" log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM’s rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines. † † footnotetext: † Equal Contributions. ‡ Corresponding authors. 1 Introduction Creating 3D content from natural language descriptions has drawn significant attention 12 ; 51 ; 50 ; 26 ; 34 . Recent models, encompassing both optimization-based paradigms 26 and high-speed feed-forward architectures 12 ; 51 ; 50 , have achieved impressive results. However, state-of-the-art models across both paradigms still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details or complex interactions. For instance, given a detailed description of the “Embracing Peace” monument, even the powerful MVDream baseline omits one of the two figures entirely (see Figure 1 top). Second, they lack robust 3D spatial understanding. For instance, SOTA feed-forward models 12 ; 51 ; 50 can fail in part assembly and spatial relationships, generating nonsensical, floating collections of parts instead of coherent objects (see Figure 1 bottom). These failures arise from inadequate priors and limited reasoning: optimization-based methods inherit weak semantic grounding and poor spatial awareness from 2D diffusion models, while feed-forward methods are constrained by the limited complexity of 3D training data (often single objects rather than multi-object scenes), leading to poor modeling of 3D structure and compositional semantics. Both ultimately lack a deep understanding of spatial relationships and language. Figure 1 : Reproducing the “Embracing Peace” Statue with VLM3D. We challenge VLM3D’s dual paradigms with San Diego’s iconic monument. Top (Optimization-based): Given a long text description, baseline MVDream 34 suffers a catastrophic semantic failure, omitting the nurse entirely. Our VLM3D critic successfully recovers both figures and their signature pose. Key details are highlighted in red. Bottom (Feed-forward model-based): Comparing to the statue’s reference image, baseline Hunyuan3D 12 generates a distorted, spatially incoherent mesh. Our VLM3D substantially corrects these spatial failures and produces a more geometrically plausible 3D asset. To address these challenges across paradigms, we propose VLM3D, a 3D generation framework that integrates large vision-language models (VLMs) 1 ; 3 as powerful, differentiable semantic and spatial critics. Instead of relying on the weak semantic grounding of CLIP-style encoders 29 ; 48 or the 2D-centric priors of diffusion models 32 ; 34 , we leverage the VLM’s rich, language-grounded understanding of both fine-grained semantics and complex 3D spatial relationships 8 ; 6 ; 49 . Our core contribution is a dual-query VLM-based critic signal that simultaneously optimizes the 3D representation for both semantic fidelity and spatial/geometric coherence. Specifically, our framework tasks the VLM to act as a dual-objective reward model. It evaluates the multi-view renderings against a structured query comprising two distinct criteria: (1) Content Match: assessing correspondence to the semantic description, and (2) Geometric Quality: assessing whether the object is geometrically sound and consistent across views. The VLM is constrained to output only Yes or No ; we then design our differentiable VLM reward based on the extracted log-odds. Backpropagating this signal provides a direct semantically-grounded and spatially-aware gradient for optimizing the 3D representation. We demonstrate the generality of the proposed VLM-based critic by applying it across two distinct paradigms: • Score Distillation Sampling: We integrate VLM3D into an optimization-based pipeline SDS 26 ; 34 . On standard benchmarks like GPTEval3D 44 , VLM3D significantly outperforms existing methods in semantic accuracy and 3D plausibility. It successfully captures complex semantics (Fig. 1 ) and resolves geometric inconsistencies (Fig. 6 ). • Feed-forward Models: We show that VLM3D can be deeply integrated into the iterative sampling loop of SOTA native 3D feed-forward models 12 ; 51 . As shown in Fig. 4 , our test-time guidance approach effectively corrects severe geometric and part-assembly errors, producing coherent 3D assets where the original models failed. To the best of our knowledge, VLM3D makes the first attempt to establish a principled and generalizable path to inject the VLM’s rich, language-grounded understanding of both semantics and space into both optimization-based and feed-forward-based 3D generative pipelines. 2 Related Works Our work builds upon the intersection of two major paradigms in text-to-3D generation and the rapid advancements in vision-language models. 2.1 Optimization-based Text-to-3D Generation The first dominant paradigm leverages 2D diffusion priors 9 ; 32 to guide the per-scene optimization of a 3D representation. SDS 26 pioneered this by using the gradients from a pre-trained 2D diffusion model to optimize a Neural Radiance Field (NeRF) 22 . This distillation process enables plausible 3D asset creation from text without 3D data 26 . Subsequent methods improved this foundation. Magic3D introduced a coarse-to-fine optimization strategy 19 . MVDream trained a diffusion model on multi-view data to produce more 3D-consistent priors 34 , while ProlificDreamer framed the process as variational inference to improve diversity 42 . Other works have focused on integrating explicit reward models, such as DreamReward, which uses a 3D reward model trained on human feedback 46 . Despite their success, these methods inherit the limitations of their 2D priors. Their guidance signals often lack fine-grained semantic understanding, relying on CLIP-style encoders 28 ; 30 that struggle with complex prompts. Furthermore, being 2D-native, these priors lack explicit 3D spatial reasoning, leading to view-inconsistencies like the Janus problem 41 ; 23 ; 10 ; 33 . Our work addresses these semantic and spatial gaps with a VLM-based reward. 2.2 Feed-Forward Text-to-3D Generation The second major paradigm is feed-forward models, developed to overcome the slow optimization of SDS. These methods aim to generate 3D assets in a single forward pass, enabling near real-time creation. Early approaches focused on generating explicit representations like point clouds or voxels. More recently, methods like Instant3D 16 , LRM 11 and GRM 45 first generate consistent multi-view images and then use a fast reconstruction model to produce NeRF or 3D Gaussian Splatting 14 assets. Note, recent native 3D diffusion models 12 ; 51 ; 50 , which we also categorize as feed-forward, require an iterative sampling process to generate the final asset. These SOTA feed-forward models are trained on large-scale 3D datasets to directly output 3D representations from text

Read the complete text