How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning
Luyu Yang Yutong Dai An Yan Viraj Prabhu Ran Xu Zeyuan Chen · 2026
Read How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning by Luyu Yang Yutong Dai An Yan Viraj Prabhu Ran Xu Zeyuan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24866.txt.
Introduction
Yang and co-authors present DreamHouse, a benchmark for physical generative reasoning in vision-language models. Most benchmarks reward visually plausible output, not whether a model grasps the steps needed to build it. DreamHouse asks for artifacts meeting geometric, structural, constructability, and code-compliance constraints at once. The domain is home timber-frame construction, with codified standards and correctness checks. Over 26,000 structures in 13 styles are verified to construction-document level. A ten-test scoring framework rates the builds, and models can iterate against mid-build states.
Catalog record
| CREATOR | Luyu Yang Yutong Dai An Yan Viraj Prabhu Ran Xu Zeyuan Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86698 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning Luyu Yang Yutong Dai An Yan Viraj Prabhu Ran Xu Zeyuan Chen Abstract The physical world is not merely visual; it is governed by rigorous structural and procedural constraints. Yet, the evaluation of vision-language models (VLMs) remains heavily skewed toward perceptual realism, prioritizing the generation of visually plausible 3D layouts, shapes, and appearances. Current benchmarks rarely test whether models grasp the step-by-step processes and physical dependencies required to actually build these artifacts—a capability essential for automating design-to-construction pipelines. To address this, we introduce DreamHouse , a novel benchmark for physical generative reasoning : the capacity to synthesize artifacts that concurrently satisfy geometric, structural, constructability, and code-compliance constraints. We ground this benchmark in residential timber-frame construction, a domain with fully codified engineering standards and objectively verifiable correctness. We curate over 26,000 structures spanning 13 architectural styles—each verified to construction-document standards (LOD 350)—and develop a deterministic 10-test structural validation framework. Unlike static benchmarks that assess only final outputs, DreamHouse supports iterative agentic interaction. Models observe intermediate build states, generate construction actions, and receive structured environmental feedback, enabling a fine-grained evaluation of planning, structural reasoning, and self-correction. Extensive experiments with state-of-the-art VLMs reveal substantial capability gaps that are largely invisible on existing leaderboards. These findings establish physical validity as a critical evaluation axis orthogonal to visual realism, highlighting physical generative reasoning as a distinct and underdeveloped frontier in multimodal intelligence. Available at https://luluyuyuyang.github.io/dreamhouse Figure 1 : DreamHouse benchmark samples. Each row shows a single structure across five representations: material schedule (member counts by subsystem), foundation, two intermediate framing stages, and complete timber frame, alongside the target exterior rendering from which the model must infer the hidden structural system. Three of the 13 architectural styles are shown: Split-Level (678 members), Cruciform (1,350 members), and Barn (845 members). Member counts span four subsystems: foundation (Fdn), floor, walls, and roof. 1 Introduction The physical world is not merely a surface; it is a rigorous system of constraints Marín et al. 2025 ; Zha et al. 2025 ; Ouyang et al. 2025 ; Wu et al. 2025 ; Stogiannidis et al. 2025 ; Chen et al. 2025a ; Pan and Liu 2025 ; Tang et al. 2025b . A floor must bear load, a rafter must respect its allowable span, and a wall must continuously transfer force from roof to foundation. These are not aesthetic choices, but fundamental physical realities. Yet, the dominant evaluation paradigm for generative vision models treats the world as purely visual, asking only if a model can produce outputs that look ‘correct Wang et al. 2025 ; Kong et al. 2025 ; Team et al. 2025 ; Duan et al. 2025 ; Alhaija et al. 2025 ; Ding et al. 2025 . Whether those outputs could physically stand is a question the field has largely left unasked. While the recent surge in vision-language models (VLMs) has yielded remarkable perceptual capabilities, evaluating their grasp of physical laws remains in its infancy Wang et al. 2025 ; Zhang et al. 2025a ; Rodionov et al. 2025 ; Chow et al. 2025 ; Chen et al. 2025c . Recent benchmarks like PhysBench Chow et al. 2025 and VSI-Bench Chen et al. 2025c probe physical reasoning and spatial recall, but they are fundamentally comprehension tasks—the model observes, answers, and is scored. They do not evaluate whether a model can build . Generating a physically realizable artifact from scratch, under strict engineering constraints and without visible ground truth, requires moving beyond passive observation. Between perceiving a structure and constructing one lies a critical gap that standard benchmarks fail to measure. To bridge this gap, we introduce DreamHouse , a novel benchmark designed to evaluate physical generative reasoning —the capacity to synthesize artifacts that concurrently satisfy geometric, structural, and code-compliance constraints. We ground our benchmark in residential timber-frame construction. This domain provides fully codified correctness criteria and discrete, verifiable components, while remaining visually complex enough to prove that perceptual plausibility alone is insufficient. DreamHouse comprises over 26,000 structurally verified models spanning 13 architectural styles. Furthermore, we provide a suite of 10 deterministic, physics-based tests covering load paths, span limits, member connectivity, etc. . Crucially, these tests operate directly on the scene graph, bypassing the need for computationally expensive simulations. The DreamHouse task is formulated as an iterative generation process: given rendered views of a target structure, a model must generate Blender Python construction code, process structured validation feedback, and refine its output until all structural tests pass. We evaluate performance across three protocols with varying degrees of external scaffolding (Planner-Atomic, Planner-Reactive and Planner-Managed) and two input conditions (bare framing visible, Frame , versus occluded by finished cladding, Facade ). This framework serves as a controlled ablation study, isolating whether generation failures stem from weak spatial reasoning, deficient planning, or an inability to self-correct. Our extensive evaluation of three frontier VLMs inclyding GPT-5 Singh et al. 2025 , Gemini 3 gem 2025 and Claude 4.5 cla 2025 across more than 20,000 independent agentic tasks reveals striking limitations in current state-of-the-art models. Notably, models that excel on standard coding and reasoning leaderboards often struggle with physical generation; highly-ranked generalists frequently underperform compared to their peers when evaluated under our structured protocols. Even the most capable model achieves a joint pass rate of merely 7.1% , successfully satisfying both structural validity and visual fidelity simultaneously. These findings demonstrate that physical generative reasoning is not a natural byproduct of general intelligence, but rather a distinct, underdeveloped capability axis demanding dedicated evaluation. Our contributions: • Physical generative reasoning as a novel VLM evaluation axis, distinct from perception, comprehension, and simulation-based benchmarks. • The DreamHouse benchmark , comprising 26,000+ verified timber-frame structures across 13 architectural styles with multi-view renderings and construction phase-wise annotations. • A 10-test structural validation suite that is deterministic and simulation-free, covering International Residential Code (IRC) compliance Code 2018 , physics, geometry, and fabrication details. • A three-protocol agentic evaluation framework , showing that scaffold design is as important as model selection for physical generation tasks. 2 Related Work Physical and Spatial Benchmarking for VLMs. A growing body of work exposes systematic gaps between VLM semantic fluency and physical-world grounding Jia et al. 2025 ; Zhang et al. 2025a ; Rodionov et al. 2025 ; Pun et al. 2025 ; Chow et al. 2025 ; Chen et al. 2025c . PhysBench Chow et al. 2025 evaluates 75 VLMs on 10K video-image-text entries spanning object properties, relationships, and dynamics, finding that models excel at static recognition but fail on Newtonian dynamics, a deficiency attributed to missing physical priors rather than perceptual limits. VSI-Bench Yang et al. 2025 measures metric visual-spatial intelligence (distances, sizes, directions) from egocentric video, showing that standard chain-of-thought prompting degrades spatial estimation, models must instead build explicit cognitive maps. Think-with-3D Chen et al. 2025c endows VLMs with 3D geometric imagination via latent alignment and RL-based spatial rewards, enabling occluded geometry completion from sparse views. These benchmarks evaluate passive physical understanding: answering questions about or reconstructing observed scenes. DreamHouse targets the harder generative gap, producing structures that are themselves physically valid, without any reference structure to recall. 3D Scene Generation. Large-scale 3D generative models Wang