Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation

Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang LiuJiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming ChenDavid Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu Maitrix.org · 2025

Read Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation by Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang LiuJiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming ChenDavid Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu Maitrix.org (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21876.txt.

Introduction

Do today's vision-language models carry an inner picture of how the world works? Qiyue Gao and a large team of co-authors build a benchmark, WM-ABench, to test that question piece by piece. Their framework borrows from psychology and splits world modeling into perception skills, like judging space, time, quantity, and motion, and prediction skills, like simulating mechanisms and chaining inferences.

The benchmark spans twenty-three fine measures across six simulated worlds with controlled what-if scenarios. In 660 experiments on fifteen leading models, the systems showed striking gaps. Nearly all scored close to chance at telling motion paths apart, and some displayed odd biases, such as treating blue objects as faster than green ones. The authors argue current models lack the disentangled grasp that real planning needs.

Catalog record

CREATORQiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang LiuJiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming ChenDavid Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu Maitrix.org
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang LiuJiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming ChenDavid Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu Maitrix.orgUC San DiegoJHUCornell TechEPFLUMich https://wm-abench.maitrix.org/ * equal contribution Abstract Internal world models (WMs) enable agents to understand the world’s state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs’ fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses perception (visual, spatial, temporal, quantitative, and motion) and prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench , a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding—e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling. 1 Introduction World models (WMs) in agents provide internal representations of the external world ( Johnson-Laird 1983 ; Wooldridge and Jennings 1995 ) . They simulate how the current state transforms to the next ( Kappes and Morewedge 2016 ; Russell and Norvig 2016 ) , enabling agents to perform deliberative planning by predicting probable future states and choosing the most favorable one ( Ha and Schmidhuber 2018b ; LeCun 2022 ; Hu and Shu 2023 ; Ai et al. 2024 ; Tian et al. 2025 ) . Different environments operate under distinct mechanisms and dynamics—for example, the transition mechanics of autonomous vehicles, surgical robots, and rover-based spacecraft vary significantly. Prior work handled this diversity by building environment-specific world models, but they failed to adapt across various real applications. Recent large-scale Vision-Language Models (VLMs; Google 2023 ; OpenAI et al. 2024 ; Anthropic 2024 ; Li et al. 2024b ; OpenAI 2025b , inter alia ) enhance generalist LLMs with visual semantics and encapsulate extensive knowledge of world dynamics, making them promising potential world models for general domains. Unlike video generative world models which directly generate the next states, VLMs can reason in their latent representation space and forecast via language. However, their language grounding and world simulation capabilities may still be insufficient in various aspects. For example, existing benchmarks have revealed their vulnerability in visual or spatiotemporal perception ( Goyal et al. 2020 ; Shangguan et al. 2024 ; Fu et al. 2024b ; Zhang et al. 2025 ) and future state prediction driven by intuitive physics ( Bakhtin et al. 2019 ; Yi et al. 2020a ; Bear et al. 2022 ) . These limitations underscore the need for a more systematic evaluation. A robust world model must integrate multiple fundamental abilities in perception and prediction, yet previous studies that assess these aspects in isolation provide only a partial view. To address this, we propose an atomic evaluation framework, systematically testing the essential aspects (and their interactions) of a VLM’s internal world model from first principles. With theories and evidence from comparative psychology and cognitive science ( Spelke 2000 ; Knill and Pouget 2004 ; Olmstead and Kuhlmeier 2015 ) , we first present a systematic conceptual framework to formalize the functioning of world models (Figure ). We decompose the process into two stages: (1) perception stage, which involves visual , spatial , temporal , quantitative , and motion perceptions ( Baillargeon et al. 1985 ; Merleau-Ponty 2004 ; Coren et al. 2004 ; Hoffmann et al. 2011 ) ; and (2) prediction stage, which involves mechanistic simulation , transitive inference , and compositional inference ( Hegarty 2004 ; Barsalou 2008 ; Prystawski et al. 2023 ) . Following this framework, we create WM-ABench , the World Model Atomic Benchmark that covers 23 fine-grained dimensions (Figure 1 ) of world modeling and over 100,000 instances curated from 6 different simulators ( Dosovitskiy et al. 2017a ; Gan et al. 2021 ; Szot et al. 2021a ; Tao et al. 2024 ; Bear et al. 2022 ; Gu et al. 2023 ) . By systematically manipulating environmental factors and simulating counterfactual actions, we generate incorrect states for models to differentiate from the correct ones to ensure controlled studies. We also measure human performance to verify the fairness and solvability of our problems. We conduct 660 experiments on 15 state-of-the-art VLMs and find that, while they excel in certain aspects of visual and quantitative perception, they are surprisingly limited in many other dimensions (Figure ). Specifically, VLMs exhibit (1) weak perception of space, time, and motion; (2) insufficient knowledge of physical causality in intuitive physics and agentic actions; (3) limited transitive and compositional reasoning capabilities, showing near-random accuracy. Our further analyses reveal a lack of independent and robust world representations in VLMs, e.g., mistakenly associating color with speed. These findings reveal significant gaps between current VLMs and human-level world modeling, shedding light on the need for deeper understanding, grounding, and reasoning over the perceptual world and its transition mechanisms before VLMs can truly serve as generalist world models. 2 The Dual-Stage Conceptual Framework We view world modeling as a two-stage process of perception and prediction ( Knill and Pouget 2004 ; Ha and Schmidhuber 2018b ) . In the first stage, agents form internal representations of the current state by sensing and encoding environmental stimuli. In the second stage, agents use these internal representations to extrapolate future states, refining their model whenever new ground-truth observations arrive. This dual-stage framework explains (1) how raw sensory signals are converted into compact world representations; and (2) how these representations then guide forward simulations. The two stages of WMs are core functions in agents’ advanced planning and decision-making. We brief our formulation here and leave the in-depth discussion in Appendix A . 2.1 Perception Stage Perception involves extracting and organizing essential information from multi-sensory cues ( Merleau-Ponty 2004 ; Coren et al. 2004 ) . It is not just bottom-up signal processing but also top-down inference grounded in prior knowledge. For instance, many intelligent animals (e.g., crows, dolphins, chimpanzees) grasp object permanence by maintaining accurate representations of hidden or partially occluded objects ( Baillargeon et al. 1985 ; Hoffmann et al. 2011 ) . While real-world perception spans multitudinous modalities (time, vision, temperature, humidity, audition, proprioception, etc.), we only focus on a limited number of perceptual dimensions. From an epistemic perspective, we design these dimensions to maximize the information captured about the external world while minimizing representational dimensions, ensuring they remain mutually orthogonal. From the practical perspective, we only cover the dimensions that modern VLMs can access. In this work, we consider 5

Read the complete text