Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, Chenxin Li, Panwang Pan, Junbin Lu, Jingyan Jiang, Xinghao Ding, Yue Huang, Zhi Wang · 2026
Read Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World by Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, Chenxin Li, Panwang Pan, Junbin Lu, Jingyan Jiang, Xinghao Ding, Yue Huang, Zhi Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12746.txt.
Introduction
Yuzhi Huang and co-authors ask whether multimodal language models can think in dynamics, tracking how scenes change over time. Static image skill is strong, yet motion and interaction may be read inconsistently. Their Dyn-Bench gathers dynamic scenes from real and synthetic video for robust tests. It probes spatial reasoning and object grounding at region level. Models could not hold top scores on both skills at once. Structured tools like mask-guided fusion helped far more than plain prompting.
Catalog record
| CREATOR | Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, Chenxin Li, Panwang Pan, Junbin Lu, Jingyan Jiang, Xinghao Ding, Yue Huang, Zhi Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46713 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12746
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, Chenxin Li, Panwang Pan, Junbin Lu, Jingyan Jiang, Xinghao Ding, Yue Huang, Zhi Wang Abstract Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at “thinking in dynamics”, i.e. , perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench , a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies ( e.g. , chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs’ dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/ . Figure 1 : Spatio-temporal dynamics reasoning benchmark Dyn-Bench . It rigorously evaluates multimodal large language models on their ability to perceive , track , and reason about dynamic contents ( i.e. , object motion, evolving scene, camera motion) in the 4D world. 1 Introduction Understanding the dynamics of 4D world, where both spatial structure and semantic content evolve over time, is fundamental to how humans interact with environment. This dynamic perception allows humans to reason about motion, interactions, and temporal changes, seamlessly integrating visual input with temporal context. While humans rely on visual intelligence to navigate and interpret static scenes, their ability to think in dynamics , by tracking and reasoning about evolving scenes through the connections between objects’ movements, interactions, and spatial relationships over time, is what enables them to understand dynamic environments. Although current MLLMs excel in spatial reasoning, their ability to understand spatio-temporal dynamics in evolving scenes remains underexplored. This motivates us to ask: To what extent can MLLMs perceive, track, and reason about the physical 4D world in a similarly coherent manner as humans? Recent advances in general MLLMs 4 ; 30 ; 15 ; 2 ; 45 ; 41 ; 1 ; 38 and spatial reasoning variants 31 ; 26 ; 44 ; 54 ; 13 ; 14 ; 3 ; 6 ; 50 ; 27 have significantly advanced visual intelligence. While these models demonstrate strong semantic and spatial reasoning capabilities on static images, their success does not readily generalize to dynamic video understanding. When applied to temporally evolving scenes, they struggle to maintain consistent representations of moving objects across frames due to challenges such as occlusion, ego-motion, and semantic ambiguity. These limitations often lead to a fragmented understanding of motion and interactions, hindering coherent reasoning in dynamic environments. To address these issues, recent region-level MLLMs 51 ; 24 ; 23 ; 20 ; 53 ; 52 ; 29 ; 39 ; 7 pursue fine-grained, object-centric understanding through explicit region–language alignment, improving localized spatial perception and tracking accuracy. However, such progress remains largely confined to spatial reasoning , without extending to the broader challenge of spatio-temporal reasoning , which requires continuous perception, tracking, and reasoning about dynamic objects over time. Recent benchmarks 56 ; 47 ; 19 ; 22 ; 49 ; 60 have expanded the evaluation of spatio-temporal reasoning beyond static visual understanding to dynamic scene reasoning. However, as summarized in Tab. 1 , most existing efforts focus on scene-level modeling, emphasizing temporal correlations, motion understanding, or scene evolution, but lack systematic evaluation from the perspective of dynamic objects . Consequently, these benchmarks fail to capture the multi-dimensional nature of spatio-temporal reasoning, particularly the ability to track moving objects and delineate their fine-grained motion boundaries over time. Without modeling object-centric dynamics, current evaluations fall short in assessing whether MLLMs can achieve coherent reasoning about motion, causality, and scene evolution in realistic 4D environments. To this end, we introduce Dyn-Bench ( Dyn amics Bench mark), a large-scale benchmark for evaluating the spatio-temporal reasoning and dynamic object grounding abilities of MLLMs in the physical 4D world. As illustrated in Fig. 1 , Dyn-Bench spans three complementary levels of dynamic scene understanding: ① Dynamic Inter-Object Perception , capturing spatial relations and interactions among moving objects; ② Dynamic Object–Scene Tracking , modeling object motion and temporal evolution across scenes; and ③ Dynamic Camera–Object Reasoning , analyzing object behavior under varying camera motions. Each level integrates spatio-temporal reasoning and dynamic object grounding tasks for joint evaluation of perception and reasoning. Constructed from eight real and synthetic datasets, Dyn-Bench contains 1k videos, 7k visual question answering (VQA) pairs, and 3k grounding annotations. We extensively evaluate general, spatial and region-level MLLMs, finding that existing models struggle to balance reasoning and grounding, often showing inconsistent understanding of object motion over time. Building on these findings, we further explore how MLLMs think in dynamics across visual and textual modalities. Our analysis reveals that conventional prompting strategies, such as chain-of-thought or caption-based cues, yield limited gains, whereas structured integration methods, including Mask-Guided Fusion and the proposed ST-TCM, enable more coherent modeling of motion and significantly improve spatio-temporal reasoning on Dyn-Bench . In summary, our contributions are as follows: \bullet We introduce Dyn-Bench , the first benchmark that comprehensively evaluates Dynamic Understanding capability of MLLMs, including spatio-temporal reasoning and dynamic object grounding in realistic 4D environments. \bullet We introduce a novel, carefully curated dataset, constructed from eight real and synthetic sources. After multi-stage filtering from extensive 2D and 4D data, the dataset includes 1k videos, 7k VQA pairs, and 3k high-quality grounding annotations. \bullet We analyze the limitations of current MLLMs in spatio-temporal reasoning, identifying issues such as inconsistent object motion tracking and poor grounding of dynamic objects. We also evaluate the impact of different prompting techniques, including chain-of-thought reasoning and mask-guided methods, demonstrating that structured approaches enhance the coherence of reasoning in dynamic scenes. \bullet We introduce Spatio-Temporal Textual Cognitive Maps (ST-TCM), a novel framework that unifies spatial, temporal, and motion information into a single representation. ST-TCM significantly enhances the ability of MLLMs to reason about dynamic objects and scenes, leading to more consistent and accurate spatio-temporal understanding. Table 1 : Comparison of Dyn-Bench with existing spatio-temporal benchmarks. Dyn-Bench offers a unified assessment covering three levels of dynamic object grounding and reasoning while spanning more diverse data domains than prior works. Numerical Statistics QA Types Detailed Features Benchmark # Videos # QA Pairs # Masklets Inter-Object Object-Scene Camera-Object Object Grounding Scene Type Dynamic Type Real-world? Metric-scale? EgoDynamic4D 12 275 927,000 - ✓ ✓ ✓ ✗ Indoor S.fisheye Mixed Yes Chat4D 58 - 879,100 - ✗ ✓ ✗ ✗