PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments
Weijie Zhou Xuantang Xiong Yi Peng Manli Tao, Chaoyang Zhao Honghui Dong Ming Tang Jinqiao Wang · 2025
Read PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments by Weijie Zhou Xuantang Xiong Yi Peng Manli Tao, Chaoyang Zhao Honghui Dong Ming Tang Jinqiao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21111.txt.
Introduction
Weijie Zhou leads seven co-authors on active visual reasoning for multimodal models in physical environments, where views are partial and occluded rather than static and complete. People solve such problems by moving, examining, and handling objects in a loop of perception and action.
The Active Visual Reasoning task formalizes that loop for models, demanding careful information gathering through interaction before answering questions. A system named PhysVLM-AVR is built for the setting. Benchmarking active exploration matters because real robots and assistants rarely enjoy the tidy full views that standard vision tests provide.
Catalog record
| CREATOR | Weijie Zhou Xuantang Xiong Yi Peng Manli Tao, Chaoyang Zhao Honghui Dong Ming Tang Jinqiao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments Weijie Zhou Xuantang Xiong Yi Peng Manli Tao, Chaoyang Zhao Honghui Dong Ming Tang Jinqiao Wang Abstract Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness in real-world environments where information is often incomplete due to occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment—moving, examining, and manipulating objects—to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR , a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k , a large-scale dataset offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR , an MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities. 1 1 footnotetext: Equal contribution 2 2 footnotetext: Corresponding authors 1 Introduction Figure 1: Passive Visual Reasoning (top) fails with partial views. Active Visual Reasoning (AVR, bottom) actively interacts to gather information for a correct answer. Visual reasoning, a fundamental capability in artificial intelligence, has enabled multimodal large language models (MLLMs) qwen2vl ; llava ; llavanext ; internvl ; claude3 to excel at tasks such as object counting CLEVR ; cleve-math ; SuperCLEVR and visual question answering (VQA) Visionr1 ; VisionR1zhan ; Visualprogramming ; vqav2 . These abilities are crucial for developing intelligent systems that can comprehend and interact with the visual world. As MLLMs continue to evolve, their potential to understand and reason about dynamic, real-world environments is increasingly recognized, paving the way for more sophisticated and autonomous agents robobrain ; roboos ; physvlm ; physical1 . Despite these advancements, current MLLM approaches to visual reasoning—including prominent tasks like VQA okvqa ; vqav2 ; eqa1 ; eqa2 and spatial reasoning spatialvlm ; spatialbot ; geminirobotics —predominantly rely on static, passive visual inputs. This inherently limits their applicability in real-world physical environments where information is often partially observable due to occlusions, containment, or limited field of view physical2 . As illustrated in Figure 1 , a passive model observing only the initial image I_{0} answers incorrectly, whereas an active agent interacts to uncover occluded objects across I_{1} and I_{2} , arriving at the correct answer. And in the task of counting all object categories in the scene, the more occluded objects there are, the worse the model performs in passive visual reasoning mode. This highlights that effective reasoning in partially observable settings necessitates interaction to actively acquire missing information. Humans, in stark contrast, naturally overcome such limitations through active perception. We instinctively move, change viewpoints, and manipulate objects to gather information needed for a specific goal. This active engagement exemplifies a core aspect of human cognition: closing the perception-reasoning-action loop to incrementally resolve uncertainties and establish the causal links essential for complex reasoning. However, existing computational frameworks often suffer from paradigm fragmentation and thus fail to holistically model the complete perception-reasoning-action loop. Conventional visual reasoning tasks Reasonrft ; Visionr1 ; VisionR1zhan (e.g., CLEVR CLEVR ) assume complete visual inputs, requiring only a single "perception → reasoning" step without active information gathering mmbench ; mmvet ; mmebenchmark . While embodied question answering (EQA) tasks (e.g., OpenEQA openeqa , RoboVQA robovqa ) involve interaction, they often focus on reasoning from single, albeit potentially long, video sequences. Similarly, embodied exploration methods askforhelp ; embodiedReasoner ; Knowledge-based-eqa , also interactive, tend to prioritize task completion metrics (e.g., navigation success) over the explicit information gain needed for nuanced reasoning. The fundamental limitation across these approaches is their failure to effectively link reasoning with the strategic, sequential actions essential for information gathering. Specifically, they often overlook how an agent’s actions dynamically alter available information and, crucially, how these changes should guide and refine the ongoing reasoning process, especially when multiple evidence-gathering steps are required. Inspired by human active perception and to address these limitations, we introduce the Active Visual Reasoning (AVR) task. AVR extends visual reasoning to partially observable, interactive environments, bridging the gap between passive observation and active, embodied understanding. Specifically, AVR combines embodied interaction with temporal visual reasoning, requiring agents to: (1) actively gather information through sequential physical actions; (2) integrate multi-step observations for coherent reasoning; and (3) dynamically adjust decisions based on incremental visual feedback. This paradigm requires models to interpret visual data and strategically decide how to interact with the environment, thereby resolving uncertainties crucial for complex reasoning. To establish a rigorous foundation for the AVR task and facilitate research in this domain, this paper makes the following primary contributions: • We formally introduce and define the Active Visual Reasoning (AVR) task, mandating active information gathering, multi-step integration, and dynamic decision-making in partially observable settings. • We develop CLEVR-AVR , a simulation benchmark featuring multi-round interactive environments. This benchmark is specifically designed to assess both the reasoning correctness and the information-gathering efficiency of agents performing AVR tasks. • We present AVR-152k , a large-scale dataset with multi-level annotations designed to train agents for AVR. Its core component, AVR-Core, models the task as a higher-order Markov Decision Process and features rich Chain-of-Thought (CoT) annotations. These CoTs articulate the structured, iterative reasoning humans employ for active information seeking, including critical steps such as: (i) identifying information uncertainty, (ii) predicting action-conditioned information gain, and (iii) selecting information-maximizing actions, thereby providing explicit supervision for these decision-making processes. • We develop and evaluate PhysVLM-AVR , an MLLM that achieves state-of-the-art performance on CLEVR-AVR while maintaining strong results on standard embodied visual reasoning tasks, and reveals insights into current MLLM limitations regarding active interaction. Collectively, our work provides a foundation for developing MLLMs capable of actively reasoning about and intelligently interacting with their physical environments, thereby narrowing the gap between static visual reasoning and embodied intelligence. 2 Related Work Visual Reasoning. Traditional visual reasoning tasks, such as Insight-V insightv , CLEVR CLEVR and its variants (e.g., CLEVR-Math cleve-math , MathVision awais2024mathvision ), have significantly advanced models’ abilities in attribute recognition, counting, and spatial reasoning robospatial . However, these tasks operate on static, fully observable images, lacking mechanisms for active information seeking in partially observable scenarios. Consequently, they do not model the sequential, interactive process essential for real-world understanding. AVR addresses this by requiring agents to perform sequential actions to actively acquire information necessary for reasoning. Embodied Question Answering. Embodied Question Answering (EQA) tasks (e.g., OpenEQA openeqa , RoboVQA robovqa , CityEQA cityeqa ) extend question answering to simulated embodied environments. However, by often relying on pre-captured data, these approaches limit agents to