PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model

Sinin Zhang, Yunfei Xie, Yuxuan Cheng, Haoyu Zhang, Tong Zhang · 2026

Read PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model by Sinin Zhang, Yunfei Xie, Yuxuan Cheng, Haoyu Zhang, Tong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24443.txt.

Introduction

Sinin Zhang and four colleagues present PhysNote, an agentic framework helping vision-language models reason about physics. Models fail through identity drift, losing object identity across frames and breaking causal chains. They also fail through volatile inference-time insights that never consolidate for reuse. PhysNote stabilizes perception with canonicalization, organizes insights into a knowledge repository, and grounds hypotheses in visual evidence before consolidating verified knowledge. Tests on PhysBench reach about 57 percent overall accuracy, about 5 percent above the best multi-agent baseline, with gains across all four domains.

Catalog record

CREATORSinin Zhang, Yunfei Xie, Yuxuan Cheng, Haoyu Zhang, Tong Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34080
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model Sinin Zhang, Yunfei Xie, Yuxuan Cheng, Haoyu Zhang, Tong Zhang Abstract Vision-Language Models (VLMs) have demonstrated strong performance on textbook-style physics problems, yet they frequently fail when confronted with dynamic real-world scenarios that require temporal consistency and causal reasoning across frames. We identify two fundamental challenges underlying these failures: (1) spatio-temporal identity drift, where objects lose their physical identity across successive frames and break causal chains, and (2) volatility of inference-time insights, where a model may occasionally produce correct physical reasoning but never consolidates it for future reuse. To address these challenges, we propose PhysNote, an agentic framework that enables VLMs to externalize and refine physical knowledge through self-generated “Knowledge Notes.” PhysNote stabilizes dynamic perception through spatio-temporal canonicalization, organizes self-generated insights into a hierarchical knowledge repository, and drives an iterative reasoning loop that grounds hypotheses in visual evidence before consolidating verified knowledge. Experiments on PhysBench demonstrate that PhysNote achieves 56.68% overall accuracy, a 4.96% improvement over the best multi-agent baseline, with consistent gains across all four physical reasoning domains. 1 Introduction Building autonomous agents that can act safely in the physical world requires physical reasoning : the ability to connect observations with the causal mechanisms that generate motion, contact, and state changes over time. Humans acquire this ability early as intuitive physics and apply it effortlessly, yet replicating it in artificial systems remains an open challenge. This capability links visual perception to prediction and control, and it is therefore central to reliable embodied interaction, from robotic manipulation to autonomous navigation. Recent Vision-Language Models (VLMs) have shown strong performance on static scene understanding and textbook-style physics problems 11 ; 24 . With increased pre-training scale, models 1 ; 10 ; 7 can often infer mathematical solutions from single-frame schematics and recognize fundamental physical concepts. However, much of this progress reflects competence in “text-based physics” 17 ; 25 , where the task is presented as a symbolic or linguistic problem and the supervision signal is contained in the prompt. In such settings, correct answers can be produced without constructing a temporally grounded representation of the underlying physical process 6 ; 22 . This limitation becomes clear when the input shifts from static images to multi-frame sequences that contain temporal evolution and causal interaction among objects 14 . In these dynamic settings, the model must track objects across time, detect contact events, and maintain consistent causal chains, capabilities that static, text-based tasks never require. Even state-of-the-art VLMs often fail at these requirements. An extensive evaluation of 75 VLMs on PhysBench 6 shows that most models achieve only around 40% accuracy on physical reasoning tasks, far below human performance, with perceptual errors and knowledge gaps accounting for the majority of failures. Notably, this deficit does not diminish with larger model size, more training data, or additional input frames, suggesting that physical understanding is not an emergent property of scale alone. Objects frequently lose their physical identity across successive frames, and causal chains can be broken by hallucinatory transitions. These findings indicate that current VLMs lack mechanisms to reliably anchor perception and reasoning in both time and space, which limits their applicability to embodied tasks that demand precise physical inference, such as manipulation where misidentifying an object or misjudging a trajectory can cause operational failures 6 . We identify two fundamental challenges that underlie these failures in dynamic physical reasoning: (1) Spatio-Temporal Identity Drift in Perception and (2) Volatility of Inference-time Insights . For the first challenge, dynamic physical reasoning requires object permanence and traceability of states across time. However, standard VLMs frequently exhibit “identity drift,” in which the semantic representation of an object shifts or disappears across successive frames. Without an explicit mechanism for spatio-temporal anchoring, a causal chain, such as a collision that leads to a particular trajectory, becomes fragmented, which prevents the model from forming a coherent representation of scene dynamics. For the second challenge, human physicists do not treat every new problem as a blank slate; they externalize insights, refine heuristics, and consolidate knowledge into reusable notes. In contrast, existing reasoning paradigms in VLMs are largely volatile . While a model may occasionally generate profound physical insights during a specific reasoning trace, these insights are transient and are not consolidated into a persistent knowledge store. Consequently, the model remains a “goldfish-memory physicist,” failing to evolve its internal world model through experience or self-reflection. To address these challenges, we introduce PhysNote , an agentic framework that enables VLMs to externalize and refine physical knowledge through self-generated “Knowledge Notes.” PhysNote is designed to support dynamic reasoning by iterating over observation of a sequence, retrieval of relevant prior notes, and synthesis of updated insights that remain available for future problems. PhysNote comprises three core components. To mitigate Spatio-Temporal Identity Drift in Perception, PhysNote employs a Spatio-Temporal Canonicalization module (Sec. 3.1 ) that assigns immutable numeric identifiers to visual tokens and enforces a Triadic Observation Template , providing stable visual anchors for object permanence across temporal sequences. To mitigate Volatility of Inference-time Insights, PhysNote introduces a Hierarchical Knowledge Architecture (Sec. 3.2 ) that externalizes self-generated insights into a three-tier repository spanning domain-level strategies and task-specific procedural details. This repository is updated through autonomous pruning and reflection, which preserves high-fidelity knowledge while systematically discarding erroneous biases. Connecting these two components, an Agentic Reasoning engine (Sec. 3.3 ) drives an iterative Hypothesis-Evidence-Validation loop: the agent gathers grounded visual evidence through the canonicalized anchors, consults the Knowledge Notes for relevant physical heuristics, and consolidates verified insights back into the repository after each problem. Experiments on PhysBench 6 show consistent gains for open-source models. On the large-scale test set, PhysNote achieves 56.68% overall accuracy, a 4.96% absolute improvement over the best multi-agent baseline, with balanced gains across all four physical domains. On the validation set, PhysNote reaches 72.86%, exceeding the Qwen2.5-VL-72B baseline by 3.01%. Ablation studies confirm that the synergy between the InfoAgent and Knowledge Notes is essential: neither component alone fully accounts for the gain. Qualitative analysis further shows that PhysNote prevents identity drift in dynamic scenes and enables the model to detect fine-grained physical cues that standard VLMs overlook. 2 Related Work 2.1 Physical Reasoning Models The quest for physical intelligence in AI has evolved from symbolic engines to data-driven neural architectures. Early works primarily focused on Physics-Specialized Models 9 ; 8 , which utilize neural networks to approximate differential equations or Lagrangian dynamics. While precise, these models are often task-specific (e.g., fluid dynamics or rigid-body collisions) and require structured representations that are unavailable in raw visual inputs. Recently, the focus has shifted to Vision-Language Models (VLMs) as generalist observers 20 . However, these models often suffer from Spatio-Temporal Identity Drift 22 . VLMs lack spatial reasoning and temporal grounding, causing objects to lose their physical identity across sequences. In contrast, PhysNote addresses this by replacing implicit visual attention with a deterministic Spatio-Temporal Canonicalization module, ensuring that visual anchors remain consistent throughout the reasoning trajectory. 2.2 Agent-Based Physical Understanding The integration of Vision-Language Models (VLMs) into Autonomous Agents has enabled models to tackle complex physical tasks via iterative prompting, such as Chain-of-Thought 19 or ReAct paradigms 23 . To enhance physical grounding, specialized frameworks like PhysAgent 6

Read the complete text