Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models
Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, Rahul G. Krishnan · 2024
Read Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models by Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, Rahul G. Krishnan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08619.txt.
Introduction
Vahid Balazadeh and co-authors teach big vision-language models physics by giving them written scene reports, in a 2024 paper. Such models often fail to predict how objects will move or fall, since they cannot turn what they know into physical forecasts. Retraining the big models for each task is costly, so the team trains small specialist models instead.
These Physics Context Builders are fine-tuned to write detailed physical descriptions of a scene, which then guide the larger model. On two test sets, one with rolling and colliding objects and one with falling towers in both simulated and real scenes, accuracy rose by up to about fourteen points. Skills learned in simulation also carried over to real scenes.
Catalog record
| CREATOR | Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, Rahul G. Krishnan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, Rahul G. Krishnan Abstract Physical reasoning remains a significant challenge for Vision-Language Models (VLMs). This limitation arises from an inability to translate learned knowledge into predictions about physical behavior. Although continual fine-tuning can mitigate this issue, it is expensive for large models and impractical to perform repeatedly for every task. This necessitates the creation of modular and scalable ways to teach VLMs about physical reasoning. To that end, we introduce Physics Context Builders (PCBs), a modular framework where specialized smaller VLMs are fine-tuned to generate detailed physical scene descriptions. These can be used as physical contexts to enhance the reasoning capabilities of larger VLMs. PCBs enable the separation of visual perception from reasoning, allowing us to analyze their relative contributions to physical understanding. We perform experiments on CLEVRER and on Falling Tower, a stability detection dataset with both simulated and real-world scenes, to demonstrate that PCBs provide substantial performance improvements, increasing average accuracy by up to 13.8% on complex physical reasoning tasks. Notably, PCBs also show strong Sim2Real transfer, successfully generalizing from simulated training data to real-world scenes. 1 Introduction Physical reasoning is a fundamental component of human intelligence, enabling interpretation of complex interactions, prediction of future events, and understanding of causal relationships in real-world environments [ 22 ] . For humans, the ability to understand the physical world is developed early and operates intuitively [ 30 , 3 , 38 ] . However, physical reasoning remains a significant challenge for artificial intelligence (AI) systems [ 23 , 35 , 45 , 10 ] , despite advances in computer vision and Vision-Language Models (VLMs) [ 32 , 2 , 25 , 46 , 36 , 26 , 39 ] . VLMs are remarkably successful on many predictive problems, spanning a broad range of tasks. This makes their use central to real-world applications requiring expertise in a variety of tasks, such as robotics and embodied AI [ 28 , 17 ] . However, current VLMs consistently fail at physical reasoning tasks, struggling with basic spatial relationships (e.g., object positioning, counting) [ 33 ] , object attributes [ 40 ] , and physical interactions (e.g., stability assessment, dynamics prediction) [ 14 , 10 ] . While humans leverage causal or physically guided knowledge for physical understanding of the world [ 5 ] , the mechanisms by which VLMs make predictions are not well understood. This leads to two important questions: what factors contribute to the lack of physical understanding in VLMs, and how can we improve it? One potential explanation for such limitations lies in VLMs’ training data. VLMs are created by fusing the representations of image and text encoders using datasets like MSCOCO [ 27 ] and Conceptual Captions [ 34 ] , which focus on general scene descriptions but lack annotations of physical relationships. Our experiments with standard benchmarks like CLEVRER [ 45 ] confirm this hypothesis, demonstrating that fine-tuning on physics-focused data can enable a relatively small VLM to approach state-of-the-art results achieved by specialized architectures [ 45 , 11 ] . However, as the ecosystem of open and closed source models grows, repeatedly fine-tuning each VLM for every regime of physical concepts it cannot reason about becomes impractical; there is a need for practical, performant and modular tools to augment VLM capabilities. To this end, we introduce Physics Context Builders (PCBs): specialized VLMs that are fine-tuned on simulation data to generate fine-grained physical descriptions that can be used by larger VLMs. Simulation can provide precise annotations of controllable physical interactions of interest, which can serve as scalable training data to teach models about physical phenomena. We show how to embed the knowledge of physical concepts into PCBs and how they can transfer this knowledge to larger VLMs; this enables us to effectively separate visual perception from reasoning, using PCBs as perception modules, while leaving reasoning to larger models. See Fig. 1 for a demonstration of PCBs. Figure 1 : Physics Context Builders (PCBs) pipeline. The training phase (top) shows how physics simulators generate images/videos with corresponding annotations, which are converted into two types of physical descriptions: Human-like Narration (providing natural language scene descriptions with object properties and spatial relationships) and Structured Physics (offering frame-by-frame structured descriptions with physical properties). These descriptions serve as training data for fine-tuning a relatively small VLM into specialized PCBs. During the inference phase (bottom), a trained PCB processes a new image/video and generates detailed physical context about the scene, which is then provided to a foundation model (e.g., GPT-4o) alongside a user query to produce physically grounded responses. PCBs offer a practical and modular approach to enhance the physical reasoning of existing large-scale VLMs without requiring expensive or infeasible modification of the larger foundation models. Our experiments on standard benchmarks demonstrate effective integration of PCBs with standard commercial VLMs. Moreover, they show that PCBs trained with simulation data successfully generalize to real-world scenarios, effectively performing simulation-to-reality (Sim2Real) transfer. In summary, our contributions are: 1. We introduce a modular framework where specialized VLMs are fine-tuned to generate detailed physical scene descriptions, enhancing the physical reasoning capabilities of foundation models without their modification. 2. We demonstrate the effectiveness of separating visual perception from reasoning through our PCB approach, providing insights into how each component contributes to physical understanding. 3. We show how simulation can be leveraged to train specialized modules that transfer successfully to real-world scenarios, avoiding expensive simulations at inference. 2 Related Work 2.1 Physical Reasoning in Vision Models Physical reasoning presents significant challenges for vision models, including both specialized architectures and large-scale VLMs. Recent benchmarks demonstrate that current VLMs struggle with basic physical understanding, often failing at tasks such as counting, depth reasoning, and physical interaction prediction [ 14 , 26 , 39 ] . These limitations are not unique to VLMs; physical reasoning has also proven difficult for single-purpose models, necessitating specialized architectures such as physics-inspired predictive models [ 4 , 16 , 13 ] , neural-symbolic executors [ 9 , 20 , 48 ] , differentiable physics engines [ 42 , 12 ] , simulation-in-the-loop approaches [ 29 , 49 ] , and task-specific architectures [ 11 ] . However, large VLMs are not easily amenable to architectural modifications and require alternative approaches to enhance their capabilities, which we consider in this work. 2.2 Evaluating Physical Reasoning in VLMs Several recent works have evaluated physical reasoning capabilities in vision-language models. Nagar et al. [31] benchmark zero-shot visual reasoning in both large language models (LLMs) and VLMs. They find that underlying LLMs, when provided with textual scene descriptions, consistently outperform VLMs that use visual embeddings. Their analysis shows that this performance gap is due to VLMs’ difficulty in translating visual information into accurate representations for reasoning. Our work builds on this insight by developing a modular approach to bridge this gap without requiring extensive retraining of VLMs. Concurrently, Chow et al. [10] introduce PhysBench, a comprehensive benchmark for evaluating physical understanding in VLMs. Their work reveals that VLMs’ physical reasoning does not scale proportionally with model size, training data, or input frame count. They identify perceptual and knowledge gaps as the