S e e P h y s : Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning
Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han,, Hang Xu, Hanhui Li, Mrinmaya Sachan, Xiaodan Liang · 2025
Read S e e P h y s : Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning by Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han,, Hang Xu, Hanhui Li, Mrinmaya Sachan, Xiaodan Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19099.txt.
Introduction
Kun Xiang and co-authors present SeePhys in a 2025 paper, a benchmark that tests whether vision helps language models reason about physics. Their questions run from middle school problems to doctoral qualifying exams. Seven core domains and twenty one kinds of diagram are covered. In most past benchmarks, pictures merely support the text. Here, three quarters of the problems cannot be solved without reading the diagram.
Even the strongest current vision models scored below sixty percent on the benchmark. The failures cluster in two places, the authors find. Models struggle to bind what a diagram shows to the physics reasoning that must follow. They also lean on text cues as shortcuts instead of truly reading the image. The benchmark is offered as a hard test of grounded visual reasoning for the next round of models.
Catalog record
| CREATOR | Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han,, Hang Xu, Hanhui Li, Mrinmaya Sachan, Xiaodan Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
S e e P h y s : Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han,, Hang Xu, Hanhui Li, Mrinmaya Sachan, Xiaodan Liang Abstract We present SeePhys , a large-scale multimodal benchmark for LLM reasoning grounded in physics questions spanning across middle school to PhD candidacy exams. The benchmark covers 7 fundamental physics domains, incorporating 21 categories of highly heterogeneous diagrams. In contrast to prior works where visual elements mainly serve supplementary purposes, our benchmark features a substantial proportion of vision-essential problems (75%) that mandate visual information extraction for problem-solving. Through extensive evaluation, we observe that even the best-performing reasoning models (e.g., Gemini-2.5-pro and OpenAI-o4-mini) achieve sub-60% accuracy on our benchmark. These results reveal fundamental challenges in current large language models’ visual understanding capabilities, particularly in: (i) establishing meaningful connections between diagram interpretation and physics reasoning, and (ii) overcoming their persistent reliance on textual cues as cognitive shortcuts. Project Page: https://seephys.github.io/ Hugging Face: https://huggingface.co/datasets/SeePhys/SeePhys 1 Introduction While mathematical reasoning has been a cornerstone for evaluating the reasoning capability of large language models (LLMs) and multimodal large language models (MLLMs) [ 8 , 14 , 11 , 18 , 21 , 6 , 23 , 5 , 50 , 42 , 45 ] , the realm of natural science, especially the discipline of physics, remains understudied as a even more diverse testbed for complex scientific reasoning. Physics reasoning inherently binds text explanations to real-world visual contexts, exposing critical gaps in their ability to emulate human-like world modeling in the context of physics problem-solving [ 10 , 24 , 51 ] . Frontier models have begun to demonstrate abstract perception of physical scenarios and logical reasoning capabilities [ 25 , 12 , 29 , 30 , 9 ] . Due to the intrinsic diversity of physics diagrams and the fact that they inherently reflect the law of physics in our world, developing a comprehensive benchmark for evaluating physics reasoning abilities and cross-modal understanding is crucial for enhancing LLMs. Early research primarily focused on assessing physical commonsense [ 4 ] and basic scientific knowledge [ 22 ] , which was later gradually extended to competition-level and university-level physics problems [ 34 , 10 ] . Due to the broad knowledge scope and high annotation difficulty inherent, some studies only considered test samples from limited knowledge levels [ 52 , 13 , 48 ] . Furthermore, other works targeted the evaluation of language models and did not incorporate visual information [ 33 , 10 , 46 ] . Notably, compared to the mathematics domain, there has been limited exploration of models’ perceptual differences in processing physics diagrams, despite their richer topological structures. Figure 1: Overview of SeePhys . It encompasses 7 core physics domains and 21 diagram types, spanning the full knowledge spectrum from middle school to PhD candidacy exams levels. With these challenges in mind, we introduce SeePhys , to measure LLMs’ capability to visually understand the law of physics. It is a fully multimodal benchmark for evaluating physics reasoning across middle school to PhD qualifying exam levels. SeePhys comprises 2,000 rigorously validated questions collected from open-source textbooks, exercises, examinations, and competitions. These questions span 7 major fields of both classical and modern physics. To assess the extent to which different models rely on visual information for reasoning, we curate two subsets with different degrees of visual information enrichment and additionally compile supplementary copies of 2,000 purely visual instances where all problem statements in texts are presented in picture form. Through meticulous selection of 21 diagram types by domain experts, each problem challenges frontier MLLMs to integrate domain knowledge with visual understanding of physics diagrams (e.g., Feynman diagrams for particle interactions and Circuit diagrams for Electromagnetism). With SeePhys , we conduct extensive experiments to evaluate 28 leading LLMs and MLLMs such as o4-mini [ 30 ] and Gemini-2.5-Pro [ 9 ] . The results reveal that even with extensive chain-of-thought, none of the current models could surpass 55% accuracy. Our analysis reveals that even non-essential diagrams can enhance physics reasoning performance when presented to MLLMs. We have also raised a challenge 1 1 1 https://www.codabench.org/competitions/7925/ in the 2nd AI for Math Workshop at ICML 2025 2 2 2 https://sites.google.com/view/ai4mathworkshopicml2025/ . Our main contributions are summarized as follows: • We propose a purely multimodal benchmark spanning multiple knowledge levels and domains. The meticulously curated benchmark comprises 2,000 rigorously annotated questions paired with 2,245 images. • By prioritizing physics’ unique blend of observation and theoretical deduction, we assess how well models emulate the way human scientists observe, deduce, and understand complex natural phenomena. Our findings reveal significant gap in models’ capabilities to leverage multimodal information. • We conduct a comprehensive evaluation of current LLMs and MLLMs, followed by an in-depth analysis of their failure modes. Results demonstrate that even frontier models struggle with physics problems across different knowledge levels in various visual contexts. Figure 2: Examples of Vision-Optional/Vision-Essential questions. In Vision-Optional samples, texts provide sufficient visual descriptions (e.g., graphical attributes and spatial relationships) to help respondents with illustration. In Essential samples, images contain indispensable problem-solving information, such as numerical values for key variables and unspecified topological structures. 2 Related Work Math reasoning benchmarks. Mathematical reasoning has emerged as a central testbed for evaluating LLMs. Early benchmarks like GSM8K [ 8 ] established the foundation for multi-step textual reasoning through elementary problems, while MATH [ 14 ] advanced the field with competition-level tasks (e.g., AMC/AIME), exposing critical limitations of early models. As these benchmarks achieved saturation, the community shifted toward higher complexity, e.g., introducing Olympiad-level challenges requiring formal theorem proving and combinatorial reasoning [ 11 , 18 ] . Concurrently, the rise of multimodal reasoning spurred benchmarks such as MathVista [ 21 ] and MATH-V [ 42 ] to integrate visual comprehension (e.g., diagrams, plots) with compositional reasoning. However, MathVerse [ 50 ] has found that MLLMs tend to rely on the reasoning capabilities of language models when performing mathematical tasks. In contrast, scientific diagrams, which are abstractions derived from real-world scenarios, with their complex visual features, may provide a more effective testbed for benchmarking models’ multimodal capabilities. Physics Reasoning benchmarks. Contemporary physics benchmarks target broad scope of domains, with frontier datasets now encompassing: (1) PhD candidacy exam problems, and (2) Olympiad-style challenges. Text-based physics benchmarks like PHYBench [ 33 ] and UGPhysics [ 46 ] provide challenging problems that test advanced reasoning skills, yet fundamentally lack the visual components necessary to assess diagram interpretation abilities. Multimodal physics benchmarks such as PhysReason [ 52 ] , OlympiadBench [ 13 ] , and PHYSICS [ 10 ] emphasize visual reasoning challenges without analysis regarding the extent of visual components’ impact. Moreover, constrained by the high annotation costs stemming from the extensive domain knowledge required and the scarcity of qualified multimodal materials, these datasets lack comprehensive coverage across knowledge hierarchies and detailed annotations of diagram types. To address these limitations, we contribute a dataset with fully multimodal and full-spectrum physics problems. 3 SeePhys 3.1 Data Collection Principles Table 1: Comparison of SeePhys and related datasets in physics. Mid: Middle school; High: High school; Olympiad: Beginner/advanced