Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science
Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Amirreza Ataei, Bang Liu · 2025
Read Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science by Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Amirreza Ataei, Bang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18319.txt.
Introduction
Sifan Wu and co-authors build a benchmark for visual reasoning in materials science in this 2025 paper. Language models that also see images have grown strong at text tasks. Tests in materials science, though, lean on words and miss the visual skill real research needs.
Their benchmark, MatVQA, holds 1,325 questions drawn from recent papers by an automated pipeline. Questions span four tasks that link structure, properties, and performance. An editing loop removed text shortcuts, so models have to read images like microscopy shots closely. Seventeen models were tested, and all showed large gaps. The data and code are public to spur further work.
Catalog record
| CREATOR | Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Amirreza Ataei, Bang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18319
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Amirreza Ataei, Bang Liu Abstract The emergence of Multimodal Large Language Models (MLLMs) that integrate vision and language modalities has unlocked new potentials for scientific reasoning, outperforming prior benchmarks in both natural language and coding domains. Current materials science evaluation datasets such as MaScQA and SciQA remain largely text-based and fail to capture the visual and research-level analytic complexity required in materials discovery and design. We introduce MatVQA, a scalable benchmark specifically designed to address this gap. Generated via an automated pipeline, MArxivAgent, from recent materials literature, MatVQA features 1325 questions across four critical structure-property-performance (SPP) reasoning tasks. Uniquely, MatVQA employs an iterative process to eliminate textual shortcuts, compelling MLLMs to perform fine-grained, low-level visual analysis of material imagery (e.g., microscopy, diffraction patterns) integrated with multi-step scientific reasoning. Benchmarking 17 open- and closed-source MLLMs on MatVQA reveals substantial gaps in current multimodal reasoning capabilities. MatVQA benchmark data, along with evaluation code, is publicly available in https://anonymous.4open.science/r/matvqa-1E01/README.md to catalyze further research in applying MLLMs to complex materials science problems. 1 Introduction Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable success in a variety of applications, including natural language understanding bedi2024testing ; brown2020language ; hollmann2025accurate , code generation jain2024livecodebench ; nam2024using , and diverse scientific domains latif2024physicsassistant ; song2023honeybee ; zhang2024honeycomb ; zhang2024mathverse . Their ability to interpret and reason over complex, multimodal data while integrating specialist domain knowledge is pivotal for tackling challenging problems in fields such as mathematics, medicine, finance, and geospatial sciences. While MLLMs have shown promise in facilitating scientific discovery in areas like biology jung2024llm ; luu2024bioinspiredllm , chemistry jablonka202314 ; zhang2024chemllm , software engineering belzner2023large , and healthcare peng2023study ; nazi2024large , their application to research-level reasoning in highly specialized, multimodal scientific tasks, particularly within materials science, remains less explored. Materials science, an interdisciplinary field at the intersection of physics, chemistry, and often biology, demands extensive domain knowledge and sophisticated reasoning capabilities to understand and design material. Initial efforts have applied Large Language Models (LLMs) to augment materials science research, such as the HoneyComb agent for material-related question answering zhang2024honeycomb . However, these approaches, along with existing materials science datasets like MaScQA zaki2024mascqa , SciQA auer2023sciqa , and MSE-MCQs wang2024evaluating , predominantly focus on text-based question answering. Consequently, they often overlook the critical aspect of visual perception and lack questions that require deep, research-level reasoning. The recent emergence of benchmarks like MicroVQA burgess2025microvqa for biological microscopy underscores the necessity of evaluating MLLMs on research-grade multimodal tasks. To fill this void, we introduce MatVQA. MatVQA is distinguished by its design for easy scalability and its specific focus on assessing two capabilities largely underaddressed by existing benchmarks: the complex, multi-step reasoning constructed from fundamental scientific principles, and the demanding, low-level visual perception required to interpret nuanced experimental data. Figure 1 : Domain attribution for MatVQA MatVQA is built by MArxivAgent , an automated and verifiable pipeline engineered for the efficient generation of challenging multiple-choice questions (MCQs) directly from arXiv materials-science papers. This automated construction underpins MatVQA’s inherent scalability, facilitating ongoing expansion and adaptation. After generation by advanced LLMs, a random 20% of the MCQs are vetted by domain experts to ensure quality. Grounded in real-world literature, the benchmark targets four research-critical structure–property–performance (SPP) tasks—since they map how a material’s structure governs its measurable properties and ultimate performance . These SPP tasks, derived from core components of scientific inquiry, by their very nature necessitate the complex reasoning MatVQA aims to evaluate, requiring tight integration of visual and textual evidence and collectively probing the core cognitive operations of materials research: (1) Quantitative SPP : Rigorously quantify how specific structural descriptors modulate properties and performance, providing the numerical design rules essential for rational materials engineering. (2) Comparative SPP : Systematically contrast competing structures under identical conditions to pinpoint the superior configuration, mirroring the materials-selection step in experimental practice. (3) Causal SPP : Infer mechanistic cause-and-effect links between structure and performance, advancing understanding beyond correlation and enabling mechanism-driven optimisation. (4) Hypothetical variation : Predict the ramifications of untested structural modifications, permitting in-silico exploration of the design space and accelerating discovery. These tasks mirror the questions materials scientists pose when characterising, designing, and optimising materials, and they rigorously test an MLLM’s ability to perform fine-grained visual perception, numerical reasoning, comparative analysis, causal inference, and forward prediction. As shown in Table 1 , MatVQA contains 1325 multiple-choice questions automatically generated by advanced language models and randomly verified by domain experts carefully. Motivated by the forensic study of multimodal biases in burgess2025microvqa , we identify two textual artifacts that can subvert genuine vision–language evaluation: (i) Language shortcuts appear when the prompt provides verbose image descriptions, weak distractors, or stylistic cues that allow the answer to be inferred from the text alone. (ii) Caption shortcuts happen when the information embedded in its stem or options—typically paraphrased from the figure caption—suffices to answer the question without inspecting the figure itself. Caption shortcuts are especially prevalent in materials-science corpora because captions explicitly condense the key morphological, crystallographic, or spectroscopic observations. Their inadvertent reuse limits MCQs to high-level descriptors (e.g., phase label, mean grain size) and excludes the low-level visual cues—diffraction peaks, defect textures, subtle contrast variations—crucial for authentic reasoning. Additional leakage stems from distractors generated with caption vocabulary, reliance on caption-embedded numerical values, and minimal emphasis on spatial or pixel-level patterns. Eliminating both shortcut classes is therefore imperative: it compels models to ground their answers in fine-grained visual evidence and provides a benchmark that more faithfully measures genuine vision–language competence. To excise these artifacts, MArxivAgent executes an iterative shortcut-elimination loop . After initial question synthesis, an evaluator agent answers the MCQ using (a) only the stem and options and (b) the stem, options plus caption but without the image. Success in either mode triggers a rewriter that removes or rephrases the incriminating text while a consistency checker enforces fidelity to the original scientific claim. By progressively eliminating both language and caption shortcuts, we elevate the benchmark from coarse- to fine-grained difficulty: solving the final questions requires precise, low-level visual scrutiny (e.g., counting diffraction spots, discerning lattice fringes) coupled with multi-hop scientific reasoning. This refinement is essential for measuring the true multimodal competence that front-line materials research demands. We evaluated a suite of 17 open- and closed-source MLLMs on MatVQA and compared the performance of a select subset against human experts and vision-language model baselines. In summary, our key contributions are: • We release MatVQA , the first benchmark designed to evaluate research-level multimodal reasoning in varies domain of materials science. • We propose four Structure-Property-Performance (SPP) tasks that encapsulate core scientific inquiries regarding material structure, properties, and performance. • We design MArxivAgent , a fully automated, three-stage pipeline that (i) extracts reasoning paths from scientific literature, (ii) iteratively eliminates language shortcuts, and (iii) subsequently removes caption shortcuts, producing high-difficulty, visually grounded MCQs. This work aims to foster the development of MLLMs capable of contributing meaningfully to materials science research by providing a challenging and relevant evaluation standard. Table 1 : Overview of multimodal science benchmarks and detailed MatVQA attributes (in (b), SPP represents structure–property–performance ). (a) Comparison with current