OmniMatBench : A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields
Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang, Jue Wang, Ran Sun, Zhuo Yang, Wanli Ouyang, Lei Bai, Tianfan Fu, Lu Chen, Xin Chen, Yuqiang Li · 2026
Read OmniMatBench : A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields by Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang, Jue Wang, Ran Sun, Zhuo Yang, Wanli Ouyang, Lei Bai, Tianfan Fu, Lu Chen, Xin Chen, Yuqiang Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29833.txt.
Introduction
Wanhao Liu and twelve colleagues present OmniMatBench, a human-calibrated benchmark for multimodal reasoning in materials science. Older benchmarks stress property prediction, knowledge questions, or characterization, leaving the path from knowledge to application thin. The bench holds over three thousand expert questions and problems across nineteen subfields, from basic knowledge to engineering and applied work. Thirteen open and closed models were tested, and the best scored just over a third overall, exposing a wide gap. Scores swung by subfield, with fixed heuristics, patchy knowledge, and weak application when aided.
Catalog record
| CREATOR | Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang, Jue Wang, Ran Sun, Zhuo Yang, Wanli Ouyang, Lei Bai, Tianfan Fu, Lu Chen, Xin Chen, Yuqiang Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76667 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniMatBench : A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang, Jue Wang, Ran Sun, Zhuo Yang, Wanli Ouyang, Lei Bai, Tianfan Fu, Lu Chen, Xin Chen, Yuqiang Li Abstract As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench , a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research. 1 Introduction Multimodal large language models (MLLMs) are increasingly expected to assist scientific research Tang et al. 2026 ; Cui et al. 2025 , yet their reliability in domain-specific scientific reasoning remains unclear. Materials science provides a particularly challenging and valuable testbed because it is inherently interdisciplinary, multimodal, and engineering-application Lendvai and Pegoretti 2024 ; Miret and Krishnan 2025 . However, existing materials-science benchmarks largely focus on property prediction Dunn et al. 2020 ; Tran et al. 2023 , knowledge QA Weng et al. 2025 ; Wu et al. 2025 ; Zaki et al. 2024 , visual recognition Cheung et al. 2025 , characterization understanding Alampara et al. 2025 , or final-answer accuracy Zhang et al. 2025 . However, existing protocols lack systematic evaluation of materials-science contexts spanning knowledge understanding, structure–property interpretation, processing, fabrication, and application decision making. To address this gap, we introduce OmniMatBench , a human-calibrated multimodal reasoning benchmark for materials science. It contains 3,171 expert-curated QA and calculation problems across 19 subfields, organized into four broad domains: Fundamental Materials Science , Structural and Engineering Materials , Materials Processing and Manufacturing , and Functional and Applied Materials . Each problem is paired with a proof record documenting its source, formulation, and solution logic, and is verified by materials-domain experts for correctness and answer consistency. This taxonomy reflects the Knowledge–Structure–Processing–Application (KSPA) perspective, covering both foundational and engineering-oriented topics, including hard-to-collect and important subfields such as powder materials, gem materials and gemology, welding technology and engineering, and nanomaterials and technology. Figure 1: Overview of the OmniMatBench, illustrating the classification of 19 distinct materials-science subfields into four overarching domains. OmniMatBench adopts a fine-grained evaluation design. QA items are annotated with expert-curated key points covering concepts, reasoning elements, scoring criteria, and common errors, while calculation (CAL) items include structured answer slots, formula traces, unit requirements, numerical precision constraints, and output schemas. This enables evaluation beyond final-answer matching and reveals failures in concept identification, formula selection, multimodal grounding, unit-aware computation, and answer formatting. We systematically evaluate 13 proprietary and open-source MLLMs on OmniMatBench . Results show that current MLLMs remain far from reliable materials-science reasoning: the strongest model, Claude Opus 4.7, achieves only 0.372 overall, with 0.443 QA F 1 and 30.10% CAL slot accuracy, while GPT-5.5 reaches 0.324 overall. Further analysis reveals strong subfield variation, fixed reasoning heuristics, uneven materials knowledge, and persistent execution failures even with formula, retrieval, and code assistance. Scientific-domain specialized models can outperform general-purpose models in some subfields, but a broader knowledge-to-execution gap remains. Our contributions are summarized as follows: • We introduce OmniMatBench , a broad-coverage materials-science benchmark with 3,171 expert-curated QA and CAL problems across 19 subfields and 4 domains, covering foundational and engineering-oriented topics. • We design a fine-grained evaluation protocol combining rubric-based QA with strict slot-based CAL evaluation, supported by expert key points, common-error annotations, formula traces, unit requirements, numerical precision constraints, and output schemas. • We evaluate 13 MLLMs on OmniMatBench and analyze their reasoning behaviors, revealing persistent gaps including fixed reasoning heuristics, uneven subfield knowledge, weak processing/application understanding, and limited knowledge application. 2 Related Works Materials Benchmarks and Evaluation Protocols. Materials benchmarks have evolved from predictive modeling to language-based scientific reasoning. Early benchmarks such as MatBench Dunn et al. 2020 , Matbench Discovery Riebesell et al. 2025 , OC20/OC22 Chanussot et al. 2021 ; Tran et al. 2023 , and JARVIS-Leaderboard Choudhary et al. 2024 focus on property prediction, stability screening, atomistic simulation, and materials discovery. Recent LLM and MLLM benchmarks, including MaScQA Zaki et al. 2024 , MSQA Cheung et al. 2025 , MatSciBench Zhang et al. 2025 , MaCBench Alampara et al. 2025 , MatVQA Wu et al. 2025 , MatQnA Weng et al. 2025 , MatCha Lai et al. 2025 , and MATRIX McGrath et al. 2026 , further evaluate materials QA, multimodal reasoning, and characterization understanding. However, most existing protocols emphasize knowledge recall, visual recognition, or final-answer accuracy, offering limited insight into formula selection, unit-aware computation, multi-slot answers, and format-constrained execution. To address these issues, OmniMatBench complements prior work by evaluating the knowledge-to-execution gap with structured slots, formula traces, retrieval settings, and fine-grained error analysis. Table 1: Comparison with materials-science benchmarks. Mod. denotes modality: MM for multimodal and Text for text-only. Numeric denotes free-form numerical or calculation-oriented outputs rather than numerical options in multiple-choice questions. Diff. is assigned by the best reported model score at the time of each benchmark release: Easy ( >65 ), Medium ( 50 – 65 ), and Difficult ( <50 ). KSPA denotes whether the benchmark includes subfields spanning the Knowledge–Structure–Processing–Application perspective. Benchmark #Items Mod. Num. Task Diff. KSPA MATRIX 470 MM ✗ Open QA Medium ✗ MatSciBench 1,340 MM ✓ QA, Cal Easy ✗ MaCBench 1,153 MM ✓ MCQ, Open QA Medium ✗ MSQA 1,757 Text ✗ QA, T/F Easy ✗ MatVQA 1,325 MM ✗ MCQ Medium ✗ MatQnA 4,968 MM ✗ MCQ, Subj. QA Easy ✗ MatCha 1,500 MM ✗ MCQ Medium ✗ OmniMatBench 3,171 MM ✓ QA, Cal Difficult ✓ 3 Dataset and Tasks 3.1 Data Collection and Processing Figure 2: Overview of the OmniMatBench data construction pipeline. OmniMatBench is constructed through a multi-stage, expert-involved data construction pipeline in collaboration with materials-science specialists. As shown in Figure 2 , the source materials are selected from classical and used materials knowledge resources, and the resulting problems are organized to cover 19 subfields aligned with the Knowledge–Structure–Processing–Application perspective. Experts first select, refine, and complete the domain knowledge from the source materials, and then classify each candidate problem into its corresponding subfield. For each problem, they identify the core formulas, common error types, and key concepts based on their domain expertise and experience, and distill them into scoring-oriented key points. Meanwhile, the complete problem provenance, adaptation rationale, and solution logic are organized into the corresponding proof file. The preliminary item pool and proof files are then checked by an LLM verifier to ensure consistency between the question and its visual, tabular, or formula inputs,