Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
Yitong Zhou, Mingyue Cheng, Qingyang Mao, Yucong Luo, Qi Liu, Yupeng Li, Xiaohan Zhang, Deguang Liu, Xin Li, Enhong Chen · 2025
Read Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables by Yitong Zhou, Mingyue Cheng, Qingyang Mao, Yucong Luo, Qi Liu, Yupeng Li, Xiaohan Zhang, Deguang Liu, Xin Li, Enhong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11375.txt.
Introduction
Yitong Zhou and co-authors build a test set that grades AI models on chemistry tables, in a 2025 paper. Tables pack much of science into grids of text, symbols, and drawings. Chemistry tables are the hardest kind: they mix reagents, conditions, and yields with molecule sketches and formulas. A model must line up words and images to read them at all.
Their benchmark, ChemTable, draws real tables from published papers. Experts marked each cell's place, the grid's logic, and domain labels. Two tasks follow: pull the table out whole, and answer questions that call for description or reasoning. Top models parsed layouts fairly well but stumbled on molecule drawings and symbol rules. Even the best closed models trailed human readers by a wide gap. The authors offer the set as a proving ground for science AI.
Catalog record
| CREATOR | Yitong Zhou, Mingyue Cheng, Qingyang Mao, Yucong Luo, Qi Liu, Yupeng Li, Xiaohan Zhang, Deguang Liu, Xin Li, Enhong Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables Yitong Zhou, Mingyue Cheng, Qingyang Mao, Yucong Luo, Qi Liu, Yupeng Li, Xiaohan Zhang, Deguang Liu, Xin Li, Enhong Chen Abstract Chemical tables encode complex experimental knowledge through symbolic expressions, structured variables, and embedded molecular graphics. Existing benchmarks largely overlook this multimodal and domain-specific complexity, limiting the ability of multimodal large language models to support scientific understanding in chemistry. In this work, we introduce ChemTable, a large-scale benchmark of real-world chemical tables curated from the experimental sections of literature. ChemTable includes expert-annotated cell polygons, logical layouts, and domain-specific labels—including reagents, catalysts, yields, and graphical components—and supports two core tasks: (1) Table Recognition, covering structure parsing and content extraction; and (2) Table Understanding, encompassing both descriptive and reasoning-oriented question answering grounded in table structure and domain semantics. We evaluated a range of representative multimodal models, including both open-source and closed-source models, on ChemTable and reported a series of findings with practical and conceptual insights. Although models show reasonable performance on basic layout parsing, they exhibit substantial limitations on both descriptive and inferential QA tasks compared to human performance, and we observe significant performance gaps between open-source and closed-source models across multiple dimensions. These results underscore the challenges of chemistry-aware table understanding and position ChemTable as a rigorous and realistic benchmark for advancing scientific reasoning 1 1 1 https://github.com/lqzxt/ChemTable . 1 Introduction Recent advances in multimodal large language models (MLLMs) have created new opportunities for mining expert knowledge from scientific literature and are increasingly viewed as catalysts for AI-driven scientific discovery [ 1 ] . A growing wave of scientific accelerators —such as MLLMs-based OCR, ChatPaper [ 2 ] , and ChatPDF [ 3 ] —demonstrates the potential of MLLMs to streamline literature parsing, summarization, and interactive reading. From a modeling perspective, MLLMs exhibit strong capabilities in semantic understanding and reasoning, making them well-suited for processing the rich multimodal content of scientific documents [ 4 , 5 ] . Benchmarks like ChartQA [ 6 ] and ChartX [ 7 ] have begun exploring visual reasoning over scientific figures. Yet, scientific literature remains one of the most semantically dense and domain-specialized corpora, and understanding it serves as both a valuable application and a rigorous testbed for evaluating the limits of MLLMs [ 8 ] . While recent benchmarks have primarily focused on figures and charts, tables remain a largely underexplored yet equally critical modality in scientific literature . In chemistry, tables are concise and information-rich representations of experimental setups, reaction conditions, and empirical results. These tables often combine symbolic expressions, structured variables, and graphical elements—posing significant challenges for existing MLLMs. Despite their importance, there is a lack of realistic, domain-specific datasets designed to evaluate the capabilities of MLLMs in scientific table understanding. This gap motivates our work: to build a comprehensive benchmark that captures the structural complexity, semantic richness, and reasoning demands of real-world chemical tables. Figure 1 : Multimodal Complexity of Chemical Tables and Challenges for MLLMs. Among scientific tables, those found in chemistry pose especially complex challenges that go far beyond standard layout parsing [ 9 ] . A typical chemical table encodes rich experimental workflows through dense symbolic notation (e.g., reagents, ligands), domain-specific abbreviations (e.g., “BINAP”, “TFA”), and embedded visual elements such as molecular structures or reaction schemes [ 10 , 11 ] . As illustrated in Figure 1 , a single row often represents a multi-variable configuration—catalyst, ligand, solvent, additive, with quantitative outcomes like yield, selectivity. These tables also rely on implicit conventions (e.g., ratio formats like “ > 19/1”) and footnoted exceptions, making their semantics both subtle and compact. Interpreting such tables requires aligning symbolic, numeric, and visual information in a domain-aware manner , which presents significant challenges for general-purpose MLLMs not trained on scientific representations [ 8 , 9 ] . To systematically address these challenges, we introduce ChemTable , a high-quality benchmark designed for recognition and understanding over real-world chemical tables. The dataset comprises over 1,300 tables curated from peer-reviewed chemistry literature, spanning diverse reaction types, experimental conditions, and reporting formats. ChemTable supports two core tasks: table Recognition [ 12 ] —including structure reconstruction and content extraction—and table Understanding [ 13 , 14 ] , formulated through more than 9,000 QA instances across two categories: (1) descriptive questions , which evaluate a model’s ability to extract key facts; and (2) reasoning questions , which require comparison, attribution, and domain-grounded inference. To enable scalable and consistent evaluation, all answers follow a short-form format suitable for MLLMs-based automatic grading. We benchmark seven widely-adopted MLLMs, including representative models from both open- and closed-source sources, and observe significant performance gaps across all tasks. Compared to human and expert performance, current MLLMs lag behind—particularly in interpreting symbolic and embedded graphical elements. Our analysis further reveals key insights, including the symbolic understanding gap, the limited transferability of MLLMs to scientific domains, and substantial variation across model families. We release ChemTable, along with evaluation tools, to support future research in multimodal scientific understanding. 2 Related Work 2.1 Existing Benchmarks on Table Recognition. Existing table recognition benchmarks mainly focus on two tasks: Table Structure Recognition (TSR) and Table Recognition (TR). The widely used ICDAR-2013 [ 15 ] dataset includes 257 manually annotated tables for TSR. Despite its small size, it remains a key benchmark due to its annotation quality. Since 2019, large-scale TR datasets [ 16 , 17 , 18 , 19 ] have emerged. However, their annotations are programmatically generated and provide only table structures without cell content, limiting their utility in deeper understanding tasks. To address this, FinTabNet [ 20 ] and PubTabNet [ 21 ] incorporate logical cell locations, enhancing TSR modeling. SciTSR [ 22 ] focuses on scientific documents, offering detailed cell content annotations, thus better supporting academic table recognition. 2.2 Existing Benchmarks on Table Understanding. Several datasets have been proposed for table understanding tasks, including WikiTQ [ 23 ] and WikiSQL [ 24 ] . These datasets consist of numerous tables extracted from Wikipedia and serve as critical early benchmarks in table understanding research. In recent years, advancements in natural language processing have extended beyond traditional homogeneous tables to incorporate additional modalities. For instance, HybridQA [ 25 ] and MMTab [ 26 ] introduce multi-modal or semi-structured sources for more complex reasoning. Moreover, domain-specific benchmarks such as FinQA [ 27 ] (financial) and SciTab [ 28 ] (scientific) address unique challenges by integrating structured tables with related textual content to support complex reasoning tasks. Table 1: Comparison of Table Understanding Datasets. We use the following shorthand: Dom. Spe. = Domain Specific, Pict. Moda. = Pictorial Modality, Text. Moda. = Textual Modality, Human Writ. = Human Written, LLM Gene. = LLM Generated. Table Modal Question Source Name Dom. Pict. Text Human LLM Spe. Moda. Moda. Writ. Gene. WikiTQ [ 23 ] ✗ ✗ ✓ ✓ ✗ WikiSQL [ 24 ] ✗ ✗ ✓ ✓ ✗ HybridQA [ 25 ] ✗ ✗ ✓ ✓ ✗ FinQA [ 27 ] ✓ ✗ ✓ ✓ ✗ SciTab [ 28 ] ✓ ✗ ✓ ✓ ✓ MMTab [ 26 ] ✗ ✓ ✗ ✓ ✓ ChemTable ✓ ✓ ✓ ✓ ✓ The ChemTable dataset is different from existing benchmarks in several key ways. It