ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

Qiang Xu, Shengyuan Bai, Yu Wang, Leqing Chen, Yuanyuan Liu Bin Feng Zijing Liu Yu Li · 2026

Read ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams by Qiang Xu, Shengyuan Bai, Yu Wang, Leqing Chen, Yuanyuan Liu Bin Feng Zijing Liu Yu Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15994.txt.

Introduction

Xu and eight colleagues introduce ReactBench, a benchmark testing topological reasoning in multimodal models on chemical reaction diagrams. Such models handle single elements and linear diagrams well, but falter on branching paths, converging flows, and cyclic dependencies, even counting endpoints. Reaction diagrams span linear chains to cyclic graphs and demand local recognition and global reasoning. With 1,618 expert-annotated question-answer pairs across four levels, tests on 24 models show a gap over 30 percent between anchor-based tasks and holistic structural reasoning. That gap marks a reasoning bottleneck, not perception.

Catalog record

CREATORQiang Xu, Shengyuan Bai, Yu Wang, Leqing Chen, Yuanyuan Liu Bin Feng Zijing Liu Yu Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34830
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams Qiang Xu, Shengyuan Bai, Yu Wang, Leqing Chen, Yuanyuan Liu Bin Feng Zijing Liu Yu Li Abstract Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies, their reasoning capabilities degrade sharply, even on tasks as basic as counting endpoints. Existing benchmarks fail to probe this gap, focusing on semantic comprehension rather than structural reasoning. We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams. These real-world scientific diagrams offer an ideal testbed because they naturally span diverse structures from linear chains to cyclic graphs, while requiring both precise local recognition and coherent global reasoning. Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions. Extensive evaluation across 17 MLLMs reveals a significant performance gap exceeding 30% between anchor-based tasks and holistic structural reasoning tasks. Controlled ablations confirm this bottleneck lies in reasoning, not perception. These findings expose a fundamental deficit in structural understanding and establish directions for advancing visual reasoning. Figure 1: Comparing General Diagram and Chemical Reaction Diagram on Element Localization Tasks. GPT-4o performs well on general diagrams for identification tasks. However, for chemical reaction diagram, the structural complexity of molecular diagrams and reaction pathways leads to errors. 1 Introduction Figure 2: Overview of ReactBench. ReactBench systematically evaluates topological reasoning on chemical reaction diagrams across four complexity-stratified dimensions. Recent advances in MLLMs have demonstrated remarkable capabilities in visual-linguistic reasoning across diverse domains 21 ; 2 ; 30 ; 3 . These models excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, real-world diagrams often encode information through complex topological structures involving branching paths, converging flows, and cyclic dependencies. The capacity of MLLMs to reason about such structural connections remains largely unexplored, as existing benchmarks 19 ; 28 ; 18 ; 17 primarily evaluate semantic comprehension rather than structural reasoning capabilities. We illustrate this gap through a controlled comparison in Fig. 1 . Both tasks require the same topological operation of identifying terminal nodes in a diagram. When presented with a simple linear topology such as a sequential coffee brewing process, GPT-4o correctly identifies the single endpoint. However, when the same task is posed on a branching chemical network, the model fails systematically, misidentifying six endpoints instead of the actual three. Notably, neither task requires domain expertise. The failure is attributable to an inability to reason about complex branching structures rather than a lack of chemical knowledge. These observations reveal a fundamental limitation in current MLLMs. The ability to perceive local elements does not guarantee comprehension of global structure. As topological complexity increases, models exhibit systematic failures in integrating visual elements into coherent structural representations. Systematically diagnosing this limitation requires a testbed that combines diverse topological complexity with demands for both local recognition and global reasoning. Chemical reaction diagrams satisfy these criteria. They naturally span structures from linear chains to cyclic graphs, and demand accurate local perception of molecules and conditions alongside coherent global understanding of reaction pathways. We introduce ReactBench , a benchmark designed to diagnose structural reasoning limitations in MLLMs through chemical reaction diagrams. ReactBench comprises 1,618 question-answer pairs carefully curated from real-world chemistry literature and patents, organized into four hierarchical task dimensions: spatial element localization, topological information extraction, pathway connectivity tracing, and structural topology reasoning. This design enables precise identification of where model capabilities degrade along the perception-to-reasoning spectrum. Our main contributions are as follows: • We present ReactBench, the first benchmark that systematically diagnoses structural reasoning limitations in MLLMs through hierarchical task decomposition, comprising 1,618 expertly annotated question-answer pairs across four skill dimensions. • We establish empirical evidence for a fundamental capability gap in current MLLMs, revealing a significant performance dichotomy across task dimensions. Accuracy exceeds 80% on information extraction and path tracing, but drops below 55% on element counting and topology classification, indicating that structural complexity is a primary determinant of task difficulty. • Through controlled ablations across multiple model families, we demonstrate that this bottleneck lies in reasoning rather than perception, and identify specific failure modes in structural understanding, providing concrete directions for improving visual reasoning in future architectures. 2 Related Work Category Number Question Dimensions Spatial Element Localization 835 (52%) Topological Information Extraction 414 (26%) Pathway Connectivity Tracing 167 (10%) Structural Topology Reasoning 202 (12%) Question types Numerical questions 1094 (67%) Multiple-choice questions 202 (13%) Free-form questions 322 (20%) Total 1618 Table 1: Key statistics of ChemReaction. Figure 3: Composition of ChemReaction. 2.1 Multimodal Large Language Models MLLMs extend text-only LLMs by adding visual comprehension capabilities. Early works like CLIP 25 established vision-language alignment through contrastive learning. Recent MLLMs mainly follow two design paradigms. The first includes encoder-decoder architectures like BLIP-2 14 that use separate visual and textual encoders with a Q-Former 36 for cross-modal fusion. The second encompasses decoder-only architectures, such as LLaVA 16 , that project visual features directly into the LLM input space. These models are typically trained through a multistage process involving pre-training on image-text pairs and instruction tuning on high-quality multimodal datasets. Recent advances include KOSMOS-1 12 , which supports multimodal in-context learning, and Qwen2.5-VL 3 and GPT-4V 22 , which demonstrate strong performance in various vision language tasks, including visual reasoning and OCR understanding. 2.2 Visual Question Answering (VQA) Benchmarks Visual Question Answering (VQA) benchmarks have been instrumental in driving the development and evaluation of Multimodal Large Language Models. Existing datasets like DocVQA 19 and ChartQA 18 primarily evaluate visual information extraction capabilities across relatively simple, standardized layouts. On the other hand, science-oriented benchmarks such as ScienceQA 17 and AI2D 11 test domain-specific scientific knowledge and basic diagrammatic understanding, yet they predominantly target non-structural or shallow visual reasoning. In these scenarios, visual inputs typically act as static semantic references rather than intricate relational networks. Consequently, while current VQA benchmarks heavily focus on general semantic comprehension and basic grounding, they lack a systematic evaluation of complex topological reasoning. Our work addresses this critical gap by specifically isolating and evaluating multi-hop, graph-theoretic reasoning within dense structural topologies. 2.3 Chemical Reaction Diagram Understanding Existing approaches to chemical reaction diagram understanding employ multi-stage pipelines that first perform Optical Chemical Structure Recognition (OCSR) to detect individual molecular structures, then convert detected elements into text representations such as SMILES 27 ; 9 ; 23 ; 6 . Representative methods include RxnScribe 23 and ReactionDataExtractor 2.0 33 . Such pipeline-based frameworks are fundamentally incompatible with end-to-end MLLMs, which directly process visual inputs without intermediate symbolic conversion. Furthermore, text-based representations discard spatial positioning and pathway directionality, precluding differentiation between recognition errors and reasoning deficits 10 ; 20 ; 26 . In contrast, ReactBench evaluates MLLMs’ topological reasoning capabilities through visual question-answering tasks, enabling direct assessment of structural understanding without pipeline-induced confounds. 3 The ReactBench Benchmark 3.1 Overview We introduce ReactBench , a specialized benchmark designed to evaluate MLLMs’ ability to comprehend and reason about chemical reaction diagrams. ReactBench consists of a high-quality dataset ChemReaction with 1,618 rigorously curated QA pairs derived from real-world chemical reaction diagrams, and a comprehensive evaluation framework assessing four key dimensions of reaction diagram

Read the complete text