Towards Large-scale Chemical Reaction Image Parsing via a Multimodal Large Language Model

Yufan CHEN, Ching Ting LEUNG, Jianwei SUN, Yong HUANG, Linyan LI, Hao CHEN, Hanyu GAO · 2025

Read Towards Large-scale Chemical Reaction Image Parsing via a Multimodal Large Language Model by Yufan CHEN, Ching Ting LEUNG, Jianwei SUN, Yong HUANG, Linyan LI, Hao CHEN, Hanyu GAO (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08156.txt.

Introduction

Yufan Chen and co-authors teach AI to read chemistry diagrams in this 2025 paper. Journals show reactions as pictures, not machine-readable data. Turning those pictures into structured records still falls mostly to human curators. The team built a multimodal language model, RxnIM, made for the task.

The model reads the drawn molecules and the printed conditions in one pass. A new method for building large training sets supports it. On standard benchmarks it scored well above earlier tools. Code, model files, and data were released openly. Large reaction databases could now be built from the back catalog of journals. That would feed better AI tools for drug and materials research.

Catalog record

CREATORYufan CHEN, Ching Ting LEUNG, Jianwei SUN, Yong HUANG, Linyan LI, Hao CHEN, Hanyu GAO
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08156

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Large-scale Chemical Reaction Image Parsing via a Multimodal Large Language Model Yufan CHEN, Ching Ting LEUNG, Jianwei SUN, Yong HUANG, Linyan LI, Hao CHEN, Hanyu GAO Abstract Artificial intelligence (AI) has demonstrated significant promise in advancing organic chemistry research; however, its effectiveness depends on the availability of high-quality chemical reaction data. Currently, most published chemical reactions are not available in machine-readable form, limiting the broader application of AI in this field. The extraction of published chemical reactions into structured databases still relies heavily on manual curation, and robust automatic parsing of chemical reaction images into machine-readable data remains a significant challenge. To address this, we introduce the Reaction Image Multimodal large language model (RxnIM), the first multimodal large language model specifically designed to parse chemical reaction images into machine-readable reaction data. RxnIM not only extracts key chemical components from reaction images but also interprets the textual content that describes reaction conditions. Together with specially designed large-scale dataset generation method to support model training, our approach achieves excellent performance, with an average F1 score of 88% on various benchmarks, surpassing literature methods by 5%. This represents a crucial step toward the automatic construction of large databases of machine-readable reaction data parsed from images in the chemistry literature, providing essential data resources for AI research in chemistry. The source code, model checkpoints, and datasets developed in this work are released under permissive licenses. An instance of the RxnIM web application can be accessed at https://huggingface.co/spaces/CYF200127/RxnIM . 1 Introduction The field of organic chemistry has witnessed a transformative shift with machine learning techniques, enabling significant advancements in retrosynthesis, reaction prediction, and condition recommendation.As researchers increasingly leverage these methodologies to explore complex chemical phenomena, high-quality machine-readable chemical reaction data is essential. Despite the wealth of chemical knowledge documented in the literature,the data required for effective machine learning applications remains largely fragmented and predominantly inaccessible in a format suitable for computational analysis. 1 , 2 , 3 , 4 , 5 . Traditional approaches to chemical reaction data extraction are predominantly manual, involving labor-intensive curation processes that are susceptible to human error and inefficiencies. This reliance on manual extraction limits the scalability of data acquisition as well as the potential for comprehensive analysis across large datasets. There is an urgent need for automated solutions that can accurately and efficiently parse chemical reaction images, transforming them into structured data that can support advanced machine learning applications. Substantial efforts have been devoted to automatic chemical reaction data extraction. 6 , 7 , 8 , 9 , 10 , 11 . For instance, the Pistachio dataset 12 , primarily derived from patent text, utilizes a classic natural language processing pipeline that encompasses syntactic parsing and named entity recognition to identify chemical names 13 , 14 , followed by event extraction to organize these chemicals into reactions. To handle the varied text found in journal articles, Guo et al. developed a deep learning approach 15 that breaks down the task into product extraction and reaction role labeling, utilizing sequence tagging techniques based on pre-trained language models. Ming et al. proposed a reaction extraction system based on large language models (LLMs) 16 , which utilized the natural language understanding ability of LLMs, and expanded the scope of existing predefined reaction roles to include important attributes that have been ignored before, thus providing a more comprehensive and accurate description of chemical reactions. Despite these advancements, existing work primarily focus only on processing textual information. Images serve as a more intuitive medium for documenting chemical reactions, providing clear visualization of molecular structures and the logical flow of multi-component and multi-step reactions. Yet, reaction image parsing remains under-explored, largely due to the complexity of reaction images and the variability of drawing styles. Previous works have attempted to detect the location of molecular objects from images 3 , 4 or recognizing their chemical structures 17 , 18 , 19 , 20 . These methods often struggle to understand the role of different components and their logical connections, which are critical for a comprehensive understanding of the chemical reactions depicted 3 , 4 . For reaction image parsing, Qian et al. developed a single encoder-decoder model 5 closely following Pix2Seq 21 attempting to parse reaction data from images directly via an image-to-sequence translation. This approach demonstrated strong promise, yet it still frequently failed to parse data from images of more complicated reaction patterns. Additionally, the multimodality of reaction information has not been explicitly addressed in previous methods. For example, for the text of reaction conditions and other auxiliary information, existing methods typically rely on external optical character recognition (OCR) tools to recognize the characters, and do not further process the information (e.g., whether the text describes the reagents, solvents, time, temperature, or yield), resulting in less comprehensive final parsed data. Recently, as an essential subset of LLMs 22 , 23 , 24 , 25 , 26 , 27 , multimodal large language models (MLLMs) represent a significant breakthrough in computer vision 28 , 29 , 30 , 31 , 32 , 33 . They have demonstrated impressive capabilities in both traditional visual tasks, such as object detection and instance segmentation, as well as in more complex tasks like referring expression comprehension and generation 34 . Moreover, MLLMs have shown exceptional OCR capabilities across various text-related visual tasks, even without specialized training on relevant datasets. Therefore, MLLMs present a promising solution for parsing data from chemical reaction images, an area that remains largely uncharted in the literature. In this paper, we present Reaction Image Multimodal large language model (RxnIM), the first MLLM specifically designed for chemical reaction image parsing. The model was trained using a three-stage training strategy with a unified language-based task instruction for different chemical reaction image parsing tasks. The first stage was pretraining the model’s object detection capability on a large dataset of synthetic reaction images. In the second stage, the model was trained to identify the reaction components and extract reaction conditions using the synthetic dataset. In the final stage, the model was fine-tuned on a smaller, manually curated dataset to enhance its performance on real reaction images. Unlike previous approaches that predict object types and relationships based on predefined rules 3 , 4 or single-task models 5 , RxnIM comprehends the entire reaction process depicted in the reaction image and integrates various parsing tasks seamlessly and flexibly, resulting in more accurate and comprehensive outputs. The model achieved an average F_{1} of 88% (soft match score, defined in Methods ) across various benchmarks for the reaction component identification task, significantly outperforming the state-of-the-art method by an average of 5%. Additionally, our tests highlighted the model’s superior abilities in interpreting textual information that describes reaction conditions. We further developed a web application that can easily be used and deployed. The web application is hosted at https://huggingface.co/spaces/CYF200127/RxnIM . The source code and data are available at https://github.com/CYF2000127/RxnIM . Since RxnIM was trained on a promising large-scale data and offered as a ready-to-use open-source tool, we believe it will greatly reduce the workload and enable the construction of high-quality datasets for the research community and promote machine-learning-driven innovations in organic chemistry. 2 Results Figure 1: Dataset generation and overview of the proposed RxnIM. a

Read the complete text