ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025

Qiang Xu, Shenyuan BaiLeqing ChenZijing Liu · 2025

Read ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025 by Qiang Xu, Shenyuan BaiLeqing ChenZijing Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16205.txt.

Introduction

Qiang Xu, Shengyuan Bai, and three co-authors present ChemO, a benchmark for advanced AI thinking in chemistry. It is built from International Chemistry Olympiad 2025 problems. Chemistry uses a unique multimodal symbolic language of structures and equations. Olympiad tests in math and physics left it as an open test. Two new ideas make automated scoring possible for problems whose answers are drawings or other visual outputs.

Assessment-Equivalent Reformulation turns visual-output problems into formats a computer can grade. A visual aid step splits a model vision from its chemical thinking, for diagnosis. Their ChemLabs multi-agent system mimics expert teams, with agents for splitting the work, perception, thinking, and checking. The top setup scored near the top of the scale, above the judged human gold medal mark.

Catalog record

CREATORQiang Xu, Shenyuan BaiLeqing ChenZijing Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16205

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025 Qiang Xu, Shenyuan BaiLeqing ChenZijing Liu Abstract Olympiad-level benchmarks in mathematics and physics are crucial testbeds for advanced AI reasoning, but chemistry, with its unique multimodal symbolic language, has remained an open challenge. We introduce ChemO, a new benchmark built from the International Chemistry Olympiad (IChO) 2025. ChemO features two key innovations for automated assessment: Assessment-Equivalent Reformulation (AER), which converts problems requiring visual outputs (e.g., drawing molecules) into computationally tractable formats, and Structured Visual Enhancement (SVE), a diagnostic mechanism to disentangle a model’s visual perception capabilities from its core chemical reasoning. To tackle this benchmark, we propose ChemLabs, a hierarchical multi-agent framework that mimics human expert collaboration through specialized agents for problem decomposition, perception, reasoning, and auditing. Experiments on state-of-the-art multimodal models demonstrate that combining SVE with our multi-agent system yields dramatic performance gains. Our top configuration achieves a score of 93.6 out of 100, surpassing an estimated human gold medal threshold and establishing a new state-of-the-art in automated chemical problem-solving. ChemO Dataset: https://huggingface.co/datasets/IDEA-AI4SCI/ChemO 1 Introduction Recent advances in Multimodal Large Language Models (MLLMs) have led to agents achieving gold-medal performance in scientific Olympiads for mathematics He et al. 2024a and physics Qiu et al. 2025 , demonstrating superhuman capabilities in complex reasoning. However, the field of chemistry presents a unique and arguably more complex frontier that remains largely unsolved. Unlike other domains, chemistry relies on a dense, symbolic visual language where parsing molecular structures, interpreting reaction schemes, and analyzing spectral data are intertwined with textual context and quantitative calculations. Existing benchmarks like ChemBench Gu et al. 2024a do not adequately capture this deep, multimodal reasoning challenge, leaving a critical gap in evaluating and advancing expert-level MLLM. To address this, we introduce ChemO , a new benchmark built from the International Chemistry Olympiad (IChO) 2025. A core challenge in creating ChemO was handling problems requiring visual outputs (e.g., drawing a molecule). Our solution is Assessment-Equivalent Reformulation (AER) , a principled methodology that transforms such problems into formats where models can generate machine-readable outputs ( e.g. , SMILES strings) rather than visual drawings, while preserving the original assessment criteria. This reformulation enables models to leverage their text generation capabilities to solve problems that would otherwise require visual output modalities. Furthermore, to diagnose model weaknesses, ChemO introduces an optional Structured Visual Enhancement (SVE) setting, which provides models with structured textual descriptions of visual elements in the problem ( e.g. , molecular structures, diagrams), helping to isolate performance bottlenecks between visual perception and core chemical reasoning. Inspired by how teams of human experts collaborate, we propose ChemLabs , a hierarchical multi-agent system designed to master the ChemO benchmark. As illustrated in Figure 1 , ChemLabs decomposes monolithic reasoning into a structured workflow with iterative refinement: a central Manager Agent decomposes problems and dispatches sub-tasks to specialized modules, including a Perception Lab to interpret chemical diagrams, a Solving Lab with domain-specific reasoners, and a two-stage Audit Lab to verify the chemical and logical integrity of proposed solutions. Through multi-agent deliberation and self-correction loops, ChemLabs iteratively refines solutions until they meet rigorous verification criteria. The main contributions of this work are: • We present ChemO , the first benchmark to formalize IChO problems. We introduce AER , a methodology that reformulates visual output problems into machine-readable formats to enable text-generation-based solving, and SVE , which provides structured textual descriptions of visual elements to isolate visual perception from chemical reasoning capabilities. • We propose ChemLabs , a hierarchical multi-agent framework that implements a divide-and-conquer strategy with iterative refinement and self-correction mechanisms for complex, multimodal scientific reasoning. • We demonstrate through extensive experiments that ChemLabs, particularly when augmented with SVE and powered by Gemini-2.5 Pro, achieves a score of 93.6/100 on ChemO, surpassing an estimated human gold-medal threshold and setting a new state-of-the-art. Our findings indicate that accurate visual perception is the primary bottleneck for MLLMs, and that SVE combined with ChemLabs successfully bridges this gap to achieve gold-medal-level performance. 2 Related Work 2.1 Multimodal Reasoning Agents Recent advances in MLLMs have enabled AI systems to jointly reason over visual and textual inputs. Foundation models such as GPT-4V ( Achiam et al. 2023 ) , Gemini 2.5 Comanici et al. 2025 , and Claude 4.5 ( cla 2025 ) have demonstrated strong performance in visual question answering and diagram understanding, while efficient open-source alternatives like MiniCPM-V ( Yao et al. 2024 ) and Qwen3-VL Yang et al. 2025a now achieve comparable results. To enable autonomous problem-solving, agent frameworks like ReAct Yao et al. 2022 integrate reasoning with tool use through iterative thought-action-observation cycles. Modern systems such as LangChain ( lan 2025 ) and AutoGPT Aut 2025 provide modular architectures for composing agents with planning, memory, and tool integration capabilities. Recent work Cherian et al. 2023 has focused on multimodal algorithmic reasoning, where agents must deduce solution strategies for vision-and-language puzzles requiring mathematical and logical skills. Domain-specific implementations like ChemCrow Bran et al. 2023 have shown that augmenting LLMs with expert-designed tools significantly improves performance on scientific reasoning tasks, motivating our approach for chemistry olympiad problem-solving. ID Problem Topic Field #Sub Problem Type Difficulty Representation P1 Sesquiterpene ozonolysis and rearrangements Organic Chemistry 9 SC(5), QI(4) Hard rxn, str, mech P2 Rapamycin stereoselective total synthesis Organic / Biochemistry 4 SC(3), QI(1) Hard rxn, str, mech P3 Pd(II) lantern self-assembly structures Inorganic Chemistry 7 SC(1), QI(2), TE(4) Hard str, graph P4 Tennis ball pressurization kinetics Physical Chemistry 4 QC(2), MR(2) Easy text, calc P5 Solar desalination energy calculations Physical Chemistry 8 QC(8) Easy text, calc P6 CO 2 reduction pathways comparison Physical Chemistry 7 QI(2), QC(4), MR(1) Easy text, calc, graph P7 Crude oil GC-MS fragmentation Analytical Chemistry 5 SC(2), QI(1), QC(2) Medium ms, str P8 Catalytic converter CO oxidation Inorganic / Physical 9 SC(2), QI(4), TE(1), QC(2) Hard rxn, ir, str, graph P9 Thiamine-dependent enzyme pathways Biochemistry 6 SC(4), QI(1), MR(1) Hard rxn, str, mech Total 59 SC(17), QI(15), TE(5), QC(18), MR(4) Table 1 : Overview of the 9 problems in the ChemO benchmark. Problem Types : Structure Construction (SC), Quantitative Calculation (QC), Qualitative Identification (QI), Tabular Enumeration (TE), and Mechanistic Reasoning (MR). Representations in problem : rxn = reaction scheme, str = molecular structure, ms = mass spectrum, ir = IR spectrum, calc = numerical calculation, graph = graphical data, mech = mechanism arrows, text = descriptive text. 2.2 Olympiads as Benchmarks As standard benchmarks become saturated, Olympiad-level competitions have emerged as rigorous testbeds for evaluating advanced reasoning in LLMs and MLLMs ( Mahdavi et al. 2025 ; Petrov et al. 2025 ; Khatibi et al. 2025 ; Shi et al. 2024 ; Zou et al. 2025 ; Chen et al. 2025 ; Zheng et al. 2025 ) . In mathematics, recent systems have achieved gold medal performance at IMO 2025, marking significant progress in formal reasoning capabilities ( He et al. 2024b ; Li et al. 2025 ) . For physics, several benchmarks have been proposed to assess multimodal physical reasoning. OlympiadBench ( He et al. 2024b ) introduces bilingual multimodal scientific problems spanning mathematics and physics at competition level. More recently, HiPhO ( Yu et al. 2025 ) provides 13 contemporary physics Olympiad exams

Read the complete text