Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

Yuyang Wu Yue Huang Shuaike Shen Xujian Wang, Shuhao Zhang Qiyao Xue Weichen Liu Runtian Gao, Jian Ma Xiangliang Zhang Olexandr Isayev · 2026

Read Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning by Yuyang Wu Yue Huang Shuaike Shen Xujian Wang, Shuhao Zhang Qiyao Xue Weichen Liu Runtian Gao, Jian Ma Xiangliang Zhang Olexandr Isayev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07251.txt.

Description

Yuyang Wu and ten co-authors test whether AI agents can price out a chemical reaction from supplier catalogs. Agents must identify chemicals, fetch quotes, pick valid purchasable pack sizes, normalize amounts, and compute a total. Their ChemCost benchmark grounds more than fourteen hundred reactions in a frozen snapshot holding thousands of chemicals and vast numbers of quotes.

Scoring needs no judge model, and each stage of the task can be diagnosed separately, from parsing to arithmetic. Noise-injected versions perturb names, quantities, and formatting. Even the strongest agents reached only about half accuracy on clean inputs, within a generous error margin. They fell further with realistic noise, exposing brittle tool use.

Catalog record

CREATORYuyang Wu Yue Huang Shuaike Shen Xujian Wang, Shuhao Zhang Qiyao Xue Weichen Liu Runtian Gao, Jian Ma Xiangliang Zhang Olexandr Isayev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07251

Browse the full library · About the Tower of Babel