FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models

Xuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, Huimin Zhao · 2025

Read FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models by Xuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, Huimin Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01055.txt.

Introduction

Xuan Liu and four co-authors built a large test set for reasoning about molecules by parts. Their paper, from twenty twenty-five, deals with language models in chemistry. Most sets ask about whole molecules and skip the role of parts called functional groups. Those parts link shape to text in ways that make model logic clearer. The team built FGBench, a set of six hundred twenty-five thousand problems at the group level.

The tasks ask models to tie group traits to molecule traits. Such fine data can teach models hidden links between parts and whole molecule acts. Readers in drug design get a bench to train and grade structure aware models. Sharp group level logic could speed the hunt for new compounds.

Catalog record

CREATORXuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, Huimin Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01055

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models Xuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, Huimin Zhao Abstract Large language models (LLMs) have gained significant attention in chemistry. However, most existing datasets center on molecular-level property prediction and overlook the role of fine-grained functional group (FG) information. Incorporating FG-level data can provide valuable prior knowledge that links molecular structures with textual descriptions, which can be used to build more interpretable, structure-aware LLMs for reasoning on molecule-related tasks. Moreover, LLMs can learn from such fine-grained information to uncover hidden relationships between specific functional groups and molecular properties, thereby advancing molecular design and drug discovery. Here, we introduce FGBench, a dataset comprising 625K molecular property reasoning problems with functional group information. Functional groups are precisely annotated and localized within the molecule, which ensures the dataset’s interoperability thereby facilitating further multimodal applications. FGBench includes both regression and classification tasks on 245 different functional groups across three categories for molecular property reasoning: (1) single functional group impacts, (2) multiple functional group interactions, and (3) direct molecular comparisons. In the benchmark of state-of-the-art LLMs on 7K curated data, the results indicate that current LLMs struggle with FG-level property reasoning, highlighting the need to enhance reasoning capabilities in LLMs for chemistry tasks. We anticipate that the methodology employed in FGBench to construct datasets with functional group-level information will serve as a foundational framework for generating new question–answer pairs, enabling LLMs to better understand fine-grained molecular structure–property relationships. The dataset and evaluation code are available at https://github.com/xuanliugit/FGBench . 1 Introduction Large Language Models (LLMs) have become increasingly popular in the chemistry domain, with applications in areas like molecular property prediction [ Liu et al. 2024 , Jablonka et al. 2024 , Qian et al. 2023 ] , molecule captioning [ Li et al. 2024 , Edwards et al. 2022 ] , and molecule generation [ Ye et al. 2025 ] . However, most existing work concentrates primarily on molecule-level predictions (Figure 1 a). This is due to the nature of current databases that typically only provide molecule-level labels, such as MoleculeNet [ Wu et al. 2018 ] and CHEMBL [ Gaulton et al. 2012 ] . Although molecule caption datasets like PubChem [ Kim et al. 2023 ] offer some functional group information associated with molecules, the data available is often ambiguous and incomplete. Figure 1: Illustration of LLMs for two molecular property prediction tasks, (a) molecular property prediction and (b) molecular property reasoning. Functional groups (FGs) are specific groups of atoms within molecules that impart unique physical and chemical properties [ Hanson 2001 ] . Scientists summarize these properties by identifying and naming these atom groups to better understand molecular behavior and interactions. For example, hydroxyl groups (-OH) are known for their polarity and ability to form hydrogen bonds, while carboxylic groups (-COOH) play a critical role in numerous organic reactions, e.g., esterification and thioesterification. Functional groups thus serve as valuable, transferable knowledge, helping humans understand relationships between molecular structures and their properties. Leveraging this knowledge can enhance LLMs by improving their understanding, prediction accuracy, and interpretability of molecular properties. Recently, there has been growing interest in incorporating functional group-level information into molecular representation pretraining [ Li et al. 2023 , Nguyen et al. 2024 , Sun et al. 2021 ] . Other studies utilizing molecular fragments in pretraining have also achieved notable results in molecular representation [ Luong and Singh 2023 ] . Although these studies primarily treat functional groups as tokens for learning molecular representations, their results underscore the importance of detailed, fine-grained molecular information in chemistry-related tasks. While existing corpora contain extensive descriptions of functional groups and their chemical roles, current molecular property databases lack explicit links between functional groups and molecular properties. This disconnect prevents LLMs from leveraging the rich, fine-grained chemical knowledge embedded in textual data. Bridging this gap by constructing a functional group-centered dataset would enable more interpretable and property-aware LLMs, ultimately advancing their reasoning capabilities in chemistry-related tasks. Reasoning in chemistry with LLMs is particularly challenging, as it demands a deep understanding of molecular structures and their relationships with specific properties (Figure 1 b). When scientists try to give a reasonable prediction on the properties of a target molecule, they often rely on functional groups and similarities to known molecules, following a three-step process: (i) Associate similar molecules , where scientists identify similar molecules from databases to retrieve their properties; (ii) Observe the difference , where they note functional group differences between the target and reference molecules; and (iii) Rephrase the problem , where they infer properties of the target molecule using prior knowledge of functional groups and the reference molecule’s overall structure. The third step is crucial for property reasoning, as functional groups provide important theoretic basis to study the structure-activity relationship (SAR). However, constructing datasets that support this type of functional group–driven reasoning (step iii) remains a substantial challenge. It requires accurately annotating functional groups in the molecules and identifying the functional group difference between two molecules. Moreover, molecule asymmetry, two-dimensional structural information, and isomerism must be carefully considered. For example, isopropanol (CH 3 CH 2 CH 2 OH) and propanol ((CH 3 ) 2 CHOH) have the same hydroxyl (-OH) functional group, but they have different properties due to different positions of this functional group. Consequently, existing annotation methods used in previous studies [ Li et al. 2023 , Nguyen et al. 2024 , Sun et al. 2021 ] are insufficient for these tasks as they only provide the functional group annotation. To better explore and enhance the reasoning capabilities of LLMs in chemistry, we introduce FGBench, a novel dataset designed specifically for molecular property reasoning at the functional group level. FGBench contains 625K molecular property reasoning problems across eight different molecular properties, each accompanied by detailed functional group information, including precise positional data. The tasks in FGBench are organized into three dimensions: (1) single functional group impact, (2) multiple functional group interactions, and (3) molecular comparisons. Within each dimension, two categories of question-answer (QA) pairs are provided: Boolean and value-based. Boolean QA pairs assess the model’ s ability to recognize trends associated with changes in functional groups, while value-based QA pairs evaluate the model’s capability to predict exact quantitative changes. Each QA pair includes clear, detailed instructions for editing molecules at the functional group level. To build FGBench, we developed a new data processing pipeline incorporating a validation-by-reconstruction strategy, which can be generalized to other molecular property datasets to ensure high-quality molecular comparisons. Furthermore, we benchmarked six state-of-the-art open-source and closed-source LLMs using a selected subset of 7K data points from FGBench. Our benchmarking provides valuable insights into the current strengths and limitations of chemical reasoning in LLMs, highlighting critical areas for improvement. In summary, our contributions are: • A novel data processing pipeline using a validation-by-reconstruction strategy for reliable functional group-level molecular comparisons, broadly applicable to various molecular datasets. • FGBench, a dataset of 625K molecular property QA pairs with detailed functional group annotations and precise positions, facilitating advanced LLM fine-tuning, chemical reasoning tasks, and SAR analyses. To our knowledge, FGBench is the first dataset explicitly targeting functional group-level molecular property reasoning. • A benchmarking

Read the complete text