Pre-trained Molecular Language Models with Random Functional Group Masking

Tianhao Peng, Yuchen Li, Xuhong Li, Jiang Bian, Zeke Xie, Ning Sui, Shahid Mumtaz, Yanwu Xu, Linghe Kong, Haoyi Xiong · 2024

Read Pre-trained Molecular Language Models with Random Functional Group Masking by Tianhao Peng, Yuchen Li, Xuhong Li, Jiang Bian, Zeke Xie, Ning Sui, Shahid Mumtaz, Yanwu Xu, Linghe Kong, Haoyi Xiong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01401.txt.

Description

Abstract Recent advancements in computational chemistry have leveraged the power of trans-former-based language models, such as MoLFormer, pre-trained using a vast amount of simplified molecular-input line-entry system (SMILES) sequences, to understand and predict molecular properties and activities, a critical step in fields like drug discovery and materials science.

Catalog record

CREATORTianhao Peng, Yuchen Li, Xuhong Li, Jiang Bian, Zeke Xie, Ning Sui, Shahid Mumtaz, Yanwu Xu, Linghe Kong, Haoyi Xiong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pre-trained Molecular Language Models with Random Functional Group Masking Corresponding to Haoyi Xiong via [email protected] Tianhao Peng, Yuchen Li, Xuhong Li, Jiang Bian, Zeke Xie, Ning Sui, Shahid Mumtaz, Yanwu Xu, Linghe Kong, Haoyi Xiong Abstract Recent advancements in computational chemistry have leveraged the power of trans-former-based language models, such as MoLFormer, pre-trained using a vast amount of simplified molecular-input line-entry system (SMILES) sequences, to understand and predict molecular properties and activities, a critical step in fields like drug discovery and materials science. To further improve performance, researchers have introduced graph neural networks with graph-based molecular representations, such as GEM, incorporating the topology, geometry, 2D or even 3D structures of molecules into pre-training. While most of molecular graphs in existing studies were automatically converted from SMILES sequences, it is to assume that transformer-based language models might be able to implicitly learn structure-aware representations from SMILES sequences. In this paper, we propose MLM-FG – a SMILES-based M olecular L anguage M odel, which randomly masking SMILES subsequences corresponding to specific molecular F unctional G roups to incorporate structure information of atoms during the pre-training phase. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities. Extensive experimental evaluations across 11 benchmark classification and regression tasks in the chemical domain demonstrate the robustness and superiority of MLM-FG . Our findings reveal that MLM-FG outperforms existing pre-training models, either based on SMILES or graphs, in 9 out of the 11 downstream tasks, ranking as a close second in the remaining ones. Remarkably, MLM-FG also surpasses 3D-graph-based models, which explicitly incorporate molecular structures into their inputs, highlighting its exceptional capacity for representation learning even without explicit 3D structural information. These results indicate that MLM-FG effectively captures the nuanced language of molecules, offering promising implications for computational chemistry and related disciplines. 1 Main While deep learning has been widely explored in cheminformatics with significant progress [ 1 , 2 ] , her potential is severely limited by the scale of labeled data. To reduce the cost of data annotation while enabling generalizable, transferable, and robust representation learning from unlabeled data, researchers extend the pre-training strategies [ 3 ] from images and texts to molecular data [ 4 , 5 , 6 , 7 , 8 ] . To work with machine learning algorithms, molecules can be represented by a chemical notation language named SMILES [ 9 ] , which explicitly represents meaningful substructures such as branches and cyclic structures. To enable pre-training, researchers use the variant of Transformers to pre-train on large-scale unlabeled SMILES strings [ 10 , 8 , 11 ] . To pre-train a molecular language model, given the SMILES string of every molecule in the training dataset, existing methods usually adopt an masked autoencoding strategy [ 8 , 10 , 11 , 12 ] . It first randomly selects a subsequence of the SMILES string. Then the strategy masks the selected part and trains models to predict the masked part. However, such random masking strategy would ignore the key chemical substructures of molecules, such as rings and functional groups [ 7 , 4 ] . For instance, consider aspirin, which is denoted by “O=C(C)Oc1ccccc1C(=O)O”. In this molecular structure, critical functional groups such as the carboxylic acid (“-COOH”) and the ester (“-COO-”) are at risk of being overlooked due to random masking. This oversight neglects their pivotal contributions to the molecular activity and properties. To the end, these methods may fail to learn the critical molecular properties, which are primarily relevant to the chemical substructures of a molecule, from SMILES strings [ 13 , 14 ] . The previous investigation has highlighted the limitations of SMILES in terms of topology awareness, underscoring its inability to explicitly encode structural information of molecules [ 15 ] . To address the issue, structure-aware pre-training methods utilizing Graph Neural Networks (GNNs) have marked a significant advancement. These approaches leverage the graph-format representation of molecules (such as the topology of every atom in a molecule onto a 2D space), enriching the learning models with a deeper understanding of molecular structures [ 7 , 6 , 4 ] . More recently, 3D graph-based molecular data representation has been introduced in GNN pre-training, where the structures of massive molecules in a 3D space have been used to boost the performance of pre-trained models by incorporating the 3D structural/topological information [ 16 , 17 ] . However, such structural/topological information is not also precise. For example, in addition to acquiring the 3D positions of every atom and the angles between every two bonds in a molecule through experiments, some studies directly convert SMILES strings or 2D topology graph of a molecule into a 3D graph using Merck molecular force field (MMFF94) [ 18 ] function in RDKit [ 16 ] . It is reasonable to doubt the “add-value” of such automatic data format conversion and the precision of converted 3D graphs. Thus, it would be challenging to pre-train a structure-aware molecular model while the precise structural information is not available. To tackle the aforementioned challenges, we propose a novel molecular representation framework MLM-FG – a SMILES-based M olecular L anguage M odel, which randomly masking SMILES subsequences corresponding to specific molecular F unctional G roups to incorporate structure information of atoms during the pre-training phase. Specifically, MLM-FG employs transformer-based models trained on a large corpus of SMILES strings for 100 million molecules. As shown in Figure 1 , given the SMILES string for every molecule in the training dataset, MLM-FG first parses the string and identify the subsequences corresponding to functional groups and key clusters of atoms in the molecules. Then MLM-FG randomly masks a certain proportion of subsequences and trains the model to predict the masked part as the pre-training task. Extensive experimental evaluations across 11 benchmark classification and regression tasks in the chemical domain demonstrate the robustness and superiority of MLM-FG . Our findings reveal that MLM-FG outperforms existing pre-training models, either based on SMILES or graphs, in 9 out of the 11 downstream tasks, ranking as a close second in the remaining ones. Remarkably, MLM-FG also surpasses 3D graph-based models, which explicitly incorporate molecular structures into their inputs, highlighting its exceptional capacity for representation learning even without explicit 3D structural information. These results show that pre-trained transformer encoders specialized in molecular SMILES demonstrate robust performance, matching or even exceeding existing supervised or unsupervised language models and GNN benchmarks in accurately forecasting a broad spectrum of molecular properties. Figure 1 : An illustration of the proposed MLM-FG framework: (1) MLM-FG adopts 12-layer multi-head transformer blocks (in either RoBERTa or MoLFormer architectures) with a hidden state dimension of D_{h} =768 for pre-training and fine-tuning, (2) MLM-FG follows a functional group-aware random masking strategy to pre-train the model on a large corpus of 10 to 100 million SMILES sequences from PubChem, and (3) MLM-FG fine-tunes the pre-trained models to support a wide range of molecular machine learning applications. 2 Results In this section, we present a series of comprehensive experiments designed to illustrate the efficacy of MLM-FG . These experiments include performance comparisons across various downstream tasks and visual analysis of pre-trained representations. To assess the impact of model architecture and data

Read the complete text