Supervised Pretraining for Material Property Prediction

Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali · 2025

Read Supervised Pretraining for Material Property Prediction by Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20112.txt.

Introduction

Chowdhury Mohammad Abid Rahman and co-authors propose a training trick for AI models that predict material traits, in a 2025 paper. Good predictions can speed the hunt for new materials with wanted traits. Deep learning models do well at the task, but they need large labeled data sets, which are slow and costly to build. Self-taught pretraining on unlabeled data helps, yet labels often exist for other tasks.

Their trick uses those spare class labels as stand-ins during pretraining. The model learns from the stand-in task first, then adapts to the real trait. A graph-based method that adds noise also makes the learned features tougher without warping the material graphs. Across six hard prediction tasks, the fine-tuned models beat baselines by clear margins. The authors call the work a first test of this label-guided route for materials.

Catalog record

CREATORChowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20112

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Supervised Pretraining for Material Property Prediction Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali Abstract Accurate prediction of material properties facilitates the discovery of novel materials with tailored functionalities. Deep learning models have recently shown superior accuracy and flexibility in capturing structure-property relationships. However, these models often rely on supervised learning, which requires large, well-annotated datasets—an expensive and time-consuming process. Self-supervised learning (SSL) offers a promising alternative by pretraining on large, unlabeled datasets to develop foundation models that can be fine-tuned for material property prediction. In this work, we propose supervised pretraining, where available class information serves as surrogate labels to guide learning, even when downstream tasks involve unrelated material properties. We evaluate this strategy on two state-of-the-art SSL models and introduce a novel framework for supervised pretraining. To further enhance representation learning, we propose a graph-based augmentation technique that injects noise to improve robustness without structurally deforming material graphs. The resulting foundation models are fine-tuned for six challenging material property predictions, achieving significant performance gains over baselines, ranging from 2% to 6.67% improvement in mean absolute error (MAE)—and establishing a new benchmark in material property prediction. This study represents the first exploration of supervised pertaining with surrogate labels in material property prediction, advancing methodology and application in the field. Introduction Advancing material discovery is fundamental to driving scientific innovation and expanding application horizons across numerous fields where the need for novel materials becomes incumbent. Such efforts require a deep understanding of material properties to unlock new functionalities and enhance performance in targeted applications [ 1 , 2 , 3 , 4 ] . Traditionally, the scientific community has relied on exhaustive searches through known databases to identify suitable materials, which is costly, time-consuming, prompt to errors, and resource-intensive. Introducing first-principles methods, particularly Density Functional Theory (DFT), established a rigorous mathematical framework for predicting material properties [ 5 , 6 ] . However, this approach remains computationally intensive and time-consuming [ 7 ] , and its accuracy is often limited by its approximations, which may only be suitable for some systems. In recent years, machine learning (ML) and deep learning (DL) models have gained traction in material property prediction, offering faster and more efficient ways to estimate material properties [ 8 , 9 , 10 , 11 , 12 , 13 ] , at least within the scope of DFT. These models have demonstrated their potential to revolutionize the field by reducing the need for time-consuming calculations. However, the challenge is that the ML/DL methods heavily depend on large, well-annotated datasets with labels for each material property category. Generating such ground-truth labels often relies on DFT-calculated data, thereby inheriting the same time and cost bottlenecks as traditional methods. Self-supervised learning (SSL) [ 14 , 15 ] , a new learning paradigm in deep learning, provides a promising alternative utilizing unlabeled data and learning useful representations without the need for explicit labels. These models can be pre-trained on vast amounts of unlabeled data to form foundation models, which can then be fine-tuned with smaller labeled datasets. These models have been popularly adapted in computer vision and natural language processing, and several algorithms have been proposed, varying according to learning objectives, including BERT [ 16 ] , SimCLR [ 14 ] , and BYOL [ 17 ] . This training paradigm has shown its utility in creating models like ChatGPT and DALL·E , which have gained widespread societal use. The success of these models reflects SSL’s potential in specialized academic research settings and real-world applications that impact society at large. This approach could have particular relevance within material science as vast databases of material crystals already contain many chemical structures without associated property labels [ 18 , 19 , 20 ] . Using these untapped data, SSL could unlock significant potential for the prediction of material properties. Although self-supervised learning (SSL) has been predominantly explored in vision and language tasks, there have been a few attempts to extend its training paradigm to other scientific disciplines, including molecular chemistry [ 21 , 22 , 23 , 24 , 25 ] . SSL has shown potential in learning better representations for complex molecular structures by leveraging unlabeled data in these fields. However, while molecules and materials share similarities, such as atomic structures and bonding patterns, materials often present more complex, periodic structures. Unlike molecules, which can be represented by molecular graphs, SMILES strings, or 3D coordinates with well-defined atomic bonding, materials can exhibit crystalline, amorphous, or composite phases that require more intricate representation methods. For instance, the periodicity of material structures, such as crystal lattices stored in CIF files or their spatial distribution in 3D voxel grids, demands specialized techniques to capture long-range order and periodic boundary conditions. However, molecular structures typically involve covalent bonds between a limited number of atoms and do not exhibit the same degree of complexity in periodicity. As a result, while SSL approaches, such as graph neural networks (GNNs), have proven effective for molecular graphs, adapting these methods to material science requires additional considerations. Techniques incorporating periodic boundary conditions or voxel-based representations are essential to accurately represent material crystals. Thus, while there is growing interest in applying SSL to material property prediction, the distinct challenges that material structures pose make it more complex than molecules. In this study, we advance self-supervised learning (SSL) for material property prediction. Although there have been some early attempts to apply existing SSL setups for this purpose, we take concrete steps forward by proposing SPMat (Supervised Pretraining for Material Property Prediction) , a novel SSL framework that integrates supervisory signals, referred to as surrogate labels, to guide the learning process. Unlike specific labels for each property category, our framework leverages general material attributes (e.g., metal vs. nonmetal) to guide the SSL learning process. To incorporate these supervisory signals, we introduce a novel technique within the standard SSL framework, applicable to both contrastive [ 14 ] and noncontrastive [ 15 ] approaches. For feature representation learning, we account for the unique characteristics of materials by employing graph neural network (GNN)-based models, specifically the Crystal Graph Convolutional Neural Network (CGCNN) [ 26 ] . CGCNN effectively encodes local and global chemical information, capturing essential material features such as atomic electron affinity, group number, neighbor distances, orbital interactions, bond angles, and aggregated local chemical and physical properties. Additionally, since SSL frameworks heavily rely on augmentation strategies to enhance learning, we explore existing augmentations and propose Graph-level Neighbor Distance Noising (GNDN), a novel augmentation approach that introduces random noise in neighbor distances, further improving model performance. In general, the proposed framework enables us to develop a foundation model that demonstrates improved generalization across several material property prediction tasks, outperforming both standard deep learning and existing SSL frameworks. Figure 1: Schematic diagram of the proposed SPMat (Supervised Pretraining for Material Property Prediction) framework. Initially, the material structure undergoes augmentation, including a novel Graph-level Neighbor Distance Noising (GNDN) augmentation, to create diverse views. Deep learning-based encoders and projectors are then used to capture the representations as embeddings, which are utilized in various pretext tasks using labels (top panel). In the downstream task the fine-tuned model starts training on top of the pre-trained encoder and predicts material properties which

Read the complete text