In context learning Foundation models for Materials Property Prediction with Small datasets

Qinyang Li, Rongzhi Dong, Nicholas Miklaucic, Jeffrey Hu, Sadman Sadeed Omee, Lai Wei, Sourin Dey, Ming Hu, Jianjun Hu · 2026

Read In context learning Foundation models for Materials Property Prediction with Small datasets by Qinyang Li, Rongzhi Dong, Nicholas Miklaucic, Jeffrey Hu, Sadman Sadeed Omee, Lai Wei, Sourin Dey, Ming Hu, Jianjun Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00133.txt.

Introduction

Qinyang Li leads eight coauthors in building a foundation model that predicts material traits from small datasets. Their 2026 system learns in context. It takes examples inside the prompt instead of long retraining. The model joins composition data with structure aware inputs. A pretrained transformer is paired with graph network embeddings. New descriptors add bond ionicity and charge transfer data.

Tests use the MatBench suite and a separate heat conduction dataset. The system matches or beats leading models at far lower training cost. With no training at all, it won five of six composition tasks. One phonon task improved by about ten percent. On heat conduction, it captured phonon scattering and mass contrast effects. The authors also show its inner space smooths raw feature clusters. That smoothing helps prediction between known points.

Catalog record

CREATORQinyang Li, Rongzhi Dong, Nicholas Miklaucic, Jeffrey Hu, Sadman Sadeed Omee, Lai Wei, Sourin Dey, Ming Hu, Jianjun Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00133

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In context learning Foundation models for Materials Property Prediction with Small datasets Thanks: Citation : Li, et al.. ICL Foundation model for material property prediction. 16 Pages…. DOI:000000/11111. Qinyang Li, Rongzhi Dong, Nicholas Miklaucic, Jeffrey Hu, Sadman Sadeed Omee, Lai Wei, Sourin Dey, Ming Hu, Jianjun Hu Abstract Foundation models (FMs) have recently shown remarkable in-context learning (ICL) capabilities across diverse scientific domains. In this work, we introduce a unified in-context learning foundation model (ICL-FM) framework for materials property prediction that integrates both composition-based and structure-aware representations. The proposed approach couples the pretrained TabPFN transformer with graph neural network (GNN)-derived embeddings and our novel MagpieEX descriptors. MagpieEX augments traditional features with cation-anion interaction data to explicitly measure bond ionicity and charge-transfer asymmetry, capturing interatomic bonding characteristics that influence vibrational and thermal transport properties. Comprehensive experiments on the MatBench benchmark suite and a standalone lattice thermal conductivity (LTC) dataset demonstrate that ICL-FM achieves competitive or superior performance to state-of-the-art (SOTA) models with significantly reduced training costs. Remarkably, the training-free ICL-FM outperformed sophisticated SOTA GNN models in five out of six representative composition-based tasks, including a significant 9.93% improvement in phonon frequency prediction. On the LTC dataset, the FM effectively models complex phenomena such as phonon-phonon scattering and atomic mass contrast. t-SNE analysis reveals that the FM acts as a physics-aware feature refiner, transforming raw, disjoint feature clusters into continuous manifolds with gradual property transitions. This restructured latent space enhances interpolative prediction accuracy while aligning learned representations with underlying physical laws. This study establishes ICL-FM as a generalizable, data-efficient paradigm for materials informatics. Introduction The discovery and design of novel materials are fundamental for developing energy storage, semiconductor technology, catalysis, and numerous other industrial applications, critical to addressing global challenges and fostering transformative progress in modern society. A central bottleneck in the material discovery process is accurate and fast prediction of fundamental materials properties such as ion conductivity, thermal conductivity, formation energy, band gap, dielectric constants, elastic moduli, etc. Such material property prediction models have been widely used in screening new functional materials from known materials repositories [ 46 ] , inverse materials design [ 67 ] , and guiding experimental explorations [ 31 ] . While quantum-mechanical simulations such as density functional theory (DFT) may achieve accurate predictions for some materials properties, they are usually computationally expensive and cannot be scaled to the millions of candidate compounds required for high-throughput virtual screening. This has motivated the rapid development of machine learning (ML) methods for materials property prediction as a complementary approach to accelerate materials discovery. In the past decade, machine-learning-based materials property prediction models have achieved significant progress [ 17 ] and are increasingly integrated into materials discovery pipelines [ 59 , 55 ] . These include composition-based property prediction algorithms based on the widely used Magpie (2016) features [ 62 ] , recent attention-based neural network Roost (2020) [ 24 ] and CrabNet (2021) [ 61 ] , neural network based ModNet (2021) [ 13 ] , and large-language-model-based Darwin (2023) [ 65 ] , which learn to quickly predict material properties directly from the composition, making them suitable for high-throughput screening candidate materials without structural information. Compared to composition-based predictors, structure-based material property prediction models have demonstrated higher performance and stronger generalization across diverse property datasets, especially those graph neural networks (GNN) based models [ 53 ] from early CGCNN [ 64 ] to more recent ALIGNN [ 10 ] , DeeperGATGNN [ 49 ] , M3GNet [ 4 ] , coGN, coNGN [ 57 ] , and CHGNet [ 14 ] . These models incorporate atomic connectivity and geometric features to extract better structural and physicochemical features from the structures for accurate property prediction. A series of architectural innovations in neural networks have been introduced to model multi-body interactions and bond angles [ 4 , 10 , 30 , 22 ] . To deal with periodicity [ 66 ] and transformations such as translation, rotation, and reflection using equivariant neural networks [ 22 , 2 , 37 , 66 ] , Transformers [ 60 , 41 ] , graph transformers [ 22 , 66 , 58 ] , transfer learning [ 41 , 26 ] , active learning [ 32 ] , self-supervised learning and the pretraining+finetuning approaches [ 21 ] , and multi-modal neural networks [ 45 , 6 ] , have been proposed to improve material property prediction. Despite substantial research progress, the challenge of accurately predicting material properties from small datasets remains unsolved. This is evident from the stagnation of prediction performance on the Matbench [ 17 ] leaderboards since 2023, where most mean absolute error (MAE) improvements of state-of-the-art (SOTA) algorithms over ALIGNN (2021) [ 10 ] are below 10%. In a separate study, the PMCGNN (2025) [ 20 ] model, which combines global and local modules, reported an MAE of 0.038 GPa for bulk moduli prediction (a 5.00% reduction compared to PotNet [ 38 ] , 2023) and 0.063 GPa for shear moduli prediction (a 3.08% reduction compared to PotNet, 2023). In another study of 2025 [ 41 ] , their sophisticated CrysCo algorithm achieves similar performance (with <3.5% improvement) for six out of eight datasets compared to ALIGNN (2021). Their transfer learning model CrysCoT also only achieved marginal improvement (<3%) compared to CrysCo as it is easy to overfit the target datasets during the finetuning stage. Similar marginal improvements can also be found in the multi-modal model [ 45 ] . Most materials property prediction tasks with high-demand industrial applications are inherently defined over small datasets such as ionic conductivity [ 11 ] , thermal conductivity [ 47 ] , elastic moduli [ 17 ] , piezoelectric tensors [ 15 ] , and dielectric properties [ 17 ] , usually containing a few hundreds to a few hundreds of samples. The complexity of the structure-property relationship coupled with such small datasets and GNNs’ high expressive power naturally leads to overfitting: the model learns the training examples including its noises instead of learning general rules. This weakness is further compounded by inherent material dataset redundancy [ 36 , 29 , 34 ] and the challenge for out-of-distribution predictions needed for discovering novel outlier/exceptional materials [ 48 , 35 ] Recently, the broader field of machine learning has been transformed by the emergence of foundation models (FMs) [ 19 , 7 , 27 , 44 ] . Unlike task-specific architectures, FMs are trained on massive and heterogeneous datasets using scalable architectures and objectives, producing general-purpose representations that can be transferred across diverse downstream tasks. This paradigm has reshaped diverse fields such as natural language processing [ 43 ] , computer vision [ 1 ] , medical imaging [ 63 ] , and chemistry [ 33 ] . Foundation models can be used for addressing the persistent ML challenges such as data scarcity and poor generalization [ 8 ] . In the field of materials science [ 51 ] , large-language model (LLM) based foundation models have been applied to diverse materials discovery tasks such as data extraction, property prediction, and synthesis planning [ 51 ] . However, it remains a challenge how to link such LLM models to highly specialized GNN models for materials

Read the complete text