Refining embeddings with fill-tuning: data-efficient generalised performance improvements for materials foundation models
Matthew P. Wilson, Edward O. Pyzer-Knapp, Nicolas Galichet, Luke Dicks · 2025
Read Refining embeddings with fill-tuning: data-efficient generalised performance improvements for materials foundation models by Matthew P. Wilson, Edward O. Pyzer-Knapp, Nicolas Galichet, Luke Dicks (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13886.txt.
Introduction
Matthew Wilson and co-authors present fill-tuning, a way to improve materials foundation models with very little new data, in a 2025 paper. Such models learn embeddings used for many tasks, but fixing one task by fine tuning spoils the rest. The team instead hunts for weak regions of the embedding space itself.
A roughness check on the space's shape points to where new data would help most. Adding just one hundred well chosen points lifted all downstream scores by near one percent across top models trained on a billion points. The cost matches plain fine tuning, but the gain is general rather than task bound.
Catalog record
| CREATOR | Matthew P. Wilson, Edward O. Pyzer-Knapp, Nicolas Galichet, Luke Dicks |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13886
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Refining embeddings with fill-tuning: data-efficient generalised performance improvements for materials foundation models Matthew P. Wilson, Edward O. Pyzer-Knapp, Nicolas Galichet, Luke Dicks Abstract Pretrained foundation models learn embeddings that can be used for a wide range of downstream tasks. These embeddings optimise general performance, and if insufficiently accurate at a specific task the model can be fine-tuned to improve performance. For all current methodologies this operation necessarily degrades performance on all out-of-distribution tasks. In this work we present ‘fill-tuning’, a novel methodology to generate datasets for continued pretraining of foundation models that are not suited to a particular downstream task, but instead aim to correct poor regions of the embedding. We present the application of roughness analysis to latent space topologies and illustrate how it can be used to propose data that will be most valuable to improving the embedding. We apply fill-tuning to a set of state-of-the-art materials foundation models trained on 𝒪 ( 10 9 ) \mathcal{O}(10^{9}) data points and show model improvement of almost 1% in all downstream tasks with the addition of only 100 data points. This method provides a route to the general improvement of foundation models at the computational cost of fine-tuning. 1 Introduction Foundational models are central to machine learning research ( Bommasani et al. 2021 ) . These models are pretrained on vast amounts of data, via self-supervised learning, enabling them to perform a variety of tasks. Many such models learn an embedding of the training data, which can be used as input for downstream tasks. The structure of this embedding, or latent space, is central to model performance, where it should encode similar points in close proximity. Initially, foundational models were the purview of language ( Devlin et al. 2018 ; Brown et al. 2020 ) , but such approaches are being applied to chemistry and materials research ( Ahmad et al. 2022 ; Takeda et al. 2023 ) . These models have been developed for a range of modalities: text ( Schwaller et al. 2019 ; Ross et al. 2022 ) , three-dimensional positions ( Batatia et al. 2023 ) and molecular graphs ( Kishimoto et al. 2023 ) . Current research aims to combine these different representations into multimodal foundation models ( Takeda et al. 2023 ; Chang & Ye 2024 ) . Training foundation models requires both vast computational resources and datasets. For language models data is plentiful and easily accessible, but for materials applications there is significantly less data available and its generation can be both expensive and slow. Therefore, pretrained models are appealing, allowing generally good performance out of the box, and if the performance is insufficient on a given task it can be increased by fine-tuning. Fine-tuning of pretrained models allows the development of more accurate models with a small amount of data and a greatly reduced training time ( Goodfellow et al. 2013 ) . During fine-tuning the user performs additional training with a dataset containing many targeted examples related to a given task, aiming to increase performance in this domain. The result of improved performance at the given task is degradation in all out-of-distribution tasks ( Luo et al. 2023 ) , leading to worse general performance. Therefore, many schemes have been developed that attempt to limit the degradation of fine-tuning on out-of-distribution tasks ( Hu et al. 2021 ; Mukhoti et al. 2023 ) . Related to fine-tuning is the field of continual learning ( Wang et al. 2024 ) , where a model continues to be updated as more data becomes available. This scenario is similar to fine-tuning, but the additional data is governed by availability and is not specific to any given task. Another related technique is continued pretraining ( Parmar et al. 2024 ) , where additional self-supervised training is performed on a pretrained model. There is growing interest in using physical models to interpret and inform machine learning research ( Yang et al. 2021 ; Niroomand et al. 2024a ) . Such analyses probe properties of the loss landscapes to explain performance and develop more accurate models ( Draxler et al. 2018 ; Zhang et al. 2018 ; Niroomand et al. 2024b ) . However, there is little research on applying these techniques to foundation models via their latent spaces. In this work, we present the application of physics-based roughness analysis ( Dicks et al. 2024 ) to foundation model embeddings. We demonstrate how roughness analysis can be adapted to query the quality of pretrained model embeddings, and propose a methodology to generate datasets for continued pretraining that preferentially sample from the poorest regions. We term this approach ‘fill-tuning’, as this targeted data generation is based on model properties, not a specific task, allowing general performance improvements by filling gaps in model understanding. We illustrate this approach on a family of state-of-the-art materials foundation models ( Priyadarsini et al. 2024 ) trained on up to 8B data points, where the injection of only 100 data points into regions where the embedding is poorest allows continued training to give almost 1% improvement across all downstream tasks. 2 Methods 2.1 Foundation Models Roughness analysis has been widely used to relate the topology of datasets to the performance of machine learning models ( Aldeghi et al. 2022 ; Graff et al. 2023 ) . Here, we extend the frustration metric ( Dicks et al. 2024 ) , previously showing state-of-the-art performance in predicting model performance from dataset structure, to foundation model latent spaces. We expect that, after successful training, a foundation model embedding will have placed similar molecules in close proximity, and such a property is essential for methods that leverage latent space interpolation ( Gómez-Bombarelli et al. 2018 ) . This condition is similar to that studied in quantitative structure-property relationships where regions of feature space that exhibit large changes in molecular similarity over small distances, termed activity cliffs ( Stumpfe et al. 2019 ) , degrade model performance. Therefore, we assume that the smoothness of properties (herein, the molecular similarity within the embedding) can be a proxy for the accuracy of downstream tasks trained from the embedding. We apply roughness analysis to the latent space of the SELFIES-TED 1 1 1 The model SELF-BART is now referred to as an earlier iteration of the SELFIES-TED model. materials foundation models ( Priyadarsini et al. 2024 ) , a family of models based on the BART architecture ( Lewis et al. 2019 ) and trained on molecular SELFIES ( Krenn et al. 2020 ) strings. There are three variants of this model, which we distinguish by the number of parameters and training data size. In all cases data is drawn from both ZINC22 ( Tingle et al. 2023 ) and PubChem ( Kim et al. 2016 ) . The models, and their names, are summarised in Table 1 . Table 1: Summary of the various SELFIES-TED materials foundation models studied in this work. Model Parameters Dataset size SELFIES-TED (small) 2.2M 8B SELFIES-TED (medium) 358M 500M SELFIES-TED (large) 358M 1B We perform our initial analysis on the SELFIES-TED (small) model, and subsequent analysis on the larger models. The SELFIES-TED (small) model consists of a transformer encoder-decoder pair. The encoder produces a