Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model

Justin Airas, Bin Zhang · 2026

Read Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model by Justin Airas, Bin Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05388.txt.

Introduction

Justin Airas and Bin Zhang distill a protein language model into a foundational implicit solvent model in a 2026 paper. Implicit solvent models promise the accuracy of explicit water simulations at a fraction of the cost. They treat the solvent as a continuum instead of thousands of molecules. After decades of work on them, their accuracy still falls short for critical applications such as protein folding and intrinsically disordered proteins, which lack a fixed shape.

The authors' strategy takes what a protein language model has learned about evolution and distills it into the solvent model, replacing hand-written formulas with learned knowledge. A transferable, data-driven solvent model would speed molecular simulations across drug discovery. In that field, solvent treatment is often the difference between a useful prediction and an expensive false lead.

Catalog record

CREATORJustin Airas, Bin Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05388

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model Justin Airas, Bin Zhang Abstract Implicit solvent models (ISMs) promise to deliver the accuracy of explicit solvent simulations at a fraction of the computational cost. However, despite decades of development, their accuracy has remained insufficient for many critical applications, particularly for simulating protein folding and the behavior of intrinsically disordered proteins. Developing a transferable, data-driven ISM that overcomes the limitations of traditional analytical formulas remains a central challenge in computational chemistry. Here we address this challenge by introducing a novel strategy that distills the evolutionary information learned by a protein language model, ESM3, into a computationally efficient graph neural network (GNN). We show that this GNN potential, trained on effective energies from ESM3, is robust enough to drive stable, long-timescale molecular dynamics simulations. When combined with a standard electrostatics term, our hybrid model accurately reproduces protein folding free-energy landscapes and predicts the structural ensembles of intrinsically disordered proteins. This approach yields a single, unified model that is transferable across both folded and disordered protein states, resolving a long-standing limitation of conventional ISMs. By successfully distilling evolutionary knowledge into a physical potential, our work delivers a foundational implicit solvent model poised to accelerate the development of predictive, large-scale simulation tools. † † email: [email protected] † † affiliation: Department of Chemistry, Massachusetts Institute of Technology, Cambridge, MA, USA 1 Synopsis We demonstrate that a graph neural network can act as a foundational protein implicit solvent model when trained to match the distribution of secondary structure predictions from a protein language model. 2 TOC Graphic 3 Introduction Implicit solvent models (ISMs) represent a promising middle ground in molecular simulation, offering a significant computational advantage over all-atom explicit solvent simulations while providing greater physical detail than lower-resolution coarse-grained models 77 ; 78 ; 85 ; 86 ; 27 ; 65 ; 35 ; 14 . The middle-ground promise of ISMs has generated significant enthusiasm. Nevertheless, despite decades of refinement 56 ; 97 ; 92 ; 96 ; 49 ; 26 ; 81 ; 80 ; 63 ; 55 ; 120 ; 25 ; 79 ; 98 ; 99 ; 3 , their accuracy remains well below that of their explicit solvent counterparts. Traditional ISMs, such as popular Generalized Born (GB) and surface-area-based formulations, systematically misrepresent the balance between solvation and intramolecular interactions, frequently leading to artifacts like the over-compaction of disordered proteins, overstabilization of \alpha -helical conformations, and exaggerated protein–protein association energies 7 ; 54 ; 111 ; 76 ; 112 ; 17 ; 19 . These deficiencies stem from two overarching limitations. First, ISMs rely on approximate analytical expressions to compute the solvation free energy, E_{\mathrm{solv}} ; such approximations cannot fully capture the complex dependence of E_{\mathrm{solv}} on molecular composition, geometry, and conformational state. Second, the parameters of ISMs are rarely optimized in a systematic or data-driven manner to reproduce results from experiment or explicit solvent simulations across diverse protein families. The rise of machine learning (ML) offers a path to overcome these challenges. By replacing the approximate analytical formulas with flexible neural network potentials, ML-based force fields could lead to ISMs with much-improved accuracy. Many promising studies have demonstrated the potential of using ML methodologies for force field development. 1 ; 69 ; 70 ; 67 ; 68 ; 28 ; 62 ; 74 ; 128 ; 21 ; 131 ; 123 ; 122 ; 46 ; 40 ; 8 ; 73 ; 87 ; 41 ; 39 ; 37 ; 38 ; 30 ; 31 ; 42 ; 48 ; 130 ; 33 ; 12 ; 43 ; 134 ; 106 ; 18 ; 6 ; 29 ; 44 ; 114 ; 126 ; 125 ; 50 ; 105 ; 9 ; 129 ; 24 ; 89 ; 83 ; 104 ; 113 ; 34 ; 51 ; 118 ; 132 ; 72 ; 103 ; 93 Graph neural networks 117 ; 53 ; 52 ; 5 ; 20 ; 82 ; 110 ; 116 ; 47 ; 61 ; 109 ; 107 ; 124 ; 115 ; 101 ; 127 ; 13 ; 10 ; 94 ; 11 ; 90 ; 2 ; 22 (GNNs) are particularly well-suited as neural network potentials. Since molecules are naturally structured as graphs, with atoms as nodes and interactions as edges, GNNs provide a native framework for learning complex, geometry-dependent energetic terms. Moreover, their architecture facilitates the straightforward incorporation of fundamental physical principles, such as translational, rotational, and permutational invariance, which is crucial for building robust and generalizable force fields. However, parameterizing a GNN as an ISM is not trivial. Unlike typical ML force fields trained on quantum mechanical energies, the target E_{\mathrm{solv}} values are not directly known, precluding simple supervised learning. Several theoretically sound approaches have been introduced to optimize ISMs by matching configurational ensembles from explicit solvent simulations. 28 ; 70 ; 1 While powerful, these methods are constrained by the finite set of proteins for which such simulation data is available, raising concerns about the transferability of the resulting models to new and diverse protein systems. In this study, we introduce a novel training strategy to address this transferability challenge by learning a proxy for E_{\mathrm{solv}} from a protein language model. We leverage the multimodal model ESM3 57 , which captures the joint distribution of sequence, structure, and function across billions of proteins and attains near-experimental accuracy in predicting 3D structures directly from sequence. Its conditional probabilities, P(\text{structure}|\text{sequence}) , and corresponding effective energies, E=-k_{B}T\log P , therefore approximate the true folding free-energy landscape. As solvation dominates folding energetics, these evolution-derived probabilities offer an excellent proxy for solvent-mediated effects. Putting this hypothesis to the test, we first demonstrate that a GNN with significantly fewer parameters can successfully reproduce the effective energies predicted by ESM3, a remarkable distillation of knowledge. The resulting GNN potential is robust enough to drive long-timescale molecular dynamics (MD) simulations. We show that for several proteins, simulations of up to 500 ns maintain stable structures near their native conformations, providing strong evidence that the evolutionary statistics within ESM3 indeed serve as a high-fidelity proxy for the protein folding free-energy landscape. To create a physically predictive model, we further combine the distilled GNN potential with a standard GB electrostatic term. This hybrid representation not only reproduces explicit-solvent folding free-energy landscapes with high fidelity but also captures the secondary-structure distributions of intrinsically disordered proteins, an area where traditional ISMs routinely fail. Together, these results demonstrate that our approach yields a genuinely transferable ISM capable of describing both ordered and disordered states within a single framework. By distilling the evolutionary knowledge encoded in a protein language model into a compact neural network potential, we establish the first foundational ISM: a scalable, data-driven model that provides a robust starting point for building the next generation of accurate and efficient protein simulations. 4 Results 4.1 Distilling solvent-sensitive secondary structure preferences into a multiscale GNN To develop an accurate and transferable machine-learning ISM, we require a flexible architecture capable of representing solvent-mediated interactions across diverse proteins. We therefore adopt Schake, a recently introduced multiscale GNN designed for protein-scale systems. 2 Schake combines (i) a short-range SAKE message-passing layer 124 that encodes detailed

Read the complete text