Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT

Wenqi ZENG, Lu ZHANG, Yuan YAO · 2025

Read Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT by Wenqi ZENG, Lu ZHANG, Yuan YAO (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02734.txt.

Introduction

Wenqi Zeng, Lu Zhang, and Yuan Yao present a GPT-style model that learns long-run motion of molecules from short runs. Their 2025 paper notes that full simulations of slow molecular change cost too much time. Short runs from varied starts can sample the space, but past learning tools handle only low-count summaries of state.

Their model, nano-GPT, reads fine state tokens and learns by swapping in its own forecasts step by step during training, which curbs error pile-up. Tests span a toy four-state system, a small peptide, and a folding protein domain. In each, the model caught long-time patterns through its attention reads. The authors pitch it as a new lens on slow molecular change.

Catalog record

CREATORWenqi ZENG, Lu ZHANG, Yuan YAO
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT Wenqi ZENG, Lu ZHANG, Yuan YAO Abstract Long-term biomolecular dynamics are critical for understanding key evolutionary transformations in molecular systems. However, capturing these processes requires extended simulation timescales that often exceed the practical limits of conventional models. To address this, shorter simulations, initialized with diverse perturbations, are commonly used to sample phase space and explore a wide range of behaviors. Recent advances have leveraged language models to infer long-term behavior from short trajectories, but methods such as long short-term memory (LSTM) networks are constrained to low-dimensional reaction coordinates, limiting their applicability to complex systems. In this work, we present nano-GPT, a novel deep learning model inspired by the GPT architecture, specifically designed to capture long-term dynamics in molecular systems with fine-grained conformational states and complex transitions. The model employs a two-pass training mechanism that incrementally replaces molecular dynamics (MD) tokens with model-generated predictions, effectively mitigating accumulation errors inherent in the training window. We validate nano-GPT on three distinct systems: a four-state model potential, the alanine dipeptide, a well-studied simple molecule, and the Fip35 WW domain, a complex biomolecular system. Our results show that nano-GPT effectively captures long-timescale dynamics by learning high-order dependencies through attention mechanism, offering a novel perspective for interpreting biomolecular processes. † † affiliation: Department of Mathematics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong SAR 999077, China † † affiliation: State Key Laboratory of Structural Chemistry, Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences, Fuzhou, Fujian 350002, China † † affiliation: Department of Mathematics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong SAR 999077, China † † affiliation: Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong SAR 999077, China † † email: [email protected] † † suppinfo: S 1 Introduction Biomolecular dynamics play a crucial role in understanding the function of biological macromolecules. These dynamics encompass a wide range of conformational changes, from short-term and local vibrations to long-term and significant conformational shifts, occurring over various time scales. They lay the structural basis for facilitating processes such as enzyme catalysis, molecular recognition, and signal transduction, thus critical to the proper functioning of proteins, nucleic acids, and other biomolecules ( Leimkuhler et al. 1996 ) . Therefore, accurately predicting the multi-scale conformational changes, especially the long-timescale dynamics, can advance our understanding about the structure-activity relationship of biomolecules, and guide the enzyme engineering and drug design. Molecular dynamics (MD) simulations are a widely used tool to investigate the conformations of biomolecules, and provides the dynamics for localized and high-frequency events occurring at nanoseconds to a few microseconds. However, the critical conformational changes for functioning are usually rare events, involving the transitions between different conformational states occurring at timescales from microseconds to milliseconds or even longer. Such a long-timescale dynamics is challenging to reach directly by conventional all-atomic MD simulations ( Dullweber et al. 1997 ) . The timescale gap between the short-term MD simulations and long-term dynamics hinders a full understanding of the conformational dynamics of the macromolecules as well as their functions. In this regard, extracting the long-term pattern from short simulations is critical to address the challenges in predicting biomolecular dynamics ( Chodera et al. 2007 ; Pan and Roux 2008 ; Noé and Nuske 2013 ) . One commonly used approach for extracting long-term dynamics from short simulations is Markov State Models (MSMs) ( Husic and Pande 2018 ) , which are effective for systems with well-defined, slow dynamics and clear timescale separation. However, the construction of accurate MSMs requires careful selection of the lag time and states. An inappropriate choice of lag time can miss fast dynamics or overly smooth slow processes, while poor state discretization can obscure meaningful transitions. Additionally, MSMs are less suited for modeling fast, hidden, or non-Markovian processes, which are increasingly common in complex molecular systems. In contrast, Long Short-Term Memory (LSTM) networks ( Gers et al. 2000 ) excel at modeling sequential data without requiring explicit selection of predefined states or lag times, unlike methods such as MSMs and HMMs ( Husic and Pande 2018 ; Bowman et al. 2009 ) . LSTMs can learn complex, high-order Markovian dynamics directly from the data and generate new frames, making them particularly well-suited for modeling conformational transitions that span multiple timescales ( Tsai et al. 2020 ; Tsai et al. 2022 ) . Tsai et al. 2020 shows that LSTM accurately capture Boltzmann statistics and kinetics across systems like alanine dipeptide and riboswitches. As a result, LSTMs offer a more efficient and flexible alternative to MSMs for biomolecular simulations. Building on previous work, Tsai et al. 2022 tackled the challenge of learning longer dynamics using even shorter frames to train the LSTM. By incorporating static and dynamic constraints within an iterative framework guided by the Maximum Caliber principle, they successfully predicted 2 ns of transition dynamics from simulations as short as 0.2 ps. However, these methods primarily address simplified systems with coarse-grained, low-dimensional projections (e.g., \phi and \psi in alanine dipeptide). In complex systems, LSTMs struggle to capture long-range dependencies, such as oversmoothing transitions in all-atom alanine dipeptide or missing the slowest dynamics in Fip35. The difficulties LSTMs encounter in capturing long-range dependencies can be traced to inherent limitations in their sequential processing structure ( Zhao et al. 2020 ; Dieng et al. 2017 ) . Although LSTMs are designed to retain and update information across time steps through gating mechanisms, the standard initialization of these gates can hinder the learning of long-term temporal correlations ( Tallec and Ollivier 2018 ) . Over time, the information stored in the LSTM’s memory cell tends to decay exponentially, restricting the network’s ability to maintain relevant information from distant time steps ( Gu et al. 2020 ) . Consequently, LSTMs are often less effective at capturing slow or rare dynamics that require the retention of information over extended periods. Generative Pre-trained Transformer (GPT)-like models ( Brown et al. 2020 ) address the long-range dependency limitations of LSTMs by utilizing parallel processing through matrix operations. The self-attention mechanism within GPT enables the model to capture more intricate relationships between distant frames in the sequence, without relying on strict sequential processing ( Tay et al. 2021 ) . Recent work by ( Bera and Mondal 2025 ) applies a decoder-only Transformer to predict future states in molecular systems. While effective on systems with limited state space and continuous trajectory inputs, their approach does not address the challenges of modeling long-range dependencies in de novo sequence generation or in systems with finer-grained state discretization. In this paper, we introduce nano-GPT, a novel method specifically designed to capture long-timescale molecular dynamics from short frames of unbiased MD simulations. By learning non-Markovian dependencies, nano-GPT generates states to extend the original trajectory, demonstrating the time evolution of states. Theoretically, we establish a connection between state embeddings in nano-GPT and kinetic time in dynamics. Experimentally, we validate nano-GPT on three systems of varying complexity: a model potential, alanine dipeptide, and the Fip35 WW domain. Nano-GPT captures both statistical and dynamic features across complex

Read the complete text