Protein Folding with Neural Ordinary Differential Equations

Arielle Sanford, Shuo Sun, Christian B. Mendl · 2025

Read Protein Folding with Neural Ordinary Differential Equations by Arielle Sanford, Shuo Sun, Christian B. Mendl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16253.txt.

Introduction

Arielle Sanford, Shuo Sun, and Christian B Mendl rebuild a protein folding model as a smooth flow instead of a tall stack. Tools like AlphaFold predict a protein's shape from its sequence using a block called the Evoformer, stacked forty-eight deep. That depth costs heavy compute and locks the design into fixed steps.

Their version swaps the stack for a neural ordinary differential equation, a model whose depth flows like time. Memory cost stays flat no matter how deep the flow runs, and solvers can trade speed against accuracy. The model trains in about seventeen hours on a single graphics card, far less than the original demands. Its shapes look plausible and catch features like alpha-helices, though they fall short of AlphaFold's accuracy. The trade shows lean, flexible folding models are within reach.

Catalog record

CREATORArielle Sanford, Shuo Sun, Christian B. Mendl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Protein Folding with Neural Ordinary Differential Equations Arielle Sanford, Shuo Sun, Christian B. Mendl Abstract Recent advances in protein structure prediction, such as AlphaFold, have demonstrated the power of deep neural architectures like the Evoformer for capturing complex spatial and evolutionary constraints on protein conformation. However, the depth of the Evoformer, comprising 48 stacked blocks, introduces high computational costs and rigid layerwise discretization. Inspired by NeuralODE , we propose a continuous-depth formulation of the Evoformer, replacing its 48 discrete blocks with a NeuralODE parameterization that preserves its core attention-based operations. This continuous-time Evoformer achieves constant memory cost (in depth) via the adjoint method, while allowing a principled trade-off between runtime and accuracy through adaptive ODE ( ODE ) solvers. Benchmarking on protein structure prediction tasks, we find that the Neural ODE -based Evoformer produces structurally plausible predictions and reliably captures certain secondary structure elements, such as α \alpha -helices, though it does not fully replicate the accuracy of the original architecture. However, our model achieves this performance using dramatically fewer resources, just 17.5 hours of training on a single GPU, highlighting the promise of continuous-depth models as a lightweight and interpretable alternative for biomolecular modeling. This work opens new directions for efficient and adaptive protein structure prediction frameworks. MSA multiple sequence alignment ODE ordinary differential equation Neural ODE Neural Ordinary Differential Equation MLP multilayer perceptron QKV query, key, and value RK4 fourth-order Runge-Kutta 1 Introduction AlphaFold 2 and the improved AlphaFold 3 network have fundamentally transformed protein structure prediction by providing high-confidence atomic-level models for a vast array of proteins, including many without experimentally determined structures [ 1 , 2 , 3 ] . This is achieved by combining physical, geometric, and evolutionary information in a deep learning framework that directly predicts 3D structures from amino acid sequences. A central component of AlphaFold 2 is the Evoformer, a deep network trunk composed of 48 non-weight-sharing blocks that progressively refine two representations: the MSA ( MSA ) and the pairwise residue map. The representations encode domain-specific knowledge, such as geometric reasoning and co-evolutionary coupling, which are essential to AlphaFold’s success. However, the model’s depth and iterative recycling strategy result in substantial computational cost for both training and inference. \Acp NeuralODE provide an alternative approach for modeling deep transformations [ 4 ] . Rather than applying a fixed, discrete sequence of layers, a NeuralODE parameterizes the derivative of the hidden state using a neural network and integrates these dynamics over a continuous depth variable using a conventional ODE solver. This formulation provides several advantages. First, by employing the adjoint sensitivity method for backpropagation, NeuralODE achieve constant memory cost with respect to depth, as intermediate activations do not need to be stored during the forward pass. Second, their evaluation strategy can be adaptive. Many ODE solvers adjust step sizes dynamically based on the complexity of the trajectory, allowing the computational effort to scale with the difficulty of the input. Third, the trade-off between speed and accuracy becomes tunable, as solver tolerances can be set to balance runtime with numerical precision, making the model adaptable to various deployment scenarios. Our work is motivated by the observation that the Evoformer’s 48 sequential blocks apply small, incremental refinements to the MSA and pair representations, which can be viewed collectively as a smooth transformation from their initial to their final state. We propose to replace this discrete stack with a single NeuralODE module that models the refinement process as a continuous-time dynamical system. In this formulation, a shared vector field governs the evolution of representations over a continuous depth variable, and an ODE solver integrates this field to produce the final state. This substitution reduces the total number of learnable parameters by reusing a single set of weights throughout the integration, and it enables training with constant memory cost via the adjoint sensitivity method. Additionally, the use of ODE solvers introduces the potential to control the trade-off between runtime and accuracy through solver tolerances, offering flexibility not available in fixed-depth architectures. For this initial investigation, we omit the recycling steps to isolate the impact of the continuous-depth reformulation. The central assumption of this approach is that a continuous-time model can approximate the cumulative effect of the 48 Evoformer blocks, which is reasonable given that each block contributes a small, progressive update to the overall representation. In this work, we use OpenFold [ 5 ] to generate reference data, and treat its Evoformer transformation as ground-truth. OpenFold is an open-source implementation of AlphaFold 2 that reproduces the original architecture and performance while offering greater flexibility for model customization. OpenFold provides access to the full model architecture, eliminating dependence on proprietary code or Google infrastructure. All training data used in this study, namely, the MSA and pairwise residue representations in various Evoformer blocks, were derived from OpenFold inference runs on monomer inputs, using a single pass without recycling. These representations are treated as ground truth trajectories that define the transformation our ODE must learn to emulate. Our implementation is available online at [ 6 ] . 2 Background 2.1 AlphaFold 2 Architecture The Evoformer, the key building block of AlphaFold 2, is a deep network trunk composed of 48 non-weight-sharing blocks that progressively refine the MSA and the pairwise residue map representations. Each block applies Transformer-style updates, rooted in the attention mechanisms introduced by Vaswani et al. [ 7 ] , including gated axial attention over the MSA and geometry-aware updates to the pairwise representation via triangle-based modules that capture interactions among residue triplets. 2.2 \Acp NeuralODE A widely used ingredient of artificial neural networks is a residual block transformation s_{t+1}=s_{t}+f(s_{t},\theta_{t}), (1) with s_{t}\in\mathbb{R}^{D} the (hidden) state at layer (or time step) t , and f a trainable function parametrized by \theta_{t} . \Acp NeuralODE [ 4 ] are motivated by the insight that Eq. ( 1 ) can be interpreted as an Euler step discretization of an ODE . Taking the limit of infinitely small time steps, one arrives at the continuous-time analogue \frac{\mathrm{d}s(t)}{\mathrm{d}t}=f(s(t),t,\theta). (2) This formulation opens the door to a plethora of well-studied numerical ODE solvers and time step adaptation. Backpropagation can be implemented memory-efficiently via the “adjoint sensitivity method”, i.e., by solving an auxiliary ODE that runs backward in time. 3 Evoformer ODE Architecture To create a continuous-depth analog of AlphaFold’s Evoformer, we define an ODE function that approximates the cumulative transformation of the MSA and pair representations over depth. This function is inspired by the core operations of a single Evoformer block and retains the bidirectional exchange of information between the MSA and pairwise residue representations The function accepts the current MSA and pair states (m,z) and returns their time derivatives (\frac{dm}{dt},\frac{dz}{dt}) . Internally, it computes one full pass of Evoformer-style updates to obtain updated states (m^{\prime},z^{\prime}) , then compares these to the inputs to estimate a continuous vector field. The difference between the updated and original states is scaled by time-dependent gating functions \sigma_{m}(t) and \sigma_{z}(t) , learned via a shallow MLP ( MLP ) over the scalar depth variable t , resulting in a smooth evolution of the representations along the depth trajectory: \displaystyle f(m,z) \displaystyle=\big(\sigma_{m}(t)\cdot(m^{\prime}-m),\sigma_{z}(t)\cdot(z^{\prime}-z)\big), (3) \displaystyle\frac{\mathrm{d}(m,z)}{\mathrm{d}t} \displaystyle=f(m,z) (4) where \sigma_{m}(t) and \sigma_{z}(t) act as mixing coefficients that modulate the magnitude

Read the complete text