Gradient Flow Matching for Learning Update Dynamics in Neural Network Training
Xiao Shou, Yanna Ding, Jianxi Gao · 2025
Read Gradient Flow Matching for Learning Update Dynamics in Neural Network Training by Xiao Shou, Yanna Ding, Jianxi Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20221.txt.
Introduction
Xiao Shou, Yanna Ding, and Jianxi Gao treat neural network training as a moving system whose path can be learned and forecast, in work from 2025. Training is costly because gradient steps repeat so many times. If the path of the weights could be extrapolated, runs could be cut short.
Their method learns the update rules of common optimizers as smooth flow fields. It bakes the shape of gradient updates into the learning goal, unlike black-box sequence models. From partial training runs, it forecasts final weights well. Scores rivaled transformer models and beat older sequence baselines. The method also carried across network designs and starts, which the authors offer as a base for studying training itself.
Catalog record
| CREATOR | Xiao Shou, Yanna Ding, Jianxi Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient Flow Matching for Learning Update Dynamics in Neural Network Training Xiao Shou, Yanna Ding, Jianxi Gao Abstract Training deep neural networks remains computationally intensive due to the iterative nature of gradient-based optimization. We propose Gradient Flow Matching (GFM) , a continuous-time modeling framework that treats neural network training as a dynamical system governed by learned optimizer-aware vector fields. By leveraging conditional flow matching, GFM captures the underlying update rules of optimizers such as SGD, Adam, and RMSprop, enabling smooth extrapolation of weight trajectories toward convergence. Unlike black-box sequence models, GFM incorporates structural knowledge of gradient-based updates into the learning objective, facilitating accurate forecasting of final weights from partial training sequences. Empirically, GFM achieves forecasting accuracy that is competitive with Transformer-based models and significantly outperforms LSTM and other classical baselines. Furthermore, GFM generalizes across neural architectures and initializations, providing a unified framework for studying optimization dynamics and accelerating convergence prediction. 1 Introduction Deep neural networks (DNNs) have achieved remarkable success in tasks such as image classification ( LeCun et al. 1998 ) and natural language processing ( Vaswani et al. 2017 ) , driven by increasingly deep and complex architectures. As models scale to hundreds of layers and billions of parameters, training them requires vast computational resources and prolonged optimization cycles. The resulting workloads scale non-linearly with model size, creating serious bottlenecks in time and resource efficiency. While inference has benefited from acceleration techniques like model compression ( Tan and Le 2019 ) and knowledge distillation ( Sanh et al. 2019 ) , improving training efficiency remains a more fundamental challenge. This is largely because gradient-based optimization is inherently iterative: algorithms like SGD and Adam ( Kingma and Ba 2014 ) update weights step-by-step, requiring thousands of iterations to reach convergence. Increasing hardware capacity helps, but cannot fully eliminate the cost of these sequential updates. Addressing this bottleneck calls for methods that rethink how optimization trajectories are modeled and predicted. Recent studies have revealed that weight evolution during training exhibits predictable patterns, opening opportunities for forecasting-based acceleration. For instance, the Weight Nowcaster Network (WNN) ( Jang et al. 2023 ) predicts near-future weights to skip small optimization steps, while LFD-2 ( Shou et al. 2024 ) performs long-horizon "farcasting" from partial weight sequences. However, these methods treat optimization as a generic sequence modeling problem, without explicitly modeling the underlying dynamics of gradient-based updates. Consequently, they overlook the structured relationship between gradients, learning rates, and weight trajectories that governs the optimization process itself. In this work, we propose Gradient Flow Matching (GFM), a continuous-time framework that models neural network training as a dynamical system governed by optimizer-aware vector fields. Leveraging conditional flow matching ( Tong et al. 2023 ) , GFM reinterprets weight trajectories not as arbitrary sequences, but as realizations of structured flows conditioned on partial observations and optimization algorithms. This perspective enables more accurate and generalizable forecasting of final weights by explicitly incorporating the dynamics of gradient-based updates. Our main contributions are as follows: • A flow-matching-compatible formulation of first-order dynamics. While continuous-time views of gradient descent are well-studied, we tailor them specifically for conditional flow matching, enabling the learning of vector fields that are both optimizer-aware and trajectory-faithful. • A conditional flow matching loss for weight evolution. We introduce a loss function that aligns learned vector fields with observed parameter updates, supporting both short-term extrapolation and long-horizon forecasting within a unified framework. • Extension to momentum and adaptive optimizers. GFM accommodates common optimizers such as Adam ( Kingma and Ba 2014 ) and RMSProp ( Shi and Li 2021 ) , capturing their dynamics within a continuous-time approximation through the learned flow field. By marrying weight trajectory forecasting with continuous-time modeling, GFM offers a unified and extensible lens for analyzing and predicting the evolution of neural network weights. 2 Related Work Weight Forecasting and Acceleration. The idea of forecasting weights to accelerate training dates back to Introspection ( Sinha et al. 2017 ) , which used past training history to predict future parameters. The Weight Nowcaster Network (WNN) ( Jang et al. 2023 ) extended this idea to short-horizon prediction, enabling periodic skipping of optimization steps. While effective on small-scale models, WNN and its variants typically forecast only a few steps ahead and treat each weight independently. To support long-range forecasting, Shou et al. ( Shou et al. 2024 ) proposed "farcasting" with LFD-2, showing that even simple models can extrapolate full training trajectories from just the initial and final weights. Knyazev et al. ( Knyazev et al. 2025 ) concurrently introduced a graph-based approach (NiNo) that incorporates architectural structure into the forecasting process. Other acceleration strategies include meta-learned optimizers such as L2O ( Chen et al. 2022 ) , which train neural networks to generate optimizer updates, albeit with significant meta-training cost. Simpler alternatives like Lookahead ( Zhang et al. 2019 ) reduce overhead by interleaving fast inner updates with slow outer updates, leveraging future trends without explicitly predicting them. Flow-Based and Continuous-Time Models. Continuous gradient flows provide an alternative perspective on optimization by interpreting training as a trajectory governed by an ordinary differential equation (ODE). Neural ODEs ( Chen et al. 2018 ) enable continuous-time modeling of hidden states in deep networks, offering flexibility for irregularly spaced observations. However, they can be computationally expensive due to the need for numerical integration at each time step, particularly in stiff or high-dimensional systems. Moreover, standard Neural ODEs are not specifically designed to capture the structure of gradient-based optimization dynamics. In practice, many forecasting methods discretize these trajectories using fixed steps, making them analogous to sequence models like LSTMs. Flow Matching (FM) ( Lipman et al. ) and its conditional variant ( Tong et al. 2023 ) offer a scalable, simulation-free way to learn continuous-time dynamics. Time Series Forecasting. Our work is related to long-term time series forecasting, where both classical and modern models have been extensively explored. Recent studies have demonstrated that simple linear models can rival deep sequence models for long-horizon prediction, as shown in DLinear and its variants ( Zeng et al. 2023 ) . Yet applying standard time-series models such as ( Nie et al. 2022 ) or RNNs directly to neural weight sequences is often impractical due to the high dimensionality and nonstationary nature of the optimization trajectory. While recent efforts explore foundation models for time series ( Ye et al. 2024 ) , their computational demands remain a challenge in our setting. Learning Curve Extrapolation. An orthogonal line of research focuses on predicting the evolution of training loss or accuracy over time, rather than modeling the underlying weight trajectories. These approaches are often used to enable early stopping or to estimate final model performance. Prior work on learning curve extrapolation includes Bayesian methods ( Klein et al. 2017 ) as well as techniques based on historical learning trends ( Gargiani et al. 2019 ) . In a related direction, architecture-based performance prediction has leveraged graph neural networks to estimate the final accuracy of unseen architectures ( Ding et al. 2025 ) . 3 Preliminary Gradient Descent. First-order gradient descent methods are the foundational tools for optimizing machine learning and deep learning models. The objective is to minimize a loss function L with respect