Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning

Minhao Yao, Ruoyu Wang, Xihong Lin, Lin Liu, Zhonghua Liu · 2026

Read Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning by Minhao Yao, Ruoyu Wang, Xihong Lin, Lin Liu, Zhonghua Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27991.txt.

Introduction

Yao and colleagues turn two training choices into statistics problems: whether to train at all, and when to stop. Viewing gradient-flow training through a random-effects lens, they recast training time as a variance parameter and derive a principled early-stopping rule from restricted maximum likelihood. Tests on deep models and a large proteomics dataset show competitive accuracy with less reliance on validation splits.

Catalog record

CREATORMinhao Yao, Ruoyu Wang, Xihong Lin, Lin Liu, Zhonghua Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122098
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning Minhao Yao, Ruoyu Wang, Xihong Lin, Lin Liu, Zhonghua Liu Abstract Gradient-flow optimization is usually viewed as an algorithmic procedure for minimizing empirical loss, with training duration selected by validation or heuristic early-stopping rules. We develop a statistical inference framework for the gradient-flow training trajectory itself. The central object is fixed-operator squared-error gradient flow: whenever the fitted value evolves through a time-invariant positive semidefinite training operator, the trained model output at each training time is exactly equivalent to the best linear unbiased predictor, or empirical-Bayes posterior mean, under a corresponding random-effects model. Under this representation, training time becomes a variance-component parameter governing how variance is reallocated from residual noise to structured signal. This turns two basic training decisions into inferential problems. First, whether training is needed is formulated as a variance-component test for signal beyond initialization. Second, how long to train is formulated as restricted maximum likelihood (REML) estimation of the training-time variance component. The resulting REML-guided early stopping rule has a spectral interpretation: it selects the training time at which optimized spectral losses become empirically decorrelated from the eigenvalues of the training operator, yielding an effective degrees-of-freedom measure for the evolving trained model. We establish asymptotic prediction optimality for fixed-design in-sample risk and, under additional kernel regularity conditions, random-design out-of-sample risk. Deep learning models in fixed-kernel gradient regimes provide canonical modern-AI instantiations of the theory. Numerical experiments and a UK Biobank proteomics application show that the proposed inferential approach attains competitive prediction accuracy while reducing the reliance on validation splits and repeated checkpoint evaluation. * Equal contribution; † Correspondence: [email protected] Keywords : Statistics for AI; fixed-operator gradient flow; random-effects models; variance components; empirical Bayes; REML; early stopping; spectral regularization; effective degrees of freedom; deep learning; prediction optimality. 1 Introduction Deep learning has achieved remarkable success across a wide range of scientific applications ( 37 ) , including protein structure prediction ( 34 ; 1 ) , molecular design ( 18 ) , and mathematical discovery ( 5 ; 13 ) . Despite this success, a principled statistical understanding of deep learning models as predictive procedures remains incomplete ( 59 ; 7 ) . A deep learning predictor is shaped by several interacting components, including the function class, architecture, loss function, optimization algorithm, and stopping rule. Recent statistical overviews have clarified many aspects of approximation, optimization, over-parameterization, and generalization ( 24 ; 7 ) , but a statistical interpretation of the training trajectory remains less developed. This limitation is central to the interpretability of deep learning models as statistical procedures and is closely related to the broader concern that deep learning often behaves as black-box models ( 57 ) . A substantial statistical literature studies deep learning models as flexible nonparametric regression estimators, focusing on how network architecture, function-class complexity, and structural assumptions on the target function or covariate distribution determine the accuracy of empirical-risk-minimizing predictors ( 60 ; 8 ; 35 ; 25 ; 33 ) . These works have greatly advanced the statistical theory of deep learning. However, they provide an incomplete account of the fitted predictor produced by actual training, since they typically focus on the global empirical-risk minimizer, which may not be attainable in practice because of the complicated loss landscape of neural network training ( 2 ) . The final predictor is therefore shaped not only by the function class and loss, but also by the optimization trajectory and the time at which training is stopped ( 27 ) . Understanding the statistical role of training itself consequently requires a complementary theory of the training path and stopping time. This leaves both a conceptual and a methodological gap. Conceptually, we lack a statistical characterization of how optimization reallocates variation from residual noise to structured signal along the training path. Methodologically, we lack a corresponding statistical framework that turns the decisions of whether to train and when to stop into explicit inferential problems, rather than relying solely on validation-based or heuristic early-stopping rules ( 52 ; 78 ; 27 ) . The central question of this paper is whether the training path itself can be given an inferential interpretation. In particular, we ask whether training time can be treated not merely as an external tuning parameter, but as an estimable statistical parameter governing signal extraction and model complexity. We address this question by developing a general optimization–inference duality for fixed-operator squared-error gradient flow. Let f_{t}({\bm{X}}) denote the vector of fitted values on n training samples {\bm{X}}=({\bm{x}}_{1},\cdots,{\bm{x}}_{n})^{\top} at training time t , and let {\bm{y}} denote the response vector. Suppose that f_{t}({\bm{X}}) evolves according to \frac{\mathrm{d}f_{t}({\bm{X}})}{\mathrm{d}t}=-{\bm{H}}\{f_{t}({\bm{X}})-{\bm{y}}\}, where {\bm{H}} is a time-invariant positive semidefinite training operator induced by a fixed positive semidefinite kernel h(\cdot,\cdot) , with entries {\bm{H}}_{ij}=h({\bm{x}}_{i},{\bm{x}}_{j}) for 1\leq i,j\leq n . The operator {\bm{H}} determines the spectral structure of the training dynamics on the observed sample, with its eigenvectors defining residual directions and its eigenvalues controlling their decay rates. This formulation includes early-stopped linear regression ( 4 ) , fixed-kernel gradient flow ( 61 ) , linearized predictor, and, as a canonical deep learning instance, infinite-width neural networks in the neural tangent kernel (NTK) regime ( 31 ) . Within our framework, the gradient-flow trajectory is exactly equivalent, at every training time t , to the best linear unbiased predictor (BLUP), or empirical Bayes posterior mean, under a corresponding working random-effects model ( 30 ; 22 ) . Thus, algorithmically, gradient flow is an optimization path; statistically, the same path is a random-effects inference path in which training time acts as a variance-component parameter governing how variation is reallocated from residual noise to structured signal. We emphasize that the scope of our contribution is broader than the NTK regime. The NTK provides a well-established fixed-kernel description of neural-network training in suitable infinite-width regimes ( 31 ; 21 ; 38 ; 6 ; 50 ; 75 ) , and serves here as a canonical deep-learning instantiation of our theory. Importantly, this fixed-kernel description is not limited to fully connected networks: NTK training dynamics have been shown to be architecturally universal for broad classes of neural network architectures, including convolutional, recurrent, residual, graph, and attention-based architectures ( 75 ) . More recently, NTK-based and linearized quantities have also been used in modern AI systems, including transformer fine-tuning and large language model adaptation ( 45 ; 3 ; 72 ; 40 ) , further illustrating the practical relevance of fixed-kernel and linearized perspectives. The optimization–inference duality developed in this paper, however, is not an NTK-specific phenomenon. It is a property of fixed-operator squared-error gradient flow more generally. This distinction is also important from the perspective of classical statistics. Although mixed-model representations of kernel ridge regression and smoothing splines are classical, those formulations typically start from a fixed kernel or an explicit quadratic penalty and lead to a static ridge-type regularization parameter ( 70 ; 65 ; 58 ; 43 ) . The random-effects representation developed here is fundamentally different: the covariance \exp(t\bm{H})-\bm{I} is determined by the residual-decay operator \exp(-t\bm{H}) of the gradient-flow trajectory itself, and the resulting dynamic BLUP at each training time t reproduces the corresponding early-stopped predictor. Thus, the novelty is not

Read the complete text