Estimating Implicit Regularization in Deep Learning

Joseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording · 2026

Read Estimating Implicit Regularization in Deep Learning by Joseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05436.txt.

Introduction

Neural networks quietly prefer simple solutions, a bias called implicit regularization. Joseph H. Rudoler, Kevin Tan, Giles Hooker, and Konrad P. Kording offer a way to measure that hidden penalty for any network. Their tool matches gradients: since regularization pushes updates away from plain loss gradients, the gap reveals the penalty at work.

The method recovered well-known penalties on test networks, including the effect of early stopping. It also worked on arbitrary networks where no formula exists. Applied to dropout, it showed the popular trick quietly acts like a squared-weight penalty, giving practitioners a clearer picture of their choices.

Catalog record

CREATORJoseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05436

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Estimating Implicit Regularization in Deep Learning Joseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording Abstract Deep learning systems are known to exhibit implicit regularization (alt. implicit bias ), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization – connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like ℓ 1 \ell_{1} and ℓ 2 \ell_{2} . It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing the effects of dropout in deep networks, showing implicit ℓ 2 \ell_{2} effects in this popular method. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically. 1 Introduction Traditional statistical learning theory teaches us that adding more parameters to our predictive models increases the chances of overfitting to training data at the expense of their ability to generalize to novel examples. One of the great challenges of modern machine learning is to understand how massively over-parametrized neural networks which can effectively memorize their training sets still manage to generalize [ 46 , 31 ] . Recent progress has shown that the solution to the overfitting problem is for learning systems to favor simplicity: successful algorithms should prioritize parsimonious or highly-compressible solutions [ 42 , 32 , 28 , 37 , 45 , 22 ] . Neural networks therefore require constraints on their solution space – whether explicit or implicit – in order to achieve the impressive practical performance we observe. We often explicitly regularize or constrain models, adding inductive biases which can improve generalization. Examples of explicit inductive biases include Bayesian priors [ 23 , 29 , 41 ] , weight decay [ 14 ] , and the translation invariance introduced in convolutional neural networks [ 20 ] . Explicit regularization is one way of enforcing simplicity. We also generally implicitly regularize. A growing literature studies the implicit regularization effects of gradient-based learning [ 35 , 4 , 3 , 11 , 30 , 27 ] and various algorithmic or architectural choices like early-stopping [ 33 , 44 , 2 ] or dropout [ 39 , 24 , 25 , 9 , 15 , 40 ] . Most theory casts implicit regularization in the form of a bias towards simpler functions (often via norm or rank minimization). It is believed that implicit regularization is essential to the success of deep learning in the over-parameterized regime [ 6 , 42 , 8 , 43 , 1 ] . Rigorous analysis of implicit regularization is generally limited to simple settings. Much existing work focuses on tasks (deep matrix factorization [ 13 , 4 ] ), models (shallow and/or linear networks [ 24 , 25 , 9 , 11 ] ), and algorithms (gradient flow or full-batch gradient descent [ 11 , 26 , 38 ] ) that differ significantly from those employed by deep learning practitioners. The claim of these theoretical studies is often that the conclusions drawn in simplified settings might extend more generally, raising the need for empirical validation in larger networks. In this paper we develop tools to empirically estimate implicit bias, with implications both for the theoretical study of learning algorithms and for training state of the art deep learning systems. Estimating implicit bias should allow theoreticians to empirically validate their analytical results and extend the study of implicit bias to complex networks that evade mathematical analysis, thereby providing a stronger bridge between theory and practice. For empirical researchers training deep neural networks, even heuristic estimates of implicit regularization can be a useful diagnostic. While there is some literature on quantifying inductive bias, broadly construed, to our knowledge we are the first to directly estimate parametrized models of implicit bias as explicit regularizers. We summarize and contrast other approaches with our own in Appendix E . Explicitly modeling the bias permits directly testing theoretical models, and generally produces structured and interpretable results. Our task is to find a mathematically expressible regularizer \mathcal{R} , such that explicitly optimizing the empirical loss \mathcal{L} augmented by \mathcal{R} is functionally the same as an implicitly biased procedure optimizing \mathcal{L} alone. Given an a priori parametric family of regularizers, \mathcal{R}(\cdot,\Lambda) , we seek to estimate the values of the regularization parameters \Lambda . As a running example, \ell_{2} regularization has been connected to common techniques like early stopping and dropout. This regularizer can be written as \mathcal{R}(\theta,\lambda)=\lambda\|\theta\|_{2}^{2} where \theta are the weights of a predictive model. Estimating \lambda determines the strength of the technique’s effective \ell_{2} regularization. Information about the implicit regularization is clearly embedded in the dynamics of learning. Specifically, if the gradient of the training loss does not converge to zero, this may reflect additional constraints imposed by the optimization procedure which we seek to recover. Alternatively, the same information can be gleaned at each timestep from discrepancies between the gradient of the loss and the update direction. These deviations from simply going down the loss gradient are indicative of the underlying regularization. We perform estimation by considering the learned weights to be the optimum of a hypothetical learning procedure that minimizes both the empirical loss and the implicit regularizer. Alternatively, one can consider deviations of weight updates from the strict empirical loss gradient to be guided by the implicit regularizer. Each of these perspectives implies a system of equations involving computable gradients of the loss and the candidate regularizer, in which the only unknowns are the regularization parameters \Lambda . This framework is general, encompassing arbitrary features (weights, activations, data, etc.) and flexible parameterizations (any differentiable function class). Our contributions are as follows: • How to estimate implicit bias. A novel empirical framework for estimating parametrized models of implicit regularization. We do so by matching the gradients of the loss to the gradients of a proposed regularizer, based on a decomposition of the empirical loss. • Characterizing successful estimation. We distinguish between two modes of success: 1) recovering ground truth regularization and 2) recovering a regularizer that mimics the effect of the implicit regularization. We show that indeed we achieve (1) in simple settings where results are predicted by theory. We can evaluate (2) by estimating a regularizer and using it to retrain the model, then comparing the retrained model to the original. We then provide a theoretical characterization of the conditions needed to

Read the complete text