Implicit Regularization and Generalization in Overparameterized Neural Networks
Zeran Johannsen · 2026
Read Implicit Regularization and Generalization in Overparameterized Neural Networks by Zeran Johannsen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07603.txt.
Introduction
Johannsen studies hidden regularization and generalization in oversized neural nets in controlled tests. Old theory predicts severe overfitting for models with far more weights than samples, yet deep nets generalize well. He tests stochastic gradient descent across batches, flat versus sharp minima, the Neural Tangent Kernel, double descent, and the Lottery Ticket Hypothesis. Findings show generalization hinges on design, tuning, and loss shape. Smaller batches yield lower test error and flatter minima, and sparse subnetworks with only 10 percent of weights stay within 1.15 points of full-model scores.
Catalog record
| CREATOR | Zeran Johannsen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62503 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07603
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Implicit Regularization and Generalization in Overparameterized Neural Networks Zeran Johannsen Abstract Modern deep neural networks are frequently trained in regimes where the number of model parameters greatly exceeds the number of training samples. Classical statistical learning theory predicts that such highly overparameterized models should exhibit severe overfitting and poor generalization performance. However, empirical evidence across computer vision, natural language processing, and reinforcement learning consistently demonstrates that large neural networks trained with gradient-based optimization methods often generalize remarkably well. This apparent contradiction has become one of the central theoretical questions in contemporary machine learning research. Prior theoretical frameworks based on classical capacity measures, such as VC dimension and uniform convergence bounds, struggle to adequately explain why heavily overparameterized neural networks avoid overfitting in practice. While these frameworks predict that increasing model capacity should degrade generalization performance, modern neural networks often show the opposite behavior. This discrepancy has motivated a growing body of research aimed at identifying mechanisms that implicitly constrain the effective capacity of neural networks during training. This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled computational experiments. The analysis examines the behavior of stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima in the loss landscape via Hessian eigenvalue estimation and weight perturbation analysis, theoretical perspectives derived from the Neural Tangent Kernel (NTK) regime through wide-network experiments, the double descent phenomenon across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments were conducted using PyTorch on CIFAR-10 and MNIST benchmarks with multiple random seeds. The findings demonstrate that generalization in overparameterized models is strongly influenced by the interaction between network architecture, optimization algorithms, and the geometry of the loss landscape. Smaller batch sizes consistently produced lower test error and flatter minima. Sparse subnetworks retaining only 10% of original parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results contribute to a deeper theoretical understanding of modern deep learning systems and highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes. I Introduction Over the past decade, deep learning has achieved remarkable empirical success across a wide range of domains, including computer vision, natural language processing, speech recognition, and scientific modeling. Many of the most successful models in these domains are heavily overparameterized, meaning that the number of learnable parameters in the network significantly exceeds the number of training examples available. Contemporary architectures such as deep convolutional neural networks and large transformer-based language models frequently contain millions to billions of parameters, often trained on datasets that are comparatively small relative to model capacity. Despite this extreme parameterization, these models frequently demonstrate strong generalization performance on unseen data. This phenomenon presents a fundamental challenge to classical statistical learning theory. Traditional theoretical frameworks suggest that models with high representational capacity should be prone to overfitting, particularly when the number of parameters exceeds the number of training samples. Concepts such as the Vapnik–Chervonenkis (VC) dimension [ 15 ] and uniform convergence bounds predict that increasing model complexity should increase the risk of fitting noise in the training data, thereby degrading generalization performance. From this perspective, heavily overparameterized neural networks should exhibit poor predictive performance outside the training dataset. However, empirical results in modern machine learning consistently contradict this expectation [ 17 ] . The discrepancy between theoretical predictions and observed performance has motivated extensive research aimed at understanding the mechanisms that enable generalization in overparameterized neural networks. One prominent line of inquiry focuses on the role of optimization algorithms, particularly stochastic gradient descent (SGD) and its variants. Although SGD is primarily designed as an optimization method rather than a regularization technique, growing evidence suggests that it implicitly biases the learning process toward solutions that generalize well. This phenomenon, commonly referred to as implicit regularization, may constrain the effective capacity of neural networks even when their parameter counts are extremely large [ 14 ] . Another important perspective concerns the geometry of the loss landscape in high-dimensional neural network parameter spaces. Empirical and theoretical studies have suggested that solutions located in flatter regions of the loss surface tend to generalize better than those located in sharp minima [ 9 , 11 ] . Flat minima correspond to regions in parameter space where small perturbations to model weights do not substantially increase the training loss, implying a degree of robustness to noise and parameter variation. Optimization algorithms such as SGD may preferentially converge to these flatter regions, thereby contributing to improved generalization behavior. In parallel with these optimization-based explanations, several theoretical frameworks have emerged that attempt to characterize neural network behavior in highly overparameterized regimes. One influential approach is the Neural Tangent Kernel (NTK) framework [ 10 ] , which analyzes the training dynamics of infinitely wide neural networks. In this regime, the evolution of network parameters during gradient descent can be approximated by kernel methods, providing theoretical insights into convergence and generalization properties. Although the NTK framework represents an idealized limit, it offers a useful analytical perspective on how large neural networks behave during training. Another key empirical observation relevant to this problem is the double descent phenomenon [ 3 , 13 ] . Classical bias–variance tradeoff theory predicts that test error should follow a U-shaped curve as model complexity increases. However, recent studies have demonstrated that in highly overparameterized regimes, test error can exhibit a second descent after the interpolation threshold, where models become large enough to perfectly fit the training data. This observation suggests that increasing model capacity beyond the interpolation point may actually improve generalization performance, challenging long-standing assumptions about model complexity and overfitting. Complementary insights are provided by the Lottery Ticket Hypothesis [ 7 ] , which proposes that large neural networks contain sparse subnetworks that are capable of achieving performance comparable to the full model when trained in isolation. According to this hypothesis, the success of overparameterized networks may stem from the presence of these favorable subnetworks, which can be discovered through the optimization process. The broader network architecture effectively acts as a search space within which training algorithms identify these high-performing subnetworks. Taken together, these perspectives suggest that generalization in modern neural networks cannot be fully explained by traditional complexity-based learning theory alone. Instead, the interaction between model architecture, optimization algorithms, loss landscape geometry, and implicit inductive biases appears to play a critical role in determining generalization performance. Understanding these interactions is essential not only for theoretical clarity but also for the practical design of more reliable and efficient machine learning systems. The objective of this study is to examine why overparameterized neural networks generalize effectively despite their extremely large parameter counts. Specifically, the research investigates how implicit regularization induced by stochastic optimization, the geometry of flat and sharp minima, kernel-based theoretical perspectives such as the Neural Tangent Kernel, and empirical phenomena including double descent and the Lottery Ticket Hypothesis collectively contribute to generalization behavior. By combining controlled computational experiments with theoretical analysis, this work aims to clarify the mechanisms that allow modern neural networks to perform well in regimes that appear incompatible with classical statistical learning principles. I-A Contributions This study makes the following