Towards Understanding the Optimization Mechanisms in Deep Learning
Binchuan Qi, Wei Gong, Li Li · 2025
Read Towards Understanding the Optimization Mechanisms in Deep Learning by Binchuan Qi, Wei Gong, Li Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23016.txt.
Introduction
Binchuan Qi, Wei Gong, and Li Li look at why training deep neural networks works, in work from 2025. They view training as estimating a probability distribution for the correct class labels. Even though the fitting error twists in complex ways as settings change, they show good solutions can still be approached.
Their approach splits the goal into two parts: shrinking the slope of the error and shrinking a structural error term. The first part is handled by standard descent methods. For the second, they prove that adding more parameters and keeping them independent helps. This gives a theory for why oversized networks and random starting values work well in practice. Tests back up the main claims.
Catalog record
| CREATOR | Binchuan Qi, Wei Gong, Li Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23016
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Understanding the Optimization Mechanisms in Deep Learning Binchuan Qi, Wei Gong, Li Li Abstract In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model’s parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness. keywords Deep Learning, Optimization Mechanism, Probability Distribution Estimation. 1 Introduction Alongside the remarkable practical achievements of deep learning, the optimization mechanisms regarding deep learning remain unanswered within the classical learning theory framework. Key insights from the studies Yun et al. ((2018)) ; Du, Zhai et al. ((2018)) ; Chizat et al. ((2018)) ; Arjevani & Field ((2022)) emphasize the pivotal role of over-parameterization in finding the global optimum and enhancing the generalization ability of deep neural networks (DNNs). Recent work has shown that the evolution of the trainable parameters in continuous-width DNNs during training can be captured by the neural tangent kernel (NTK) Jacot et al. ((2018)) ; Du, Lee et al. ((2018)) ; Zou et al. ((2018)) ; Arora et al. ((2019)) ; Mohamadi et al. ((2023)) ; Wang et al. ((2023)) . An alternative research direction attempts to examine the infinite-width neural network from a mean-field perspective ( Sirignano & Spiliopoulos, (2018) ; Mei et al., (2018) ; Chizat & Bach, (2018) ; Nguyen & Pham, (2023) ) . However, in practical applications, neural networks are of finite width, and under this condition, it remains unclear whether NTK theory and mean-field theory can adequately characterize the convergence properties of neural networks Seleznova & Kutyniok ((2021)) . Therefore, the mechanisms of non-convex optimization in deep learning, and the impact of over-parameterization on model training, remain incompletely resolved. Motivation . A substantial proportion of machine learning tasks can be conceptualized within the domain of probability distribution estimation. Supervised classification and regression tasks are mathematically characterized by the process of learning the conditional probability distribution of labels given input features. Generative learning involves estimating the underlying joint distribution of features. Consequently, adopting a perspective of probability distribution estimation to analyze deep learning is a logical and coherent choice, and it is applicable to various types of learning tasks. One challenge in analyzing the optimization mechanisms in deep learning lies in the diversity of loss functions used, which leads to different characteristics of the optimization objectives. This makes it difficult to handle them within a unified framework. However, Blondel et al. ((2019)) demonstrated that most loss functions currently used in practical applications can be expressed in the form of Fenchel-Young loss. Therefore, using Fenchel-Young loss is a reasonable and effective approach to reduce the complexity of analysis. Contribution . The contributions of this paper are stated as follows. 1. This paper proves that when using the Fenchel-Young loss, classification problems in machine learning are equivalent to conditional probability distribution estimation given the features. Additionally, this paper demonstrates that the Fenchel-Young loss possesses implicit regularization capabilities, thereby justifying its use as a standalone optimization objective. 2. This paper proves that although the problem of fitting and learning conditional probability distributions constitutes a non-convex optimization problem when using DNNs, its global optimum is equivalent to its stationary points. Specifically, we can approximate the global optimal solution by reducing both the gradient norm and structural error. This conclusion elucidates the non-convex optimization mechanism underlying model training in deep learning. 3. This paper demonstrates that under the assumption of gradient independence, the structural error is controlled by the number of model parameters; that is, the larger the number of parameters, the smaller the corresponding structural error. This conclusion provides theoretical insights into techniques such as over-parameterization, random parameter initialization, and dropout. 4. Key conclusions drawn from this framework are validated through experimental results. Organization . The paper is organized as follows:Section 2 reviews the research progress and current status. Fundamental concepts are defined in Section 3 . The proposed method and conclusions are detailed in Section 4 . Experimental settings and results are presented in Section 5 . The conclusions are summarized in Section 6 . 2 Related Work Despite the inherent non-convexity of the objective functions, empirical evidence indicates that gradient-based methods, such as stochastic gradient descent (SGD) are capable of converging to the global minima in these networks. Existing studies Du, Zhai et al. ((2018)) ; Chizat et al. ((2018)) ; Arjevani & Field ((2022)) indicate that over-parameterization plays a pivotal role in finding the global optimum and enhancing the generalization ability of DNNs. Therefore, analyzing the optimization mechanisms of deep learning from the perspective of model parameter scale has become an important research direction. The NTK thus has emerged as a pivotal concept, as it captures the dynamics of over-parameterized neural network trained by GD Jacot et al. ((2018)) . It is already known in the literature that DNNs in the infinite width limit are equivalent to a Gaussian process Neal & Neal ((1996)) ; Williams ((1996)) ; Winther ((2000)) ; Neal ((2012)) ; Lee et al. ((2017)) . The work of Jacot et al. ((2018)) elucidates that the evolution of the trainable parameters in continuous-width DNNs during training can be captured by the NTK. Some work has shown that with a specialized scaling and random initialization, the parameters of continuous width two-layer DNNs are restricted in an infinitesimal region around the initialization and can be regarded as a linear model with infinite dimensional features Du, Zhai et al. ((2018)) ; Li & Liang ((2018)) ; Du, Lee et al. ((2018)) ; Arora et al. ((2019)) . Since the system becomes linear, the dynamics of GD within this region can be tracked via properties of the associated NTK and the convergence to the global optima with a linear rate can be proved. Later, the NTK analysis of global convergence is extended to multi-layer neural nets Allen-Zhu, Li & Liang ((2018)) ; Zou et al. ((2018)) ; Allen-Zhu, Li & Song ((2018)) ; Mohamadi et al. ((2023)) ; Wang et al. ((2023)) . An alternative research direction attempts to examine the infinite-width neural network from a mean-field perspective ( Sirignano & Spiliopoulos, (2018) ; Mei et al., (2018) ; Chizat & Bach, (2018) ; Nguyen & Pham, (2023) ) . The key idea is to characterize the learning dynamics of noisy SGD as the gradient flows over the space of probability distributions of neural network parameters. When the time goes to infinity, the noisy SGD converges to the unique global optimum of the convex objective function for the two-layer neural network with continuous width. NTK and mean-field methods provide a fundamental understanding on training dynamics and convergence properties of non-convex optimization in infinite-width neural networks. However, the assumption of infinitely wide neural networks does not hold in real-world applications,