Optimized Gradient Clipping for Noisy Label Learning
Xichen Ye, Yifan Wu, Weizhong Zhang, Xiaoqiang Li, Yifan Chen, Cheng Jin · 2024
Read Optimized Gradient Clipping for Noisy Label Learning by Xichen Ye, Yifan Wu, Weizhong Zhang, Xiaoqiang Li, Yifan Chen, Cheng Jin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08941.txt.
Introduction
Xichen Ye and co-authors proposed in 2024 a smarter way to clip gradients when training neural networks on mislabeled data. Clipping keeps noisy examples from yanking the model around, but the usual fixed threshold ignores how the mix of clean and noisy gradients shifts during training. Their method resets the threshold at every step.
It estimates, from modeled distributions, how much of the surviving gradient comes from noise, then tightens or loosens the clip to hold that share in check. Theory certifies the noise tolerance of the scheme. Experiments across synthetic and real label noise showed consistent gains over fixed clipping. The authors call it a small change with broad effect. Robust training, they argue, should adapt as fast as the gradients do.
Catalog record
| CREATOR | Xichen Ye, Yifan Wu, Weizhong Zhang, Xiaoqiang Li, Yifan Chen, Cheng Jin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08941
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimized Gradient Clipping for Noisy Label Learning Xichen Ye, Yifan Wu, Weizhong Zhang, Xiaoqiang Li, Yifan Chen, Cheng Jin Abstract Previous research has shown that constraining the gradient of loss function w.r.t. model-predicted probabilities can enhance the model robustness against noisy labels. These methods typically specify a fixed optimal threshold for gradient clipping through validation data to obtain the desired robustness against noise. However, this common practice overlooks the dynamic distribution of gradients from both clean and noisy-labeled samples at different stages of training, significantly limiting the model capability to adapt to the variable nature of gradients throughout the training process. To address this issue, we propose a simple yet effective approach called Optimized Gradient Clipping (OGC), which dynamically adjusts the clipping threshold based on the ratio of noise gradients to clean gradients after clipping, estimated by modeling the distributions of clean and noisy samples. This approach allows us to modify the clipping threshold at each training step, effectively controlling the influence of noise gradients. Additionally, we provide statistical analysis to certify the noise-tolerance ability of OGC. Our extensive experiments across various types of label noise, including symmetric, asymmetric, instance-dependent, and real-world noise, demonstrate the effectiveness of our approach. 1 Shanghai University 2 Fudan University 3 Hong Kong Baptist University 4 Shanghai Key Laboratory of Intelligent Information Processing 5 Shanghai Collaborative Innovation Center of Intelligent Visual Computing {yexichen0930, victorwu, xqli}@shu.edu.cn, {weizhongzhang, jc}@fudan.edu.cn, [email protected] Code — https://github.com/Virusdoll/OGC 1 Introduction The effectiveness of supervised deep learning relies heavily on the availability of large-scale, qualifiedly-annotated data. Research has shown that an over-parameterized Deep Neural Network (DNN) can easily fit a dataset with randomly assigned labels ( Zhang et al. 2017 ) , underscoring the importance of high-quality annotations. However, in real-world scenarios, the annotation process will inevitably introduce noisy labels (incorrectly annotated labels), which is liable to compromise the performance of the model. Therefore, the study of noisy label learning has attracted significant attention. The goal of noisy label learning is to train a model on a given corrupted dataset, which contains mislabeled samples, while mitigating the adverse effects of noisy labels, and ultimately enabling the trained model to generalize well on a clean evaluation set. Figure 1: KDE visualizations of gradient distributions for clean and noisy labels, along with decision boundary visualizations for a simple binary classification task that utilizes gradient clipping with a fixed threshold. The leftmost plot in the first row shows the raw training data. The subsequent plots illustrate shifts in gradient distributions (before clipping) at various training epochs (50, 500, 1000, and 1500). The second row displays the corresponding decision boundaries for each epoch. Cross Entropy (CE), the most widely used loss function for classification, is nevertheless vulnerable to label noise. In contrast, symmetric loss functions like Mean Absolute Error (MAE) are noise tolerant but less effective for training classification models ( Ghosh, Kumar, and Sastry 2017 ; Zhang and Sabuncu 2018 ) . This discrepancy has driven the studies into robust loss functions, which primarily focus on how to make full use of both CE and MAE ( Zhang and Sabuncu 2018 ; Feng et al. 2020 ; Englesson and Azizpour 2021 ; Wang et al. 2019 ; Ma et al. 2020 ; Ye et al. 2023 ) and have achieved notable success. Besides these, another promising approach, clipping-based techniques, also warrants greater attention. Among them, Menon et al. (2020) first introduced the PHuberCE method, which refines traditional gradient clipping by applying gradient clipping to model-predicted probabilities (see the definition in Section 3.1 ), offering an improved approach to handling noisy labels. More recently, Wei et al. (2023) proposed LogitClip, which involves clipping the logits to indirectly constrain the upper and lower bounds of the gradients. Specifically, the two aforementioned clipping-based methods rely on the validation set to determine a fixed optimal clipping threshold, overlooking the dynamic nature of gradients during the training process. However, as demonstrated by the simple binary classification experiment in Figure 1 , we observe distinct shifts in the gradient distribution throughout training, which significantly affect the suitability of a fixed threshold. Considering the fact that employing a higher threshold can enhance the model’s fitting ability, yet fails to bound the decreasing noisy gradients; conversely, employing a strict threshold ensures the noisy gradients remain bounded but compromises the model’s fitting ability, we expect a dynamically adjustable threshold for gradient clipping that continuously adapts in response to the shifts in the gradient distribution. Motivated by this, in this paper, we propose an optimize-based strategy, Optimized Gradient Clipping (OGC), which ensures that the threshold adjusts itself as the distribution of gradients evolves during training, maintaining an optimal balance between fitting ability and noise suppression. Specifically, OGC 1) utilizes a 2-component Gaussian Mixture Model (2-GMM) to model the distribution of cross-entropy losses for clean and noisy samples, 2) and estimates the ratio of the noisy gradients to the clean gradients after clipping. This allows us to determine a clipping threshold at each training step that restricts the gradient ratio to a predefined limit, effectively bounding the influence of noisy gradients. Theoretically, we demonstrate that the dynamic threshold obtained by OGC can effectively enhance the noise-tolerant ability of CE under either symmetric, asymmetric, or instance-dependent noise, which is originally a non-robust loss function. Furthermore, we conduct extensive experiments on various types of label noise, including, symmetric, asymmetric, instance-dependent, and real-world noise, to verify the effectiveness of our proposed method. Our key contributions are summarized as follows: • We first recognize the crucial role that gradient distribution shifts play in noisy label learning and propose a simple yet effective strategy to obtain a dynamic threshold that adapts to these continual shifts. • We provide a theoretical analysis demonstrating that the threshold obtained by OGC can make non-robust loss functions robust against various types of label noise. • We conduct extensive experiments to evaluate the effectiveness of our proposed method across multiple types of label noise, including symmetric, asymmetric, instance-dependent, and real-world noise. 2 Related Works Robust loss functions. In recent years, several robust-loss-based methods have been developed for learning with noisy labels. Ghosh, Kumar, and Sastry (2017) theoretically demonstrated the robustness of symmetric loss functions, such as Mean Absolute Error (MAE), to label noise. Building on this, Zhang and Sabuncu (2018) introduced the Generalized Cross Entropy (GCE), which is a generalization of Cross Entropy (CE) and MAE. Further, Wang et al. (2019) combined CE with scaled MAE to propose Symmetric Cross Entropy (SCE). Additionally, Menon et al. (2020) introduced a composite loss-based gradient clipping applied to CE, resulting in PHuber-CE. Feng et al. (2020) utilized the Taylor series to derive an alternative representation of CE, leading to the development of Taylor-CE. Moreover, Ma et al. (2020) proposed the Active Passive Loss (APL), aiming to create fully robust loss functions. Zhou et al. (2021) developed Asymmetric Loss Functions (ALFs) to address limitations posed by symmetric conditions. Wei et al. (2023) introduced logit clipping (LogitClip), a technique that clamps the norm of the logit vector to a constant upper bound. Recently, Ye et al. (2023) proposed Normalized Negative Loss Functions (NNLFs), which prioritize learning from well-understood samples. Other methods for noisy label learning. In addition to robust loss functions, various