Label Smoothing is a Pragmatic Information Bottleneck

Sota Kudo [email protected] Nara Institute of Science and Technology · 2025

Read Label Smoothing is a Pragmatic Information Bottleneck by Sota Kudo [email protected] Nara Institute of Science and Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14077.txt.

Introduction

Sota Kudo reframes label smoothing, a common training trick for classifiers, as a practical form of information bottleneck, in a research paper from 2025. Label smoothing softens the target answers during training, which often improves how models generalize. Why it works has stayed unclear.

Under set assumptions, the author shows in theory and tests that smoothed training lands on the same answer as the classic bottleneck method. That method keeps only the input traits that matter for the target. Smoothing also ignores irrelevant input traits, matching bottleneck behavior. The paper offers a simple lens on a widely used trick.

Catalog record

CREATORSota Kudo [email protected] Nara Institute of Science and Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Label Smoothing is a Pragmatic Information Bottleneck Sota Kudo [email protected] Nara Institute of Science and Technology Abstract This study revisits label smoothing via a form of information bottleneck. Under the assumption of sufficient model flexibility and no conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as a practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable. Figure 1: Empirical IB curve and solutions for cross entropy and label smoothing loss under practically reasonable assumptions. Here, X is the input, Y is the target, and T is the model output. Theoretical results are shown in Section 4.2 . 1 Introduction Label smoothing (LS) ( Szegedy et al. 2015 ) is an operation that softens a hard one-hot vector by taking a weighted average with another distribution, such as a uniform one. Despite its simplicity, it has been empirically shown to improve the performance of deep learning models and used to train SOTA models in various tasks such as vision ( Zoph et al. 2018 ; Huang et al. 2018 ; Real et al. 2019 ; Wortsman et al. 2022 ; Liu et al. 2022 ) , speech recognition ( Chorowski & Jaitly 2016 ) , machine translation ( Vaswani et al. 2017 ; NLLB Team et al. 2022 ) , and multi-modal models ( Yu et al. 2022 ) . The specific effects of label smoothing have become an active area of research. It has been pointed out that it reduces robustness against adversarial attacks ( Zantedeschi et al. 2017 ) , improves calibration ( Müller et al. 2019 ) , mitigates label noise ( Lukasik et al. 2020 ; Chen et al. 2020 ; Liu 2021 ) , does not allow for sparse output distributions ( Meister et al. 2020 ) , accelerates the convergence of stochastic gradient descent ( Xu et al. 2020 ) , degrades selective classification ( Xia et al. 2024 ) , and learns low-variance features ( Chidambaram & Ge 2024 ) . In addition to these points, label smoothing has also been shown to be a regularization limiting model outputs or representations. It can be seen as penalizing overconfident (i.e., low-entropy) predictions ( Pereyra et al. 2017 ; Meister et al. 2020 ) . Furthermore, it has been experimentally demonstrated that it compresses the internal representations of the model, such as the penultimate layer or the logits ( Müller et al. 2019 ) . These insights suggest a connection between label smoothing and information-theoretic regularization techniques for representations. Parallel to these developments, the machine learning community has also been exploring the Information Bottleneck (IB) framework ( Tishby et al. 2000 ) , a method for supervised representation learning based on information theory. The objective of IB is to compress the information about the input while preserving as much information as possible about the target labels. From the perspective of statistical learning theory, the compression is known to function as a form of regularization ( Shamir et al. 2010 ; Vera et al. 2018 ; Vera et al. 2020 ; Kawaguchi et al. 2023 ) . Furthermore, by compressing the input, the model is encouraged to ignore factors that do not contain information about the target ( Achille & Soatto 2017 ) , or factors that do not provide any additional information about the target when conditioned on another variable ( Ahuja et al. 2021 ) , which do not directly cause the target, thereby enabling the construction of more reliable models. In addition to these theoretical discussions, several practical benefits of applying IB methods to deep learning models have been reported, including robustness against adversarial attacks ( Alemi et al. 2016 ) , out-of-distribution detection ( Alemi et al. 2018 ; Pan et al. 2020 ) , domain generalization ( Ahuja et al. 2021 ; Li et al. 2022 ) , and calibration ( Alemi et al. 2018 ) . Considering these accumulated research findings on IB, studying the relationship between label smoothing and IB can significantly advance our understanding of the behavior of label smoothing or SOTA models that utilize it, and open up new directions for its application. Motivated by this insight, this work revisits label smoothing through the lens of the Information Bottleneck principle. Our contributions are threefold: • We clarify the correspondence, similarities, and differences between label smoothing and the Variational Information Bottleneck ( Alemi et al. 2016 ) (Section 4.1 ). Based on this correspondence, where the representation corresponds to the model output, we interpret label smoothing as a form of IB. Under the practically reasonable assumptions of model flexibility and the absence of multiple different labels for the same input, we theoretically show that label smoothing can explore the optimal solutions of IB with arbitrary compression levels within the range of interest (see Figure 1 ) (Section 4.2 ). This suggests that label smoothing enables simple implementations and avoids the practical issues associated with IB (Section 4.3 ) by limiting the scenarios where its strictness as an IB holds to typical cases. Thus, we argue that “label smoothing is a pragmatic information bottleneck.” • We experimentally validate the theoretical findings and show that, conversely, in situations where different labels exist for the same input, label smoothing is not necessarily optimal from the perspective of IB (Section 5.2 ). This demonstrates not only the effectiveness of label smoothing, but also its limitations as an IB method in certain situations. • We experimentally demonstrate the effectiveness of label smoothing as IB. In particular, we consider nuisance factors, which do not contain information about the target, and redundant factors, which do not provide additional information about the target when conditioned on another factor. We show experimentally that label smoothing makes the model less sensitive to these factors (Section 5.3 ). These results show cases where label smoothing enhances independence from factors that do not have a direct causal relationship with the target, which is an important property for building reliable models. 2 Related work 2.1 Similarities between Label Smoothing and the Information Bottleneck Several studies have presented fragmentary findings suggesting a potential connection between label smoothing and the information bottleneck. Müller et al. 2019 experimentally demonstrated that applying label smoothing reduces the mutual information between the input data and output logits, and mentioned its potential relationship to the information bottleneck. Furthermore, their visualization of penultimate layer representations showed that label smoothing causes data points to cluster around the centers of their respective classes. This behavior is consistent with the fact that the deterministic IB ( Strouse & Schwab 2016 ) favors hard clustering ( Kolchinsky et al. 2018 ) . Such intra-class information compression explains the degradation in transferability ( Kornblith et al. 2020 ) caused by label smoothing. Theoretically, connections between label smoothing and confidence penalty have also been pointed out

Read the complete text