Bayesian Self-Distillation for Image Classification
Anton Adelöw, Matteo Gamba & Atsuto Maki · 2025
Read Bayesian Self-Distillation for Image Classification by Anton Adelöw, Matteo Gamba & Atsuto Maki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24162.txt.
Introduction
Anton Adelöw, Matteo Gamba, and Atsuto Maki rethink training targets for image classifiers. Ordinary training forces one hard label per image. That habit breeds overconfidence. Calibration and robustness suffer for it. Self-distillation softens the targets using the model's own predictions. Yet it still leans on hard labels.
Their framework builds a tailored target distribution for each sample. Statistical inference over the model's predictions produces it. After initialization, no hard targets are needed at all. Benchmarks showed consistent accuracy gains. One large network gained two points on ImageNet. Calibration error fell sharply in other tests. A ResNet on CIFAR-100 cut its error by forty percent. With a contrastive loss added, robustness under label noise reached state-of-the-art levels.
Catalog record
| CREATOR | Anton Adelöw, Matteo Gamba & Atsuto Maki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bayesian Self-Distillation for Image Classification Anton Adelöw, Matteo Gamba & Atsuto Maki Abstract Supervised training of deep neural networks for classification typically relies on hard targets, which promote overconfidence and can limit calibration, generalization, and robustness. Self-distillation methods aim to mitigate this by leveraging inter-class and sample-specific information present in the model’s own predictions, but often remain dependent on hard targets, reducing their effectiveness. With this in mind, we propose Bayesian Self-Distillation (BSD), a principled method for constructing sample-specific target distributions via Bayesian inference using the model’s own predictions. Unlike existing approaches, BSD does not rely on hard targets after initialization. BSD consistently yields higher test accuracy (e.g. +1.4% for ResNet-50 on CIFAR-100) and significantly lower Expected Calibration Error (ECE) (-40% ResNet-50, CIFAR-100) than existing architecture-preserving self-distillation methods for a range of deep architectures and datasets. Additional benefits include improved robustness against data corruptions, perturbations, and label noise. When combined with a contrastive loss, BSD achieves state-of-the-art robustness under label noise for single-stage, single-network methods. The source code is available at https://github.com/antonadelow/BSD . 1 Introduction Despite the widespread use of deep neural networks for classification tasks, the implications of using hard targets for loss computation have received relatively little attention. Intuitively, classes can exhibit varying degrees of similarity, and individual samples may exhibit different levels of resemblance to both their assigned and other classes. This nuanced, sample-specific information – often referred to as dark knowledge ( 19 ) – is not taken into account when relying solely on hard targets. For instance, misclassifying an image of a dog as a teapot is fundamentally different from misclassifying it as a cat. Nevertheless, deep neural networks are typically trained with hard targets, which fail to capture inter-class relationships or sample-specific ambiguities, contributing to poor calibration ( 12 ) . Self-distillation , an efficient variant of knowledge distillation ( 18 ) , leverages dark knowledge by using a model’s own predictions as soft targets during training. In contrast, conventional knowledge distillation provides soft targets to a model by leveraging another, often larger, network. Existing self-distillation methods, however, exhibit varying limitations. Some modify the network architecture with intermediate classifiers ( 53 ; 52 ) , increasing parameter count and computational cost. Others impose a consistency loss between current predictions and those from the last epoch or mini-batch ( 21 ; 38 ) . Yet, epoch-wise construction introduces variance under augmentations, while targets from the last mini-batch may be of limited effectiveness, as they originate from nearly identical model states. Neural Network f(\mathbf{x}_{i}) Input Image Input \mathbf{x}_{i} Predicted Target \hat{\mathbf{y}}_{i}^{t} Prior Target p(\mathrm{z}_{i})=\text{Cat}\left(K,\frac{\bm{\alpha}_{i}}{\sum_{j}\bm{\alpha}_{ij}\}}\right) Posterior Target p(\mathrm{z}_{i}|\mathbf{x}_{i},\hat{\mathbf{y}}_{i}^{t})=\text{Cat}\left(K,\frac{\gamma\bm{\alpha}_{i}+\hat{\mathbf{y}}_{i}^{t}}{\sum_{j}\gamma\bm{\alpha}_{ij}+1}\right) \oplus \times\gamma\rightarrow Figure 1: Bayesian update of a target. At each epoch, the target distributions are updated using the model’s own predictions. Here, \oplus denotes the Bayesian update, and \times indicates discounting of the previous posterior parameters by \gamma\in[0,1] . See Section 3 for details. Fundamentally, existing self-distillation frameworks constrain their predictions by incorporating a loss term derived from the hard targets (e.g. 9 ; 21 ; 38 ). This reliance interferes with the goal of learning richer and more calibrated predictions, and leaves the model sensitive to label noise. In the overparameterized regime, sensititivity to label noise can exacerbate double descent ( 33 ) , a phenomenon where the test error initially decreases, then increases, before decreasing again as model capacity or training time increase. We hypothesize that a self-distillation method decoupled from the hard targets during training could provide smoother and more robust probability estimates, thereby mitigating these effects and providing more predictable training dynamics. To address these limitations, we propose Bayesian Self-Distillation (BSD), an efficient method for leveraging dark knowledge by constructing rich, sample-specific target distributions in a single training run. BSD treats a model’s own predictions as evidence for Bayesian inference, without explicitly relying on the hard targets after initialization. A diagram depicting BSD is included in Figure 1 . Our main contributions can be summarized as follows: • We propose Bayesian Self-Distillation (BSD), a lightweight and principled method for self-distillation that operates independently of hard targets after initialization. • We provide the first, to our knowledge, theoretical formalization of dark knowledge and use it to show that BSD captures rich sample-specific information. • We demonstrate through extensive experiments that BSD consistently outperforms both existing architecture-preserving self-distillation methods and conventional knowledge distillation in generalization and calibration for a variety of datasets and network architectures. • We show that BSD provides robustness against data corruptions, perturbations, and label noise. Notably, it mitigates epoch-wise double descent under label noise and achieves state-of-the-art performance for single-stage, single-network methods when combined with a contrastive loss. 2 Related work Self-Distillation (SD) originates from Knowledge Distillation (KD) ( 18 ) , a technique introduced for model compression using a teacher–student framework. Later work demonstrated that identical (Born-Again) networks can be trained sequentially with KD to improve generalization ( 9 ) , thereby giving rise to self-distillation. SD methods differ primarily in how they construct the teacher signal. One family of approaches relies on architectural structure, such as introducing auxiliary classifiers or branches to provide internal supervision ( 53 ; 52 ; 56 ) . Another category leverages temporal knowledge transfer, using model snapshots from earlier epochs ( 48 ) , moving averages of past predictions (Temporal Ensembling, TE) ( 24 ) , or moving averages of model parameters (mean teacher) ( 44 ) over epochs. A third set of methods leverages consistency, encouraging alignment between samples of the same class ( 50 ) or between consecutive predictions of the same sample ( 21 ; 38 ) . This category includes Progressive Self-Knowledge Distillation (PS-KD) ( 21 ) , which uses the model’s prediction from the last epoch as a soft target, and Self-Distillation from the Last mini-Batch (DLB) ( 38 ) , which constructs overlapping mini-batches and leverages the prediction from the previous mini-batch. Despite their differences, these strategies remain anchored to the original hard labels, which can be particularly problematic in settings where labels are noisy. Label noise remains an obstacle in deep learning, contributing to overfitting and the undesirable training dynamics. Proposed solutions range from regularization techniques such as label smoothing ( 42 ) and Mixup ( 51 ) , to robust loss functions like Symmetric Cross Entropy ( 45 ) . More sophisticated methods include sample selection via optimal transport 8 ; 3 , label correction by interpolation between the prediction and label ( 47 ) , or even complex multi-stage ( 27 ) or multi-network training pipelines ( 54 ) . These methods, however, often rely on heuristics rather than reformulating the learning objective to explicitly model uncertainty . In contrast, probabilistic modeling provides a principled way to estimate uncertainty and improves interpretability. Bayesian methods in deep learning have gained attention for their ability to model predictive uncertainty. Methods like Variational inference ( 2 ) , Monte Carlo dropout ( 10 ) , and deep ensembles ( 25 ) can be used to quantify uncertainty, improve calibration and increase robustness to out-of-distribution data. Recently, Bayesian optimization of hyperparameters has been paired with born-again networks to improve generalization over generations of networks. Related work has also shown that KD itself can serve as a mechanism to transfer better