Improving Group Fairness in Knowledge Distillation via Laplace Approximation of Early Exits
Edvin Fasth, Sagar Singh · 2025
Read Improving Group Fairness in Knowledge Distillation via Laplace Approximation of Early Exits by Edvin Fasth, Sagar Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01070.txt.
Introduction
Knowledge distillation trains a small student model to copy a large teacher, saving data and computing power. Yet students learn simple surface cues first. That can hurt groups whose labels line up with stray features in the input, lowering fairness even when overall accuracy looks fine. Fasth and Singh propose using Laplace approximation at the early exits of a network. Those exits are points where a prediction can be made before the final layer, and the method gives honest estimates of uncertainty there.
Those estimates are then used to reweight hard or unclear examples during training. The student gets more practice where it is most unsure, instead of relying on plain confidence margins as earlier work did. The authors plan benchmarks with a Bert-based model on the MultiNLI dataset to check whether the method raises group fairness.
Catalog record
| CREATOR | Edvin Fasth, Sagar Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Group Fairness in Knowledge Distillation via Laplace Approximation of Early Exits Edvin Fasth, Sagar Singh Abstract Knowledge distillation (KD) has become a powerful tool for training compact student models using larger, pretrained teacher models, often requiring less data and computational resources. Teacher models typically possess more layers and thus exhibit richer feature representations compared to their student counterparts. Furthermore, student models tend to learn simpler, surface-level features in their early layers. This discrepancy can increase errors in groups where labels spuriously correlate with specific input attributes, leading to a decline in group fairness even when overall accuracy remains comparable to the teacher. To mitigate these challenges, Early-Exit Neural Networks (EENNs), which enable predictions at multiple intermediate layers, have been employed. Confidence margins derived from these early exits have been utilized to reweight both cross-entropy and distillation losses on a per-instance basis. In this paper, we propose that leveraging Laplace approximation-based methods to obtain well-calibrated uncertainty estimates can also effectively reweight challenging instances and improve group fairness. We hypothesize that Laplace approximation offers a more robust identification of difficult or ambiguous instances compared to margin-based approaches. To validate our claims, we benchmark our approach using a Bert-based model on the MultiNLI dataset. 1 Introduction Knowledge Distillation (KD) Hinton et al. 2015 offers a sample-efficient approach for training a student model by leveraging a pretrained teacher model. The training loss for the student is a combination of the standard cross-entropy loss calculated on the ground-truth labels and a knowledge distillation (KD) loss that encourages the student’s predictions to align with the "soft targets" provided by the teacher. Let D={(x_{i},y_{i})}_{i=1}^{N} represent the training dataset, \mathcal{L}_{student}=(1-\alpha)\mathcal{L}_{CE}(f_{s}(x_{i}),y_{i})+\alpha\mathcal{L}_{KD}(f_{s}(x_{i};T),f_{t}(x_{i};T)) , with T being the temperature parameter usually kept high during training. This process effectively transfers the teacher’s generalization capabilities to the student. A significant advantage of KD is the flexibility to choose a student model with significantly fewer parameters than the often computationally intensive teacher, thereby facilitating easier deployment. One issue identified in the original knowledge distillation paper is the occurrence of bias during the training process. Since the student model learns from the teacher’s output probabilities, any biases present in the teacher’s predictions can be inherited—or even amplified—by the student. This becomes particularly problematic in settings where the teacher model has overfit to spurious correlations in the data or exhibits imbalanced performance across subgroups. As a result, the distillation process may unintentionally reinforce unfair or misleading patterns, highlighting the need for strategies that account for such biases during student training. Reweighting losses for spuriously correlated groups has been shown to improve group fairness metrics. However, this approach typically requires prior knowledge of the group during training and lacks adaptivity based on the student model’s current performance on challenging groups. To address these limitations, Tiwari et al. Tiwari et al. 2024 observed that student models often exhibit overconfidence on difficult instances in their early layers. They proposed utilizing Early-Exit Neural Networks (EENNs) to obtain uncertainty estimates from these intermediate layers and subsequently employing this information to dynamically reweight both the teacher’s distillation loss and the student’s cross-entropy loss for each training sample. Meronen et al. Meronen et al. 2024 proposed an alternative method for obtaining well-calibrated uncertainty estimates from EENNs. Their approach treats the final layer of the classification heads in early exit blocks as Bayesian parameters, employing Laplace approximations. This offers a computationally efficient way to estimate the uncertainty associated with making a prediction at an early exit. Furthermore, this method tends to yield high uncertainty for difficult instances, attributed to potential model misspecification. In this paper, we investigate whether this Laplace approximation-based uncertainty estimation in early exit layers can be effectively used to reweight challenging training instances. Section 1.1 outlines the work of Tiwari et al. Tiwari et al. 2024 as the primary motivation for our approach. Section 2 details the Laplace approximation method used for uncertainty estimation. The experimental setup and results are presented in Section 3 . Finally, Section 5 provides concluding remarks. 1.1 Related Works Mediating Bias Using Early Readouts Our work closely builds upon the approach by Tiwari et al. to mitigate featural bias in student models Tiwari et al. 2024 . To address the issue of bias propagation during knowledge distillation, Tiwari et al. introduced the DEDIER model. Their method adds the student model with an auxiliary network designed to estimate the confidence of early-layer predictions. This auxiliary network generates instance-level uncertainty estimates, which are then used to dynamically reweight the losses from training samples during distillation. They assign higher weights to instances with low uncertainty predictions and thus overconfident instances in the early exits, then multiply these weights with KD loss from the teacher and reduces the impact of potentially biased or misleading data points. DEDIER thus introduces a more debiased training process that maintains the student model’s efficiency and compactness while achieving improved performance on classification tasks. The overall loss was given by: \mathcal{L}_{student}=\sum_{D_{w}}(1-\lambda)\cdot l_{ce}\\ +\lambda\cdot\textbf{wt}\cdot l_{ke} where \bf{wt}=\exp^{\beta.\bf{cm}.\alpha} and \bf{cm(p)}=p_{max}-\max_{p_{k}\in\bf{p}-p_{max}}p_{k} . The parameters \beta and \alpha are hyperparameters. While DEDIER offers a more robust framework compared to standard knowledge distillation, it remains reliant on the accuracy of the auxiliary network’s confidence margins. This dependency has been identified as a potential limitation. Specifically, within subgroups of the dataset where classification performance is lowest, the prediction confidence margins tend to be higher than the average confidence across the dataset, as illustrated in Figure 1 . In the next section we’ll see how to use laplace approximation as another approach towards uncertainty estimation in early exits and how to use them for adjusting the loss function. Figure 1 : Confidence margin at different feature levels for the worst-performing group compared to other groups in the original DEDIER paper. 2 Laplace Approximation for uncertainty estimates Identifying the need for better prediction of confidence margins, Meronen et al. Meronen et al. 2024 targeted overconfidence and miscalibration in their intermediate classifiers. To do so, the paper proposed a probabilistic framework that enhanced the uncertainty quantification at each exit point. By applying last-layer Laplace approximations and model-internal ensembling, the model captured both aleatoric and epistemic uncertainties more accurately. This led to improved calibration and more trustworthy confidence scores, which in turn allowed for better decision-making under computational budget constraints. The proposed method demonstrated superior performance on many benchmark datasets (CIFAR-100, ImageNet, Caltech-256) across multiple metrics, including accuracy, negative log predictive density (NLPD), and expected calibration error (ECE), without retraining or significant computational overhead. The original DEDIER paper defines the confidence margin according to Eq. 2 p_{\text{max}}=\max_{p_{k}\in\mathbb{P}}p_{k} (1) \mathbf{cm}(\mathbf{p})=p_{\text{max}}-\max_{p_{k}\in\mathbb{P}\setminus\{p_{\text{max}}\}}p_{k} (2) This is used to re-weight the training samples for samples where the auxiliary network in the student model does not produce the correct label, according to Eq. 3 wt=\begin{cases}1,&\text{if }y^{(\text{aux})}=y\\ e^{\beta\cdot\mathrm{cm}(\mathbf{p}^{(\text{aux})})^{\alpha}},&\text{otherwise}\end{cases} (3) Instead of learning point estimates for the parameters in the auxiliary network, we apply a Laplace approximation over the final linear layer. Specifically, we approximate the posterior over the output weights of the auxiliary network using the Bayesian update rule from Eq. 4 : p(\theta\mid\mathcal{D}_{\text{train}})=\frac{p(\mathcal{D}_{\text{train}}\mid\theta)\,p(\theta)}{\int p(\mathcal{D}_{\text{train}}\mid\theta)\,p(\theta)\,d\theta}=\frac{\text{[likelihood]}\times\text{[prior]}}{\text{[model evidence]}} (4) In practice, we apply this approximation to the final weight matrix \hat{\mathbf{W}}_{\text{MAP}} of the auxiliary linear classifier. Let \hat{\boldsymbol{\phi}}_{i} denote the BERT feature vector (CLS token) for input \mathbf{x}_{i} . The predictive distribution over