Knowledge Distillation of Uncertainty using Deep Latent Factor Model

Sehyun Park, Jongjin Lee, Yunseop Shin, Ilsang Ohn, Yongdai Kim · 2025

Read Knowledge Distillation of Uncertainty using Deep Latent Factor Model by Sehyun Park, Jongjin Lee, Yunseop Shin, Ilsang Ohn, Yongdai Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19290.txt.

Introduction

Sehyun Park and four co-authors introduce Gaussian distillation, a way to compress a deep ensemble's uncertainty into a single student model. Ensembles give the most reliable uncertainty estimates, but they are heavy in compute and memory, which blocks use on small devices and in fast applications.

Ordinary distillation shrinks models yet tends to flatten the very variation that carries uncertainty. Gaussian distillation instead fits a distribution over the ensemble using a deep latent factor model, a special Gaussian process, then teaches the student that distribution. Preserving honest uncertainty in a compact model matters wherever a wrong, confident answer is costly.

Catalog record

CREATORSehyun Park, Jongjin Lee, Yunseop Shin, Ilsang Ohn, Yongdai Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE140 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowledge Distillation of Uncertainty using Deep Latent Factor Model Sehyun Park, Jongjin Lee, Yunseop Shin, Ilsang Ohn, Yongdai Kim Abstract Deep ensembles deliver state-of-the-art, reliable uncertainty quantification, but their heavy computational and memory requirements hinder their practical deployments to real applications such as on-device AI. Knowledge distillation compresses an ensemble into small student models, but existing techniques struggle to preserve uncertainty partly because reducing the size of DNNs typically results in variation reduction. To resolve this limitation, we introduce a new method of distribution distillation (i.e. compressing a teacher ensemble into a student distribution instead of a student ensemble) called Gaussian distillation, which estimates the distribution of a teacher ensemble through a special Gaussian process called the deep latent factor model (DLF) 1 1 1 The source code of DLF is publicly available at https://github.com/sehyun1094/DLF by treating each member of the teacher ensemble as a realization of a certain stochastic process. The mean and covariance functions in the DLF model are estimated stably by using the expectation-maximization (EM) algorithm. By using multiple benchmark datasets, we demonstrate that the proposed Gaussian distillation outperforms existing baselines. In addition, we illustrate that Gaussian distillation works well for fine-tuning of language models and distribution shift problems. 1 Introduction While DNNs have succeeded tremendously in various AI tasks, the rapid increase in their model sizes has raised a concern about high computational resource demands, which limits their applications to real world applications such as on-device AI ( sanh2019distilbert ) , and thus developers have increasingly compressed large-scale language models into much smaller models ( sanh2019distilbert ; gu2023minillm ; team2023gemini ; abdin2024phi ) . A representative tool for compression is knowledge distillation (KD), which constructs a smaller DNN that mimics a given large-scale DNN ( hinton2015distilling ) . Another concern is that the inherent over-parameterization of a single DNN makes them susceptible to overfitting, leading to overconfident predictions. When training predictive models, it is essential to learn models not only accurate but also reliable. For reliable prediction, proper quantification of uncertainty has become an important topic in AI research ( gal2016uncertainty ; malinin2018predictive ; mariet2020distilling ) . Deep ensemble (an ensemble of multiple DNNs) has received much attention not only for its strong predictive performance but also for its ability to quantify prediction uncertainty ( lakshminarayanan2017simple ; hendrycks2019benchmarking ) . An ensemble of DNNs can mitigate overconfident predictions by reflecting the uncertainty (i.e., the variation of multiple predictions made by members of an ensemble) when making a final decision. Since deep ensemble requires even more computational resources than DNNs, KD of deep ensemble is necessary for improving its applicability. Several works have focused on distilling a given teacher ensemble to a student ensemble instead of distilling a teacher ensemble to a single student DNN to keep the uncertainty as much as possible ( mariet2020distilling ; tran2020hydra ; wen2020batchensemble ; nam2021diversity ; nam2022improving ; zhang2023adaptive ) . Algorithms for KD of deep ensemble can be roughly categorized into two approaches: one-to-one distillation and distribution distillation. One-to-one distillation compresses each member in a teacher ensemble to a smaller DNN, which becomes a member of a student ensemble. Various weight-sharing architectures for student DNNs, along with their corresponding learning algorithms, have been proposed ( tran2020hydra ; nam2021diversity ; nam2022improving ; ferianc2022simple ) . On the other hand, distribution distillation treats each ensemble member in a teacher ensemble as an independent realization of a certain distribution whose parameters are modeled by a student DNN. ( malinin2019ensemble ) and ( ryabinin2021scaling ) assume that the conditional class probability vector of each member in a teacher ensemble follows a Dirichlet distribution and devise a method to estimate the parameters in the Dirichlet distribution using a student DNN. There are still limitations in existing KD methods for deep ensemble. One-to-one distillation methods tend to lose a significant amount of uncertainty in a teacher ensemble when they compress large DNNs into smaller ones, while performance of Dirichlet distillation (the distribution distillation with a Dirichlet distribution) is inferior to one-to-one distillation partly ( tran2020hydra ; nam2022improving ) because of instability in learning the parameters in the Dirichlet distribution. The aim of this paper is to propose a new distribution distillation method that is numerically stable in learning and superior to other baselines in uncertainty quantification. In our proposed method, we treat each member in a teacher ensemble as an independent realization of a Gaussian process and estimate the mean and covariance functions of the Gaussian process based on observed predictions of members in a teacher ensemble. For this purpose, we propose the deep latent factor (DLF) model where the mean and covariance functions are modeled by a student DNN and implement an EM algorithm to estimate the maximum likelihood estimator (MLE) of the student DNN. We call our method Gaussian distillation . Our contributions are summarized as follows. • We propose a new distribution distillation method based on a specially designed Gaussian process called the DLF model that achieves superior performance in uncertainty quantification to other baselines. • We develop an EM algorithm to estimate the student DNN in the DLF model. In particular, we propose a way of finding a good initial solution by maximizing the penalized complete log-likelihood. • We do numerical experiments to show that Gaussian distillation outperforms other baselines for both regression and classification. We also illustrate that Gaussian distribution is a useful tool for fine-tuning language models. • We apply the pre-trained DLF to distribution shift problems and show numerically that it outperforms baselines. 2 Preliminaries 2.1 Prediction uncertainty In a nutshell, quantifying prediction uncertainty in supervised tasks involves efficiently estimating the predictive distribution of the output y given a new input denoted as p(y|\boldsymbol{x}^{\text{n}ew}) . The variation in the predictive distribution can be used as a measure of uncertainty in prediction. A typical way of estimating the predictive distribution begins with a parametric generative model for the input and output pair. Let p(y|\boldsymbol{x},\theta) be the conditional distribution of the output y\in\mathcal{Y} given an input \boldsymbol{x}\in\mathcal{X}\subset\mathbb{R}^{d}, where \theta\in\Theta is an unknown parameter. Then, we try to estimate \theta based on training data (\boldsymbol{x}_{1},y_{1}),\ldots,(\boldsymbol{x}_{m},y_{m}) such that p(y|\boldsymbol{x},\hat{\theta}) is as close as possible to p^{*}(y|\boldsymbol{x}), where \hat{\theta} is an estimate of \theta and p^{*}(y|\boldsymbol{x}) is the true conditional distribution. For example, the MLE minimizes the empirical KL divergence between p(y|\boldsymbol{x},\theta) and p^{*}(y|\boldsymbol{x}). It is well known, however, that the variation in p(y|\boldsymbol{x},\hat{\theta}) is smaller than that in p^{*}(y|\boldsymbol{x}) because p(y|\boldsymbol{x},\hat{\theta}) does not take into account the uncertainty in estimating \hat{\theta} . Thus, making a decision solely with p(y|\boldsymbol{x},\hat{\theta}) would lead in overconfident results. A proper uncertainty quantification in prediction should consider not only uncertainty in p^{*}(y|\boldsymbol{x}) (aleatory) ( gal2016uncertainty ; malinin2018predictive ) but also uncertainty in \hat{\theta} (epistemic). A popular way of considering both aleatory and epistemic uncertainties in prediction is to use an ensemble. We construct multiple estimates \hat{\theta}_{1},\ldots,\hat{\theta}_{n} of \theta and then estimate the predictive distribution as \hat{p}(y|\boldsymbol{x})=\sum_{i=1}^{n}p(y|\boldsymbol{x},\hat{\theta}_{i})/n, which we call the averaged prediction model. For deep learning, the two most representative methods of constructing multiple estimates are deep ensemble ( lakshminarayanan2017simple ; dietterich2000ensemble ; laurent2022packed ) and Bayesian DNNs ( neal1996monte ; welling2011bayesian ; gal2016dropout ; wilson2020bayesian ; izmailov2021bayesian ; sharma2023bayesian ; kong2023masked ) . Deep ensemble generates multiple

Read the complete text