Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA

David Smerkous, Qinxun Bai, Fuxin Li · 2024

Read Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA by David Smerkous, Qinxun Bai, Fuxin Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00259.txt.

Description

Abstract Particle-based Bayesian deep learning often requires a similarity metric to compare two networks. However, naive similarity metrics lack permutation invariance and are inappropriate for comparing networks. Centered Kernel Alignment (CKA) on feature kernels has been proposed to compare deep networks but has not been used as an optimization objective in Bayesian deep learning.

Catalog record

CREATORDavid Smerkous, Qinxun Bai, Fuxin Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA David Smerkous, Qinxun Bai, Fuxin Li Abstract Particle-based Bayesian deep learning often requires a similarity metric to compare two networks. However, naive similarity metrics lack permutation invariance and are inappropriate for comparing networks. Centered Kernel Alignment (CKA) on feature kernels has been proposed to compare deep networks but has not been used as an optimization objective in Bayesian deep learning. In this paper, we explore the use of CKA in Bayesian deep learning to generate diverse ensembles and hypernetworks that output a network posterior. Noting that CKA projects kernels onto a unit hypersphere and that directly optimizing the CKA objective leads to diminishing gradients when two networks are very similar. We propose adopting the approach of hyperspherical energy (HE) on top of CKA kernels to address this drawback and improve training stability. Additionally, by leveraging CKA-based feature kernels, we derive feature repulsive terms applied to synthetically generated outlier examples. Experiments on both diverse ensembles and hypernetworks show that our approach significantly outperforms baselines in terms of uncertainty quantification in both synthetic and realistic outlier detection tasks. 1 Introduction Figure 1 : Overview of feature repulsive loss construction: Starting with a batch of examples (left), optionally including synthetic outliers, ensemble features at each layer l are used to construct centered Gram matrices projected onto the unit hypersphere (middle). The hyperspherical energy is then calculated between models, weighted by layer, and incorporated into the loss function (right). Bayesian deep learning has always garnered substantial interest in the machine learning community. Instead of a point estimate which most deep learning algorithms obtain, a posterior distribution of trained models could significantly improve our understanding about prediction uncertainty and avoid overconfident predictions. Bayesian deep learning has potential applications in transfer learning, fairness, active learning, and even reinforcement learning, where reducing uncertainty can be used as a powerful intrinsic reward function ( Yang & Loog, 2016 ; Ratzlaff et al., 2020 ; Wang et al., 2023 ) . One line of approach to Bayesian deep learning is to add noise to a single trained model. Such noises can either be injected during the training process, e.g. as in the stochastic gradient Langevin dynamics ( Welling & Teh, 2011 ) , or after the training process ( Maddox et al., 2019 ) . However, many such approaches often underperform the simple ensemble method ( Lakshminarayanan et al., 2017b ) which merely trains several deep networks with different random seeds. Intuitively, an ensemble, because it starts from different random initializations, might be able to “explore" a larger portion of the parameter space than those that are always nearby one specific model or training path. Because of this, ensembles may capture different modes and therefore better represent the posterior distribution of “well-trained" network functions ( Fort et al., 2019 ; Wilson & Izmailov, 2020 ) . However, a critical question is, how different are the networks in an ensemble from one another? And can we utilize the idea of diversification to further improve these networks by making them even more diverse? In order to answer these questions, we first need a metric to compare those networks, which is in itself a significant problem; regular L1/L2 distances, either in the space of the network parameters, or in the space of the network activations, are not likely to work well. First, they suffer from the curse of dimensionality due to the excessive number of parameters in modern deep networks. Moreover, there is the peculiar permutation invariance , where one can randomly permute the different channels of each layer and result in a network that has vastly different parameters and activations, yet represents the same function. The popular RBF kernel lacks this permutation invariance inhibiting methods like Stein Variational Gradient Descent (SVGD) from working effectively on larger networks ( D' Angelo & Fortuin, 2021 ) . Therefore, a proper kernel for comparing network functions should address these critical issues by being effective in high-dimensional spaces and invariant to permutations of neural network channels. Kornblith et al. (2019) proposed an interesting approach for performing this comparison based on Centered Kernel Alignment (CKA). The idea is, instead of directly comparing activations or parameters, comparison is made between the Gram matrices of the same dataset fed into two different networks. Each example will generate a feature vector at each layer of the network, and a kernel matrix can be constructed based on the similarity between all example pairs in the dataset. Then, a CKA metric measures the similarity of these two Gram matrices as the similarity of the two networks. This idea addresses the permutation invariance issue and generates meaningful comparisons between deep networks. In this paper, we propose to explicitly promote diversity of network functions by adding CKA-based loss terms to deep ensemble learning. Given that CKA projects all kernels on a hypersphere, we further propose to use Hyperspherical Energy ( \mathrm{HE} ) minimization as an approach to more evenly distribute the ensemble of neural networks on the hypersphere. Experiments on synthetic data, MNIST, CIFAR, and TinyImageNet show that our approach maintains the predictive accuracy of ensemble models while boosting their performance in uncertainty estimation across both synthetic and realistic datasets. Besides, we demonstrate that our method can also be applied to training hypernetworks, improving the diversity and uncertainty estimation of the networks generated by a hypernetwork. Additionally, we propose using synthetic out-of-distribution (OOD) examples, to reduce their likelihood, and introducing feature repulsive terms on synthetic outlier examples to enhance OOD detection performance. We hope that our approach provides a different perspective to variational inference methods and contributes to improving uncertainty estimation in deep networks. Code is publicly available at https://github.com/Deep-Machine-Vision/he-cka-ensembles . 2 Related Work Uncertainty Estimation. A large body of literature has studied the problem of uncertainty estimation for neural networks. Bayesian neural networks ( Gal & Ghahramani, 2016 ; Krueger et al., 2017 ; Nazaret & Blei, 2022 ) approximate a posterior distribution over the model parameters and therefore estimate the epistemic uncertainty of the predictions. Non-Bayesian approaches, on the other hand, rely on bootstrap ( Osband et al., 2016 ) , ensemble ( Lakshminarayanan et al., 2017b ; Wen et al., 2020 ; Park & Kim, 2022 ) , and conformal prediction ( Bhatnagar et al., 2023 ) to generate multiple neural networks of the same structure. Our approach is most closely related to ensemble methods for estimating predictive uncertainty. We follow the common practice of evaluating uncertainty by distinguishing between inlier and outlier images for datasets like CIFAR/SVHN and MNIST/FMNIST. Most approaches typically evaluate the separation or distance within the feature space of a model between inliers and outliers ( Mukhoti et al., 2023 ; Van Amersfoort et al., 2020 ; D' Angelo & Fortuin, 2021 ; Ovadia et al., 2019b ; Lakshminarayanan et al., 2017a ; Liu et al., 2020 ) . Deep deterministic uncertainty (DDU) utilizes a single deterministic network with a Gaussian mixture model (GMM) fitted on the observed inlier features before the last layer and calculates separation of inliers and outliers using feature log density. We refer the reader to Mukhoti et al. (2023) for more information. For a more comprehensive survey and benchmarking

Read the complete text