Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty

Jake Thomas, Jeremie Houssineau · 2024

Read Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty by Jake Thomas, Jeremie Houssineau (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08225.txt.

Introduction

Jake Thomas and Jeremie Houssineau offer a new way to handle uncertainty in active learning, where a model picks which data to learn from next. Their 2024 paper separates two kinds of uncertainty. One kind comes from gaps in knowledge and can shrink with more data. The other is built into the system and cannot. Telling them apart is hard, and no single picking rule always wins.

Their method joins probability with possibility theory, using the second to stand for knowledge gaps alone. From that mix they derive new picking rules with useful traits. To prove the idea, they build a new kind of Gaussian process model and test it on classification tasks with both made-up and real data, where it performs strongly.

Catalog record

CREATORJake Thomas, Jeremie Houssineau
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty Jake Thomas, Jeremie Houssineau Abstract A popular strategy for active learning is to specifically target a reduction in epistemic uncertainty, since aleatoric uncertainty is often considered as being intrinsic to the system of interest and therefore not reducible. Yet, distinguishing these two types of uncertainty remains challenging and there is no single strategy that consistently outperforms the others. We propose to use a particular combination of probability and possibility theories, with the aim of using the latter to specifically represent epistemic uncertainty, and we show how this combination leads to new active learning strategies that have desirable properties. In order to demonstrate the efficiency of these strategies in non-trivial settings, we introduce the notion of a possibilistic Gaussian process (GP) and consider GP-based multiclass and binary classification problems, for which the proposed methods display a strong performance for both simulated and real datasets. 1 Introduction Active learning is a supervised learning problem in which the learner chooses which data points to learn from, with the goal of minimising the number of data points required to obtain an efficient model. In the sequential problem, at each time step, the learner is presented with a set of unlabelled points. Using the previously observed data, the learner must determine the most informative unlabelled point to query. Active learning strategies often aim to reduce epistemic uncertainty (EU), which is due to a lack of knowledge, rather than aleatoric uncertainty (AU), which is a fundamental aspect of the system that cannot be minimised. Distinguishing between EU and AU is however a significant challenge with the standard approach where both are modelled with probability distributions. The difficulties arising when using probability theory to model EU are highlighted, e.g., by the paradoxes ( Dawid et al.,, 1973 ) appearing when using improper priors to model the absence of information, or by the fact that the standard notions of entropy and mutual information do not satisfy certain desirable properties ( Wimmer et al.,, 2023 ) . In this work, we propose to leverage another representation of uncertainty with the aim of better capturing EU. Alternatives to the standard inference frameworks have often been studied under different forms since Fisher’s own attempt in the early 20th century ( Fisher,, 1935 ) . Possibility theory is one such approach which was first studied in the 70s ( Zadeh,, 1978 ) . Possibility theory assumes various forms, and we consider one ( De Cooman,, 2001 ) that closely resembles probability theory to facilitate the reformulation of the statistical models on which we will apply active learning. We consider in particular the formalism introduced by Houssineau, (2018) as it allows for leveraging a possibilistic analogue of Bayesian inference which preserves many of the properties of the standard approach while enabling a mixed probabilistic/possibilistic modelling. In Bayesian active learning ( Houlsby et al.,, 2011 ; Rodrigues et al.,, 2014 ) , the inference is often carried out by using Gaussian processes (GPs) ( Williams and Rasmussen,, 2006 ) as they provide a versatile approach which yields a full posterior distribution of model uncertainty across input space, hence allowing for easy interpretability and use of the model. However, what is usually modelled by a GP is exactly the part that will be replaced by possibility functions in this work, and there is no possibilistic version of the notion of GP in the literature. We therefore introduce such a concept and show that it behaves similarly to the standard notion of GP. After discussing some related work in Section 2 , we outline relevant aspects of possibility theory in Section 3 and introduce two novel strategies for active learning in Section 4 , along with a discussion of their properties. In Section 5 , we propose possibilistic analogues of the concepts of precision, covariance, and GP, and prove that these definitions preserve many of the useful properties of their probabilistic equivalents. We also highlight the key differences from the probabilistic case, including important distinctions around the roles of covariance and precision functions. Equipped with all the required tools, we assess the performance of the proposed active learning strategies in Section 6 . Our contributions are as follows: 1. We propose two new acquisition functions for Bayesian active learning, including a new notion of epistemic uncertainty for which advantageous properties are highlighted 2. Based on new results on the behaviour of the covariance in possibility theory, we introduce a novel notion of possibilistic GP, which enables our acquisition functions to be implemented for non-trivial GP classification problems 3. We show that the proposed approach displays a strong performance on both simulated and real datasets, with at least one of our acquisition functions outperforming the baselines in each case. 2 Related Work This paper builds on existing work in information-theoretic active learning, focusing on varying measures of uncertainty over models and acquisition functions. The standard measure of uncertainty in the probabilistic context is the entropy over model parameters \theta given labeled data \mathcal{D} , as used in several approaches ( Sebastiani and Wynn,, 2000 ; MacKay,, 1992 ) . The corresponding greedy acquisition function selects the unlabelled input x that maximally decreases the expected entropy over the parameters, i.e., \arg\max_{x}H(\theta|\mathcal{D})-\mathbb{E}_{y\sim p(y|x,\mathcal{D})}\left[H(\theta|y,x,\mathcal{D})\right] . This requires knowledge of the posterior model parameters and is commonly rearranged to the BALD (Bayesian Active Learning by Disagreement) objective via the joint conditional mutual information ( Houlsby et al.,, 2011 ) , focusing on the difference in entropy in label space, \arg\max_{x}H(y|x,\mathcal{D})-\mathbb{E}_{\theta\sim p(\theta|x,\mathcal{D})}\left[H(y|x,\theta)\right] . Despite their ubiquity, the performance of these active learning methods is not always consistent, and neither of them clearly outperforms standard baselines. Recent studies have shown that possibility theory can be applied to complex uncertainty representation and inference problems. For instance, Houssineau and Bishop, (2018) introduced a possibilistic analogue of the Kalman filter, and Houssineau, (2021) defined a possibilistic version of spatial point processes. These results emphasise the feasibility of reformulating GPs and introducing new active learning strategies in the context of possibility theory. Related to this work, Hüllermeier and Waegeman, (2021) provide a comprehensive review of AU and EU in machine learning, supporting the motivation for our mixed possibilistic-probabilistic approach. Wimmer et al., (2023) define a list of desirable properties that should be satisfied by notions of total uncertainty, AU, and EU, and show that the usual entropy and mutual information do not satisfy some of these properties. This list of properties is extended by Sale et al., (2023) who propose a family of solutions based on distances between a given second-order probability distribution and instances of such distributions identified as having no EU, AU or total uncertainty. Denœux, (2023) explores reasoning with fuzzy and uncertain evidence using epistemic random fuzzy sets, which aligns with our approach of combining possibility and probability theories. Caprio et al., (2023) introduces a generalisation of Bayes’ theorem when both the likelihood and prior are only known to be contained within a given class of probability distributions, which could lead to generalisations of our approach where the likelihood is assumed to be known. Nguyen et al., (2019) ; Nguyen et al., (2022) propose EU sampling in active learning, using the likelihood ratio and

Read the complete text