Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition
Arthur Hoarau, Benjamin Quost, Sébastien Destercke, Willem Waegeman · 2025
Read Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition by Arthur Hoarau, Benjamin Quost, Sébastien Destercke, Willem Waegeman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18268.txt.
Introduction
Arthur Hoarau, Benjamin Quost, Sébastien Destercke, and Willem Waegeman present a 2025 paper on how smart systems should gather data when their guesses can be wrong in two different ways. One kind of error comes from noise in the data itself, and the other comes from having too little data. Most work treats the first kind as fixed, but the team argues that data from new sources, such as images or sound, can cut it.
Their plan lets a system choose both how many samples to take and which source to sample from. Taking more samples of one kind cuts the second kind of error, while adding a new source can cut the first. The authors test the idea on two data sets that mix several kinds of input, and they show the plan leads to clear, usable choices.
Catalog record
| CREATOR | Arthur Hoarau, Benjamin Quost, Sébastien Destercke, Willem Waegeman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition Arthur Hoarau, Benjamin Quost, Sébastien Destercke, Willem Waegeman Abstract To generate accurate and reliable predictions, modern AI systems need to combine data from multiple modalities, such as text, images, audio, spreadsheets, and time series. However, collecting training and test data for many modalities is challenging and time-consuming, creating a need for cost-efficient multi-modal data acquisition. In this paper we advocate that this can be realized by disentangling epistemic and aleatoric uncertainty. It is commonly assumed in the machine learning community that epistemic uncertainty can be reduced by collecting more data, while aleatoric uncertainty is irreducible. We claim that this assumption can be challenged in modern multi-modal AI systems, and we introduces an innovative data acquisition framework where uncertainty disentanglement leads to actionable decisions, allowing cost-efficient sampling in two directions: sample size and data modality. The main hypothesis is that aleatoric uncertainty decreases as the number of modalities increases, while epistemic uncertainty decreases by collecting more observations. We provide a theoretical analysis and proof-of-concept implementations on various multi-modal datasets to prove the usefulness of our framework, which combines ideas from active learning, active feature acquisition and uncertainty quantification. 1 Introduction Machine learning models are increasingly used to make decisions affecting people’s lives, such as in medical diagnosis, law enforcement, police investigation, insurance and loan approval, or environmental permit deliverance. Because incorrect decisions can have significant consequences, these systems must not only be accurate, but also provide a credible measure of their own uncertainty, warranting whenever model outputs should be trusted. Therefore, the problem of uncertainty quantification is currently the focus of many research works, a number of which are dedicated to distinguish between two specific types of uncertainty, known as aleatoric and epistemic ( Hora, 1996 ; Kendall and Gal, 2017 ; Hüllermeier and Waegeman, 2021 ; Bengs et al., 2023 ; Lakshminarayanan, 2024 ) . Aleatoric uncertainty is due to inherent randomness, and is generally assumed to be irreducible. Epistemic uncertainty arises from a lack of knowledge about the true data-generating distribution, and can be reduced by collecting more data. Typically, a machine learning model uses an estimate of the (unknown) true probability distribution of the data based on a training set. Epistemic uncertainty reflects the difference between this estimate and the actual distribution, and methods estimating this type of uncertainty usually construct a (second-order) distribution over probability vectors (see Section 2 for a short review of these methods). To generate accurate and reliable predictions, modern AI systems often need to combine data from multiple modalities, such as text, images, audio, spreadsheets, and time series. This requirement has led to the emergence of a research field known as multi-modal machine learning ( Baltrusaitis et al., 2018 ; Rudovic et al., 2019 ; Liang et al., 2024 ; He et al., 2024 ) , which aims to coherently integrate multiple modalities. This is typically achieved through multi-modal fusion strategies ( Wang et al., 2021 ; Zhao et al., 2024 ; Meng et al., 2020 ) . Such multi-modal acquisition introduces new opportunities and challenges for disentangling uncertainty. More specifically, the irreducibility assumption of aleatoric uncertainty can be challenged when information is obtained from different modalities. Future-proof general AI systems should be able to “search for evidence” in a cost-efficient manner to make complex decisions with high reliability. For example, in medical diagnostics, an AI assistant for patient examination could begin with simple low-cost procedures, such as blood or urine tests, and potentially follow up with more complicated (and expensive) investigations, such as MRI, CT, or genetics-based scans that uncover additional modalities. One then expects the aleatoric uncertainty about the diagnosis to decrease as more informative (yet more expensive) modalities are queried ( Acosta et al., 2022 ) . Similar AI assistants can also be useful in other application domains, such as police investigations, customs checks, and failure monitoring in industrial processes. Motivated by potential use cases in such domains, this paper introduces a novel data acquisition framework for multi-modal data by disentangling aleatoric and epistemic uncertainties, which we believe is crucial for multi-modal sampling problems. This original approach involves reducing both types of uncertainties through querying additional instances and modalities, which will be realized with a two-stage backtracking protocol that “searches for evidence”. Several data acquisition approaches, such as active learning ( Settles, 2009 ) and active feature acquisition ( Li and Oliva, 2020 ) , have demonstrated that iteratively gathering more knowledge can enhance performance in machine learning. However, knowledge acquisition (whether through new observations or additional modalities) can be costly, making an effective data collection strategy essential. Our new protocol combines active learning and active feature acquisition for multi-modal sampling. On a high level, this protocol proceeds as follows. For a test instance, we first assess by means of conformal prediction ( Sadinle et al., 2019 ) whether a new incoming sample should be rejected based on the uncertainty of the prediction, which is measured via the size of the conformal set. If the prediction is not rejected, our framework has statistical guarantees about the predictive performance. If the prediction is rejected, we assess in a next step whether the primary source of uncertainty is a lack of representative training data (high epistemic uncertainty) or a lack of representative features (higher aleatoric uncertainty). This uncertainty disentanglement leads to two possible actions: collect more labeled training data, or measure an additional modality for the test instance. The algorithm stops when it is sufficiently certain about the prediction, or when the two types of uncertainty cannot be further reduced. In the latter case, no prediction is made. As a practical illustration, let us consider the MIMIC-IV dataset ( Johnson et al., 2023 ) , which is a real medical dataset that will be more thoroughly explained in the experimental section. The dataset has three modalities that can be queried to obtain reliable predictions: the online medical record of the patient, classical lab tests (such as blood sample tests) and microbiological cultures. The proposed framework can then be used to predict the disease of a new patient in a cost-efficient manner, by examining as few modalities as possible for that patient. The paper is structured as follows. Section 2 presents related work on uncertainty disentanglement, conformal prediction, active learning and active feature acquisition. Section 3 formalizes the proposed protocol of active learning for multi-modal data acquisition, together with different methods that disentangle aleatoric and epistemic uncertainty. We also provide theoretical arguments that justify the usefulness of the protocol. Section 4 presents three experiments, on a simple dataset to illustrate the behavior of the method, on the BIOSCAN-5 dataset with multiple modalities (including images and DNA sequences) and on MIMIC-IV, a real world medical dataset. Finally, Section 5 concludes the article. 2 Background and related work Uncertainty disentanglement. To disentangle aleatoric and epistemic uncertainty in classification and regression settings, numerous methods have been proposed in recent years ( Huang et al., 2024 ) . In classification settings, most of these methods leverage the training phase to provide a prediction for a test instance in the form of a second-order distribution over first-order categorical distributions (as opposed to a classical supervised learning model, like a