Credal Prediction based on Relative Likelihood
Timo Löhr ∗, Paul Hofman ∗, Felix Mohr, Eyke Hüllermeier · 2025
Read Credal Prediction based on Relative Likelihood by Timo Löhr ∗, Paul Hofman ∗, Felix Mohr, Eyke Hüllermeier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22332.txt.
Introduction
Timo Löhr, Paul Hofman, Felix Mohr, and Eyke Hüllermeier propose, in a 2025 paper, a grounded way to predict with sets of probability distributions instead of one. A credal set openly shows what a learner does not know. The question is which distributions belong in the set.
Their answer uses relative likelihood: keep every model whose fit stays above a chosen share of the best model's fit. The cutoff has a plain meaning and trades correctness against precision in a controlled way. Modified ensemble methods then approximate the set in practice. On benchmark data the approach represented uncertainty better than rivals without losing accuracy. The authors argue honest sets beat false precision when stakes ride on a forecast.
Catalog record
| CREATOR | Timo Löhr ∗, Paul Hofman ∗, Felix Mohr, Eyke Hüllermeier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Credal Prediction based on Relative Likelihood Timo Löhr ∗, Paul Hofman ∗, Felix Mohr, Eyke Hüllermeier Abstract Predictions in the form of sets of probability distributions, so-called credal sets, provide a suitable means to represent a learner’s epistemic uncertainty. In this paper, we propose a theoretically grounded approach to credal prediction based on the statistical notion of relative likelihood: The target of prediction is the set of all (conditional) probability distributions produced by the collection of plausible models, namely those models whose relative likelihood exceeds a specified threshold. This threshold has an intuitive interpretation and allows for controlling the trade-off between correctness and precision of credal predictions. We tackle the problem of approximating credal sets defined in this way by means of suitably modified ensemble learning techniques. To validate our approach, we illustrate its effectiveness by experiments on benchmark datasets demonstrating superior uncertainty representation without compromising predictive performance. We also compare our method against several state-of-the-art baselines in credal prediction. 1 Introduction The distinction between two types of uncertainty, referred to as aleatoric and epistemic, is receiving increasing interest in machine learning ( Hüllermeier and Waegeman, 2021 ) . Roughly speaking, the aleatoric uncertainty of a predictive model is caused by the inherent randomness of the data-generating process, whereas epistemic uncertainty is caused by the learner’s lack of knowledge about the true (or best) predictive model. While aleatoric uncertainty is irreducible, epistemic uncertainty can be reduced on the basis of additional information, e.g., by collecting more training data. * * * ∗ equal contribution Aleatoric uncertainty can be captured adequately in terms of (conditional) probability distributions, whereas the representation of epistemic uncertainty requires “second-order” formalisms more general than (single) probability distributions. In the Bayesian approach to machine learning, second-order probability distributions are used for this purpose ( Depeweg et al., 2018 ; Kendall and Gal, 2017 ) : The learner maintains a probability distribution over the model space, which, if each model is a probabilistic predictor, results in predictions in the form of probability distributions of probability distributions (on outcomes). The Bayesian approach is theoretically appealing but computationally demanding, and commonly criticized for the need to specify a prior distribution (strongly influencing prediction and uncertainty quantification) ( Gawlikowski et al., 2023 ) . Figure 1 : Overview of our method . Language inference example from the ChaosNLI dataset, showing a premise-hypothesis pair with three possible labels: Entailed , Neutral and Contradicted . Middle: Illustration of the learning mechanism of our Credal Relative Likelihood (CreRL) framework, showing ensemble member training up to different \alpha thresholds. Right: Unlike baselines, which are centered around the MLE and remain fixed, our method allows to adapt credal set size to improve coverage of the ground-truth distribution. An alternative second-order formalism is offered by credal sets ( Walley, 1991 ) , i.e., sets (instead of distributions) of probability distributions that are commonly assumed to be closed and convex ( Cozman, 2000 ) . Although a set appears to provide weaker information than a distribution, a set-based representation also has advantages. In particular, it can be argued that sets are more apt at representing ignorance in the sense of a lack of knowledge ( Dubois et al., 1996 ) , because distributions always involve additional assumptions beyond the mere distinction between plausible and implausible candidate models. Decision-making on the basis of credal sets has been proposed as a reasonable alternative to Bayesian decision-making ( Levi, 1978 ; Girón and Ríos, 1980 ) . The learning of credal predictors (i.e., models making predictions in the form of credal sets) has been considered in machine learning in the past ( Zaffalon, 2001 ; Corani and Zaffalon, 2008 ) and, in light of the quest for proper representation of epistemic uncertainty, received renewed interest more recently ( Wang et al., 2024a ; Nguyen et al., 2025 ; Cella and Martin, 2024 ) . A major question to be addressed in this regard concerns the distinction between plausible and implausible models. In this paper, we adopt a theoretically grounded approach for constructing credal sets based on the notion of relative likelihood. Our approach explicitly defines the prediction target as the set of all probability distributions generated by plausible learners, namely those whose relative likelihood exceeds a specified threshold (cf. Section 3 ). That said, realizing this in complex machine learning settings comes with several practical challenges, notably the questions of representation, approximation, and inference: How to formally represent subsets of a high-dimensional model space, how to approximate such sets algorithmically, and how to infer credal predictions from them? We propose methods to address these questions based on suitably modified ensemble techniques (cf. Section 4 ). We highlight the effectiveness of our approach on real-world datasets, evaluating its performance in terms of coverage and efficiency as key criteria of credal prediction. Also, we analyze its performance on downstream tasks such as Out-of-Distribution (OoD) detection (cf. Section 5 ). Contributions. In summary our contributions are as follows: 1. We develop a theoretically sound approach to learning credal predictors, which is grounded in the statistical notion of relative likelihood. 2. We cast the learning task as optimizing multi-objective generalization performance, namely as finding a compromise between coverage and efficiency of credal predictions. 3. We propose an adaptive and conceptually intuitive ensemble-based method for approximating sets of plausible models and the induced credal predictions. 4. We empirically compare our method to representative baselines and, for the first time, compare state-of-the-art credal predictors by coverage and efficiency. Our approach achieves superior uncertainty representation and strong OoD performance. Related Work. In machine learning, uncertainty is often represented by (approximate) Bayesian methods ( Mackay, 1992 ; Blundell et al., 2015 ; Lakshminarayanan et al., 2017 ; Gal and Ghahramani, 2016 ; Daxberger et al., 2021 ) . An important characteristic of such representations, especially for uncertainty tasks, is diversity ( D’Angelo and Fortuin, 2021 ; Wood et al., 2023 ) , which can be enforced by regularization ( de Mathelin et al., 2023 ) , varying hyper-parameters ( Wenzel et al., 2020 ) , or increasing diversity in representations ( Lopes et al., 2022 ) . Alternatively, credal sets have been used in the fields of imprecise probability and machine learning to represent model uncertainty ( Zaffalon, 2001 ; Corani and Zaffalon, 2008 ; Corani and Mignatti, 2015 ) . Such sets can be generated based on the relative likelihood, also referred to as normalized likelihood ( Antonucci et al., 2012a ) and have been used in machine learning with simple model classes ( Senge et al., 2014 ; Cella and Martin, 2024 ) . In this work, we build upon these approaches and address the challenges that emerge when adapting the relative likelihood to a setting with complex predictors. Recently, credal sets have also been applied in the context of deep learning. Some approaches use ensemble learning to derive class-wise lower and upper probabilities ( Wang et al., 2024b ; Nguyen et al., 2025 ) . Others train models to directly predict probability intervals ( Wang et al., 2024a ) . Hybrid methods, combining multiple uncertainty frameworks, have also been proposed. Caprio et al. (2023) combine Bayesian deep learning and credal sets by considering sets of priors.