ECUAS n \text{ECUAS}_{n} : A family of metrics for principled evaluation of uncertainty-augmented systems

Lautaro Estienne, Erik Ernst, Matías Vera, Pablo Piantanida, Luciana Ferrer · 2026

Read ECUAS n \text{ECUAS}_{n} : A family of metrics for principled evaluation of uncertainty-augmented systems by Lautaro Estienne, Erik Ernst, Matías Vera, Pablo Piantanida, Luciana Ferrer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20490.txt.

Introduction

Lautaro Estienne and four colleagues offer a new family of scores for systems that report both a prediction and their own doubt. Existing checks grade predictions and doubt separately, or fix one price for refusing to answer. That can mislead high-stakes choices, where costs differ by case. Their ECUAS scores, built as proper scoring rules, let the user tune how much bad predictions count against shaky doubt.

Catalog record

CREATORLautaro Estienne, Erik Ernst, Matías Vera, Pablo Piantanida, Luciana Ferrer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80290
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20490

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ECUAS n \text{ECUAS}_{n} : A family of metrics for principled evaluation of uncertainty-augmented systems Lautaro Estienne, Erik Ernst, Matías Vera, Pablo Piantanida, Luciana Ferrer Abstract In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users – human or downstream systems – to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems – i.e., systems that output both predictions and uncertainty scores – are currently being assessed in the literature in a variety of ways, using separate metrics to evaluate the predictions and the uncertainty scores, setting a cost function with a fixed rejection cost or integrating over a coverage-risk curve. We argue that these evaluation approaches are inadequate for assessing overall performance of the UA system for decision making under uncertainty and propose a novel family of metrics, ECUAS n \text{ECUAS}_{n} , formulated as proper scoring rules for the task of interest. The parameter n n controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case. We demonstrate the advantages of the ECUAS n \text{ECUAS}_{n} metrics both theoretically and empirically, through experiments on diverse classification and generation datasets, including a manually annotated subset of TriviaQA. 1 Introduction Modern machine learning systems are deployed across a wide range of settings, from automated decision-making pipelines to human-in-the-loop applications. In high-stakes scenarios, where incorrect outputs may have severe consequences, it is important for users to have access to information about the uncertainty with which a prediction is made, allowing them to accept or reject it depending on the error trade-off for their use case. As a result, a growing class of models now produce, alongside their candidate predictions, a measure of uncertainty or, a common counterpart, confidence. Properly evaluating these systems – which we shall call uncertainty-augmented (UA) systems – is therefore critical to ensure their usefulness, reliability, and safety. We use the term UA system to refer both to traditional classifiers with a reject option – sometimes called selective classifiers 18 – as well as modern generative systems that output an uncertainty or confidence score along with their generated answer 53 , 78 , 55 . The goal of this paper is to develop a principled way to comprehensively assess the quality of UA systems grounded in decision theory which, as a large body of literature has argued 32 , 73 , 67 , 70 , 16 , is the rational way to make decisions and assess performance under uncertainty. In current literature, performance of UA systems is often measured through two or more separate metrics 38 . The quality of the candidate answers is assessed with accuracy or error rate, while the quality of the uncertainties is measured using binary classification metrics, taking the uncertainty as a predictor of incorrect answers. These metrics include the area under the ROC curve (AUC) 15 , 92 , 91 , 19 , expected calibration error (ECE) 43 , 75 , 47 , 26 , Brier score 45 , 78 , 58 , 81 , and cross-entropy 52 , 60 , 62 , 40 . This evaluation strategy does not adequately reflect the value that the system provides to its user – a human or a downstream stage – who will use the uncertainties to decide whether to accept or reject the candidate answer. What is needed is a metric that reflects the quality of that final decision given by the candidate answer or a rejection. This can naturally be done based on the principles of decision theory. Perhaps the earliest work taking a decision theoretical approach to classification with a reject option was Chow’s 1957 paper on character recognition 11 . In that work, performance is measured through a cost function which includes costs for accepted incorrect answers depending on the ground truth, as well as a cost for rejection. Decisions are then made by minimizing the expectation of this cost over the predictive class distribution provided by a classifier. Later, El-Yaniv et al. 18 showed that the this cost function can be expressed as a combination of two values: the selective risk, defined as the expected misclassification cost of accepted predictions and the coverage, given by the probability that the prediction is accepted. Several other works followed Chow’s proposal, using or studying the cost-based approach for evaluation in selective classification, e.g., 27 , 24 , 9 , 2 . The cost-based approach to evaluation of UA system, while simple and principled, is often seen as problematic since it requires the specification of a rejection cost which may vary with the application or across different users or even instances. To solve this problem, Nadeem et al. 65 proposed to compute the area under the accuracy vs coverage curve, obtained by sweeping a threshold over the uncertainty and integrating over the resulting curve. This was later generalized to the area under the risk-coverage curve (AURC) 29 , a widely used metric in the literature of UA systems 28 , 42 , 5 , 10 , 97 , 84 , 66 , 82 , 83 , 96 , 90 , 93 , 14 , 98 , 25 , 23 , 8 , 86 , 7 , 79 , 48 , 1 , 87 . Subsequent works proposed variants of the AURC to address some of its shortcoming 95 , 80 . Crucially, the AURC and its variants do not assume that the uncertainties will be given a probabilistic interpretation. In particular, it does not penalize uncertainties that are under or overestimated, as long as the ranking is preserved. We see this as a major flaw, since uncertainties that do not faithfully reflect the probability of correctness are not easy to interpret. In this work, we construct evaluation metrics for UA systems that, like AURC, do not commit to a specific rejection cost but that, unlike the AURC, do reward uncertainties that are interpretable probabilistically. Our contributions can be summarized as follows: A principled replacement for the AURC grounded in decision theory. In this work, we define the expected cost for uncertainty augmented systems ( \text{ECUAS}_{n} ), a family of metrics to evaluate the quality of UA systems based on first principles from Bayes decision theory 72 . The \text{ECUAS}_{n} metrics are given by the expectation of a cost function which is constructed as an integral over a family of cost functions parameterized by the rejection cost. Crucially, these cost functions are proper scoring rules (PSR) 36 , 73 , 31 , a family of metrics proposed decades ago for the assessment of predictive distributions. As such, they reward systems that produce probabilistic uncertainties, a property that is missing from the widely-used AURC metric. The \text{ECUAS}_{n} metrics can be used for evaluation of UA classifiers and UA generative systems. Insight on optimality of uncertainty scores. We provide a theoretical explanation for the empirical observation reported in prior works that better uncertainty scores can be obtained for generative systems using the distribution over semantic equivalence classes rather than over individual predictions 20 , 50 . To our knowledge, this is a novel theoretical explanation for this empirical finding. Empirical validation We compare our proposed metrics with traditional metrics for this task on

Read the complete text