Conformal Prediction and Human Decision Making

Jessica Hullman, Yifan Wu, Dawei Xie, Ziyang Guo, Andrew Gelman · 2025

Read Conformal Prediction and Human Decision Making by Jessica Hullman, Yifan Wu, Dawei Xie, Ziyang Guo, Andrew Gelman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11709.txt.

Introduction

Jessica Hullman and co-authors ask when prediction sets actually help people decide in a 2025 paper. Conformal prediction is a popular method that returns a set of plausible answers with a promised coverage rate, instead of one answer plus a confidence number. Such sets are touted for high-stakes fields like medicine and finance, yet their real value to a human chooser is unclear.

The authors build a decision theory frame that treats predictions as signals feeding a choice. Inside it, they compare ideal use of calibrated probabilities with ideal use of prediction sets. They then spell out strategies a person might follow with a set in hand. Several tensions surface: the coverage promise does not map neatly onto what decision makers need. The paper ends with research directions for uncertainty tools that serve people better.

Catalog record

CREATORJessica Hullman, Yifan Wu, Dawei Xie, Ziyang Guo, Andrew Gelman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conformal Prediction and Human Decision Making Jessica Hullman, Yifan Wu, Dawei Xie, Ziyang Guo, Andrew Gelman Abstract Methods to quantify uncertainty in predictions from arbitrary models are in demand in high-stakes domains like medicine and finance. Conformal prediction has emerged as a popular method for producing a set of predictions with specified average coverage, in place of a single prediction and confidence value. However, the value of conformal prediction sets to assist human decisions remains elusive due to the murky relationship between coverage guarantees and decision makers’ goals and strategies. How should we think about conformal prediction sets as a form of decision support? We outline a decision theoretic framework for evaluating predictive uncertainty as informative signals, then contrast what can be said within this framework about idealized use of calibrated probabilities versus conformal prediction sets. Informed by prior empirical results and theories of human decisions under uncertainty, we formalize a set of possible strategies by which a decision maker might use a prediction set. We identify ways in which conformal prediction sets and posthoc predictive uncertainty quantification more broadly are in tension with common goals and needs in human-AI decision making. We give recommendations for future research in predictive uncertainty quantification to support human decision makers. We thank John Cherian, Tiffany Ding, and Jason Hartline for feedback on a draft. 1 Quantifying uncertainty in black-box predictions What does it mean to target a quantification of prediction uncertainty to a decision maker’s task? Can we do this using general methods that work regardless of the specific decision setting? These statistical questions are as relevant as ever as machine learning models are deployed to assist human decision makers in domains like medicine, criminal justice, and finance. Methods for rigorously quantifying uncertainty in model predictions—even if the model is a black box—are in high demand. Consider for example a telehealth doctor who can access predictions from a computer vision model while evaluating images of a skin condition submitted by a prospective patient. Presenting information about the model’s confidence in its predictions is generally thought to help the human expert use the information more effectively. If, for example, the model assigns a moderate probability to a relatively rare but deadly condition, the doctor might advise them to schedule an appointment directly with a specialist rather than first seeing a primary care physician. Presenting uncertainty may also motivate “practico-epistemic” actions performed for the sake of epistemic gain, such as when the ambiguity about the true condition motivates the doctor to gather more evidence, as well as useful diversification in the choice of action ( Manski, 2009 ) , such that the doctor varies their advice across similar patients to balance expected false positives and false negatives in the face of ambiguity about the best treatment policy. A common minimum goal for such settings is that a predictive model is calibrated : the rate at which events predicted by the model are realized approximately matches the probability assigned by the model to those events Dawid (1985) ; Foster and Vohra (1998) . Here we focus on multiclass prediction, where we assume a classification model \hat{f} trained on historical data. When we apply the model to an input feature vector \textbf{x}=(x_{1},x_{2},...,x_{d})\in\mathcal{X}\subseteq\mathbb{R}^{d} , we obtain a predicted label \hat{f}(\textbf{x})=\hat{y}\in\mathcal{Y} , but also have access to a predicted probability \hat{p}_{y}(\textbf{x})=Pr(Y=y|\textbf{x}) for each possible label y\in\mathcal{Y} . Calibration in this setting means that for all predicted values b\in[0,1] , for all pairs of inputs and labels (X=\textbf{x},Y=y) for which \hat{p}_{y}(\textbf{x})=b , then (b\times 100)\% of the time, y is the true label. Calibration is useful when we expect decisions to be based only on predictions, because even if decision makers weigh errors differently than the machine learning model, calibrated predictions allow them to trade off their true risk according to their own decision problem. However, some machine learning paradigms result in predictive uncertainty estimates that are not calibrated. Deep learning models for multi-class classification, for example, produce pseudo-probabilities with predictions called softmax values. These are achieved by normalizing the activations produced by the last hidden layer in the neural network to a [0,1] range, but this normalization does not enforce calibration with reality or even internal consistency: these numbers when will not in general follow the laws of probability (that \mbox{Pr}(A\vee B)=\mbox{Pr}(A)+\mbox{Pr}(B) if the events A and B are disjoint, \mbox{Pr}(A\wedge B)=\mbox{Pr}(A)\times\mbox{Pr}(B) if they are independent). Beyond their lack of coherence, insufficient data, dataset biases, or implications of particular choices of architectures, hyperparameters, or model classes can make model confidence estimates vulnerable to overfitting, and thus unreliable for decision making ( Guo et al., 2017 ) . Bayesian methods like Bayesian neural networks ( Neal, 2012 ; Gal and Ghahramani, 2016 ) can provide more meaningful probabilities by approximating a distribution over network weights. However, their computational requirements in high-dimensional settings like images or text limit their use. In theory, empirically calibrated probabilities can be also achieved by training models with a proper loss function ( Blasiok et al., 2024 ) , or applying calibration procedures ( Hébert-Johnson et al., 2018 ; Guo et al., 2017 ; Gopalan et al., 2022 ) that learn adjustments to model-predicted probabilities to achieve calibration on held-out data. For example, recent work in theoretical computer science demonstrates the possibility of learning multicalibrated predictors–predictors that are simultaneously calibrated for all intersecting groupings of the data, not just marginally ( Hébert-Johnson et al., 2018 ) –by iteratively post-processing predictions via a boosting procedure. However, the amount of data required to achieve such guarantees renders them impractical in common scenarios like multiclass classification with a moderately large label space. Figure 1: Conventional (top-1) prediction (left) compared to split conformal prediction (right) using a nonconformity score based on softmax output. Enter conformal prediction, a class of distribution-agnostic approaches for quantifying uncertainty in predictions that produces sets of predictions that promise to contain the true label with a user-defined probability 1-\alpha . In contrast to returning a single top prediction with the highest model confidence score (Figure 1 left), split conformal prediction algorithms return a set of predictions designed to achieve 1-\alpha coverage on average (Figure 1 right). In the common variant of split conformal prediction with marginal coverage, this set is constructed by comparing a score, representing the “non-conformity” of the new instance with what the model has learned, to a threshold chosen to achieve the expected coverage over the randomness in the selection of test points and a set of held-out calibration instances for which ground truth is known. Conformal prediction has been argued to provide meaningful uncertainty quantification to improve decisions from predictions in scenarios where models are deployed to assist doctors or other experts ( Banerji et al., 2023 ; Kiyani et al., 2025 ; Lu et al., 2022 ; Straitouri et al., 2023 ) . Relative to a single top prediction with a confidence value, prediction sets may better convey to decision makers that multiple labels are consistent with the available information. By providing a finite set of possibilities with some probabilistic assurance, they can guide decision makers to focus their attention on a limited number of possibilities. For example, in the absence of a prediction set, which possibilities the telehealth doctor considers

Read the complete text