E-Values Expand the Scope of Conformal Prediction

Etienne Gauthier, Francis Bach, Michael I. Jordan · 2025

Read E-Values Expand the Scope of Conformal Prediction by Etienne Gauthier, Francis Bach, Michael I. Jordan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13050.txt.

Introduction

Etienne Gauthier, Francis Bach, and Michael Jordan show how e-values can widen the scope of conformal prediction, in a 2025 paper. Conformal prediction gives uncertainty sets with guarantees that need no assumptions on the data spread. The standard method ranks a new point's score against a calibration set. That rank method can be restated with p-values, the usual currency of evidence.

Their alternative builds on e-values instead. E-values allow tricks that p-values cannot match. The team present three uses: prediction that stays valid across batches at any stopping time, fixed-size sets whose coverage adapts to the data, and prediction when the ground truth itself is ambiguous. The examples show e-values flexibly expand the conformal toolbox for practice.

Catalog record

CREATOREtienne Gauthier, Francis Bach, Michael I. Jordan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

E-Values Expand the Scope of Conformal Prediction Etienne Gauthier, Francis Bach, Michael I. Jordan Abstract Conformal prediction is a powerful framework for distribution-free uncertainty quantification. The standard approach to conformal prediction relies on comparing the ranks of prediction scores: under exchangeability, the rank of a future test point cannot be too extreme relative to a calibration set. This rank-based method can be reformulated in terms of p-values. In this paper, we explore an alternative approach based on e-values, known as conformal e-prediction. E-values offer key advantages that cannot be achieved with p-values, enabling new theoretical and practical capabilities. In particular, we present three applications that leverage the unique strengths of e-values: batch anytime-valid conformal prediction, fixed-size conformal sets with data-dependent coverage, and conformal prediction under ambiguous ground truth. Overall, these examples demonstrate that e-value-based constructions provide a flexible expansion of the toolbox of conformal prediction. 1 Introduction Conformal prediction ( Vovk et al. 2005 ) is a widely used statistical framework that provides predictive models with valid uncertainty quantification. The core objective is to construct conformal sets that, given features X\in\mathcal{X} , contain the true target Y\in\mathcal{Y} with high probability. These guarantees are distribution-free, meaning they hold under any underlying data-generating distribution \mathbb{P} , provided the data are exchangeable. Consider a dataset of n exchangeable data points \{(X_{i},Y_{i})\}_{i=1,...,n} , where each pair (X_{i},Y_{i}) is drawn from some distribution \mathbb{P}=\mathbb{P}_{X}\otimes\mathbb{P}_{Y|X} over \mathcal{X}\times\mathcal{Y} . This dataset, referred to as the calibration set, is used to construct conformal sets. Let \alpha\in(0,1) be a given error level. Given a new pair (X_{n+1},Y_{n+1})\sim\mathbb{P} such that (X_{1},Y_{1}),...,(X_{n},Y_{n}),(X_{n+1},Y_{n+1}) remain exchangeable, the objective is to construct a conformal set \hat{C}_{n}(X_{n+1}) satisfying \mathbb{P}(Y_{n+1}\in\hat{C}_{n}(X_{n+1}))\geq 1-\alpha, (1) where the probability is taken over both the calibration set \{(X_{i},Y_{i})\}_{i=1,...,n} and the new data point (X_{n+1},Y_{n+1}) . In conformal prediction, uncertainty quantification relies on a score function S:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R} , which is typically derived from a pre-trained model f . The choice of S depends on the specific problem setting. In this work, we consider negatively-oriented scores, where smaller values indicate better performance, and assume these scores are positive. 1 1 1 The assumption that scores are negatively-oriented is not restrictive, as any positively-oriented score can be transformed, for example, via S_{\text{negative}}=1/(S_{\text{positive}}+\varepsilon) with \varepsilon preventing division by zero. Similarly, most scores used in conformal prediction are already nonnegative. We explicitly assume nonnegativity to simplify the construction of e-variables in conformal e-prediction, as in ( 4 ). An example of such a score is the cross-entropy loss used in classification tasks: S(X,Y)=-\log p_{f}(Y|X), (2) where p_{f}(Y|X) is the predicted probability assigned to the true class Y . In standard conformal prediction methods, the idea is to rank nonconformity scores S_{i}:=S(X_{i},Y_{i}) and compare their relative positions. This approach relies on the exchangeability of the scores, ensuring that the rank of a new test point is unlikely to be overly extreme compared to those in the calibration set. For completeness, we include an overview of this classical framework in Appendix A . Despite its theoretical appeal and simplicity, traditional conformal prediction faces significant limitations in practice. Some of these have to do with the lack of conditional guarantees, and there has been significant recent work aiming to address that limitation ( Jung et al. 2023 ; Gibbs et al. 2024 ) . In the current paper, we highlight three additional scenarios where conventional methods fall short: batch anytime-valid conformal prediction, fixed-size conformal sets with data-dependent coverage, and conformal prediction under ambiguous ground truth. As we emphasize in the next three subsections, these scenarios all reflect real-world problems where one would like to make use of conformal methods. While conventional conformal prediction based on p-values falls short of addressing these problems, we will show that they are all solved by using e-values to construct conformal sets instead of p-values, a methodology known as conformal e-prediction . Our overall suggestion is that conformal e-prediction provides a flexible tool relative to classical conformal prediction that enhances the applicability of conformal methods in the complex, diverse, and dynamical settings of modern machine learning. The code implementation for our experiments is available at https://github.com/GauthierE/evalues-expand-cp . 1.1 Batch anytime-valid conformal prediction In Section 2 , we consider the following problem: data arrive sequentially in batches b_{t} , t=1,2,\dotsc, with each batch containing n_{t}\geq 0 calibration data points S_{1}^{t},...,S_{n_{t}}^{t} and a test data point S_{n_{t}+1}^{t} . Importantly, the batches arrive sequentially, one after the other, and the number of batches is possibly unknown in advance. Between batches, the data distribution may shift, meaning the data from one batch is not identically distributed with the next. We simply assume that the data within each batch, including both calibration and test data points, are exchangeable, conditional on the previous data batches. Given \alpha\in(0,1) , the goal is to construct a sequence of batch anytime-valid conformal sets \hat{C}_{t} for all t\geq 1 , based on all past batches and the calibration data of batch t , satisfying: \mathbb{P}(\forall t\geq 1,\ S_{n_{t}+1}^{t}\in\hat{C}_{t})\geq 1-\alpha, (3) where the probability is taken over all data points. If ( 3 ) holds, we call \{\hat{C}_{t}\}_{t\geq 1} a sequence of batch anytime-valid conformal sets. As we now discuss, this problem arises in many real-life scenarios where data are received sequentially. 1.1.1 Motivating example 1: Pharmaceutical drug deployment across hospitals A real-world example of this problem arises in pharmaceutical drug deployment across hospitals. A company introduces a new drug, and a regulatory agency seeks statistical guarantees on its efficacy. The drug’s effect on patient i in hospital b_{t} is denoted by S_{i}^{t} , where each hospital corresponds to a data batch. After an initial calibration phase in hospital b_{t} , where the drug is tested on n_{t} volunteer patients, the goal is to construct a conformal prediction set for S_{n_{t}+1}^{t} , representing a future patient in that hospital. The deployment is inherently sequential: hospitals test the drug at different times, and results arrive progressively. The agency must provide anytime-valid statistical guarantees for all hospitals, constructing conformal prediction sets for future patients S_{n_{t}+1}^{t} across all sites, regardless of how many batches t have been received. Moreover, data distributions vary across hospitals due to differences in patient demographics, lifestyles, and medical histories. For instance, urban and rural hospitals may observe different drug efficacy patterns. The regulatory agency must account for these distributional shifts while leveraging prior batches to enhance predictive reliability. 1.1.2 Motivating example 2: Quality control in supply chains In a production supply chain, a company regularly receives shipments of parts from a supplier. Each batch contains q_{t} units, each with a quality rating S_{i}^{t} , assessed by the company. The batch size q_{t} varies based on demand. While unit quality within a batch is typically exchangeable, it may fluctuate across batches due to changes in the supplier’s production process (e.g., machinery wear) or transportation conditions. To manage this uncertainty, the company can apply batch anytime-valid conformal prediction to obtain quality guarantees. Specifically, for each batch, it selects n_{t}<q_{t} units to measure quality, forming a calibration set S_{1}^{t},...,S_{n_{t}}^{t} . Then, it predicts the quality of a randomly chosen product S_{n_{t}+1}^{t} from the remaining units. Since quality assessment is costly, n_{t} is typically small relative to q_{t} . The key advantage of this method is its

Read the complete text