Test-time augmentation improves efficiency in conformal prediction
Divya Shanmugam, Helen Lu, Swami Sankaranarayanan, John Guttag · 2025
Read Test-time augmentation improves efficiency in conformal prediction by Divya Shanmugam, Helen Lu, Swami Sankaranarayanan, John Guttag (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22764.txt.
Introduction
Divya Shanmugam and co-authors show in a 2025 paper that test-time augmentation sharpens conformal prediction. Conformal classifiers output sets of possible labels with a coverage promise. Those sets are often uselessly large. Augmenting each test input with transformed copies adds useful bias.
Prediction sets shrank by about a tenth on average in their tests. No retraining is needed, and any conformal score can pair with the trick. Three datasets, three models, and shifted data all showed gains. Smaller honest sets make uncertainty tools practical. The method slots into existing conformal pipelines cheaply.
Catalog record
| CREATOR | Divya Shanmugam, Helen Lu, Swami Sankaranarayanan, John Guttag |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-time augmentation improves efficiency in conformal prediction Divya Shanmugam, Helen Lu, Swami Sankaranarayanan, John Guttag Abstract A conformal classifier produces a set of predicted classes and provides a probabilistic guarantee that the set includes the true class. Unfortunately, it is often the case that conformal classifiers produce uninformatively large sets. In this work, we show that test-time augmentation (TTA)–a technique that introduces inductive biases during inference–reduces the size of the sets produced by conformal classifiers. Our approach is flexible, computationally efficient, and effective. It can be combined with any conformal score, requires no model retraining, and reduces prediction set sizes by 10%-14% on average. We conduct an evaluation of the approach spanning three datasets, three models, two established conformal scoring methods, different guarantee strengths, and several distribution shifts to show when and why test-time augmentation is a useful addition to the conformal pipeline. 1 Introduction Conformal prediction has emerged as a promising way to equip existing classifiers with statistically valid uncertainty estimates. It does so by replacing the prediction of the most likely class with an uncertainty set –a set of classes accompanied by a probabilistic guarantee that the true class appears in the set [ 36 ] . Conformal prediction faces two limitations in practice. First, achieving a suitably strong guarantee often leads to prediction sets that are uninformatively large [ 10 , 47 ] . Large prediction sets have been shown to decrease performance when provided as a decision-making aid [ 3 , 47 , 10 ] . Second, conformal classifiers inherit the instability of the underlying models. As a result, prediction sets can change significantly in response to small input perturbations, a well-known weakness of neural networks [ 39 ] . Applying a horizontal flip to each image in ImageNet, for example, changes the prediction set sizes for 75% of examples at a coverage guarantee of 99%. Such behavior also represents a barrier to broader use. In this work, we show that test-time augmentation (TTA), a widely-used technique in computer vision, has the potential to address both limitations. TTA involves generating an ensemble of predictions by perturbing the input with label-preserving transformations. It has previously been shown that TTA can be used to make non-conformal classifiers more robust to small input perturbations [ 7 ] , more accurate [ 37 ] , and better calibrated [ 18 ] . However, previous work has not explored the utility of TTA in the context of conformal prediction. We propose test-time-augmented conformal prediction , which transforms a classifier’s predictions using a learned test-time augmentation policy prior to conformal classification. By using distinct sets of labeled data to learn the test-time augmentation policy and the conformal classifier, we preserve the assumption of exchangeability, and thereby the coverage guarantee associated with the conformal predictor. In experiments testing the performance of conformal predictors subject to distribution shift, we see that test-time augmentation reduces prediction set sizes by 14% on average, with no loss of coverage. And even when there is no distribution shift, we see a reduction of 10% on average. Moreover, we find that classes with the largest average prediction set sizes benefit most from the introduction of test-time augmentation. We also show that test-time augmentation can bridge gaps between classifiers of different sizes (e.g., we show that test-time augmentation combined with ResNet-50 produces smaller set sizes than ResNet-101 without test-time augmentation). Our analysis of why test-time augmentation reduces prediction set sizes reveals a previously unknown effect of test-time augmentation. Specifically, TTA increases the predicted probability of the true class even when it is predicted to be unlikely (for example, promoting the true class from 200th most likely to 100th most likely). Although such behavior has no impact on the maximum predicted probability — commonly the focus of literature on test-time augmentation [ 2 , 37 , 32 ] — it is valuable in conformal prediction. This is because when the true class is promoted to a higher rank among a classifier’s predicted probabilities, the conformal classifier includes fewer incorrect classes to meet the conformal guarantee. Figure 1 : We illustrate the addition of test-time augmentation to conformal calibration in green (left) and provide a snapshot of the improvements it can confer (right). We show results on Imagenet, with a desired coverage of 95%, for the 20 classes with the largest predicted set sizes on average (computed over 10 calibration/test splits). Contributions We make three contributions. To begin, this is the first work to propose combining test-time augmentation, a popular technique in computer vision, with conformal prediction. Second, we present a method that reduces the prediction set sizes of existing conformal predictors by using automatically learned test-time augmentations. Finally, we demonstrate, in an extensive set of experiments, that our approach to combining conformal prediction and test-time augmentation leads to smaller prediction sets. 2 Related work Conformal prediction was first introduced by Gammerman et al. [16] , and further developed by Saunders and Holloway [35] and Vladmir Vovk [42] . We review efforts to ensemble conformal predictors and efforts to reduce prediction set sizes below. Ensembles in conformal prediction Several methods that generate ensembles of conformal predictors are known to improve efficiency. These methods include cross-conformal prediction [ 43 ] , bootstrap conformal prediction [ 44 ] , aggregated conformal prediction [ 5 , 25 ] , and out-of-bag conformal prediction [ 26 ] . The approaches primarily differ in how data is sampled to create the training dataset for the classifier and the calibration dataset for the conformal predictor. However, all require training multiple base classifiers or conformal predictors. Our approach is distinct: we propose a technique to generate an ensemble from a single model by perturbing the input, which requires no additional base models and no additional conformal predictors. Efficiency in conformal prediction There are two ways to improve efficiency in split conformal prediction: adjustments to the conformal score or improvements to the underlying classifier. Many works have proposed new procedures to estimate and apply thresholds on conformal scores [ 40 , 4 , 1 , 33 , 13 ] . Romano et al. [34] proposed APS, a conformal score based on the cumulative probability required to include the correct class in a prediction set. Angelopoulos et al. [1] built on this work to propose RAPS, which modifies APS by penalizing the inclusion of low-probability classes. Comparatively little work has focused on improvements to the underlying model. Jensen et al. [20] ensemble a set of base classifiers, where the classifiers are created by training models on subsets of the training data. Stutz et al. [38] provide a new way to train the base classifier and conformal predictor jointly through a novel conformal training loss. In contrast, our work focuses on improving the underlying model without retraining, and can be easily combined with any of the above procedures. Test-Time Augmentation Test-time augmentation (TTA) is a popular technique to improve the accuracy, robustness, and calibration of an existing classifier by aggregating predictions over a set of input transformations [ 37 , 32 , 48 , 15 , 2 , 9 , 18 ] . TTA has been applied to a diverse range of predictive tasks across domains ranging from healthcare [ 8 ]