CAOS: Conformal Aggregation of One-Shot Predictors

Maja Waldron · 2026

Read CAOS: Conformal Aggregation of One-Shot Predictors by Maja Waldron (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05219.txt.

Introduction

One labelled example is sometimes all a new task offers, and Maja Waldron's 2026 paper adds trustworthy uncertainty estimates to that setting. One-shot prediction adapts a pretrained foundation model using a single example, but it lacks principled uncertainty quantification. Conformal prediction gives finite-sample coverage guarantees, yet standard split conformal methods waste scarce data on splitting and lean on a single predictor, both costly when labels are precious.

Waldron proposes CAOS, which blends several one-shot predictors, weighting them adaptively, and calibrates with a leave-one-out scheme that uses every labelled point. Honest uncertainty from one example matters in fields like medicine and science, where a new category may have exactly one confirmed case. A bare prediction is not enough to act on.

Catalog record

CREATORMaja Waldron
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAOS: Conformal Aggregation of One-Shot Predictors Maja Waldron Abstract One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS) , a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage. 1 Introduction Large pretrained foundation models enable rapid adaptation to new tasks using only a handful of labeled examples, without task-specific fine-tuning ( Brown et al. 2020 ; Bommasani 2021 ; Radford et al. 2021 ; Siméoni et al. 2025 ) . In both vision and language, such one-shot and few-shot prediction paradigms are particularly attractive in settings where labeled data are scarce or expensive to obtain. In these settings, each labeled example can serve as a demonstration that induces a task-specific predictor for new inputs. When multiple labeled examples are available, this naturally gives rise to a collection of one-shot predictors rather than a single fixed model. The quality and relevance of these predictors may vary substantially across test inputs, making it unclear which prediction to trust and how to quantify uncertainty in a principled way. A motivating example arises in medical facial analysis, where clinicians annotate a small number of images with task-specific facial landmarks for pre- and post-operative assessment. Modern vision foundation models can transfer landmark information from a single annotated image to a new image via patch similarity ( Siméoni et al. 2025 ) , yielding a collection of one-shot predictors whose usefulness varies across patients and poses. Beyond point predictions, practical deployment in such settings requires reliable uncertainty quantification. Conformal prediction provides a principled framework for constructing prediction sets with finite-sample marginal coverage guarantees ( Shafer & Vovk 2008 ; Angelopoulos et al. 2024 ) . Applied naively, a common approach is to equip each one-shot predictor with its own conformal prediction set using split conformal calibration. However, data splitting is statistically inefficient in the low-data regimes that motivate one-shot learning ( Bashari et al. 2025 ) , and selecting or aggregating predictors adaptively typically violates the assumptions required for conformal validity ( Liang et al. 2024 ) . These challenges motivate the following question: Can we aggregate the predictions of all one-shot predictors induced by the available labeled data in a manner that adapts to each test input, uses labeled data efficiently for calibration, and still yields valid conformal prediction sets? In this work, we answer this question in the affirmative by introducing Conformal Aggregation of One-Shot Predictors (CAOS) , a conformal framework for one-shot prediction. At a high level, CAOS adaptively aggregates nonconformity scores across multiple one-shot predictors while allowing all labeled examples to participate in calibration through a leave-one-out construction. Despite breaking classical exchangeability at the score level, CAOS achieves valid finite-sample marginal coverage guarantees. Our contributions are threefold: (i) we introduce CAOS , a data-efficient conformal framework for aggregating one-shot predictors without requiring disjoint calibration splits; (ii) we establish finite-sample marginal coverage guarantees for CAOS via a novel monotonicity-based reduction to full conformal prediction; and (iii) we demonstrate empirically that CAOS yields substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage in both vision and language one-shot tasks. 2 Related Work In-Context Learning (ICL) , first popularized for language models ( Brown et al. 2020 ) and closely related to zero-shot and few-shot prediction paradigms ( Bommasani 2021 ) , is now a standard mechanism for adapting large pretrained models in both language and vision ( Radford et al. 2021 ; Siméoni et al. 2025 ) . In parallel, conformal prediction provides a distribution-free framework for uncertainty quantification with finite-sample marginal coverage guarantees ( Shafer & Vovk 2008 ; Angelopoulos et al. 2023 ; Angelopoulos et al. 2024 ) , and is particularly appealing in foundation-model settings because it can treat the underlying predictor as a black box. A growing body of work applies conformal prediction to foundation models in zero-shot, one-shot, and few-shot regimes ( Fisch et al. 2021 ; Quach et al. 2023 ; Su et al. 2024 ; Wang et al. 2024 ; Fillioux et al. 2024 ) ; see also ( Park et al. 2023 ; Vishwakarma et al. 2024 ; Pantelidis et al. 2025 ; Ye & Wen 2025 ; Lin et al. 2025 ; Tayebati et al. 2025 ; Silva-Rodríguez et al. 2025a ; Silva-Rodríguez et al. 2025b ; Wang et al. 2025 ; Xu & Lu 2025 ) . Many of these methods adopt split conformal calibration for simplicity and modularity. In the one-shot setting, each labeled example induces its own predictor, yielding a collection of predictors rather than a single model. Existing conformal approaches typically do not address uncertainty quantification for collections of one-shot predictors whose relevance may vary across test inputs. The one-shot, low-data regime also sharpens a classical trade-off in conformal prediction between statistical and computational efficiency. Split conformal is computationally efficient but can be statistically inefficient when labeled data are scarce ( Bashari et al. 2025 ) , while full conformal can be more statistically efficient but is often computationally prohibitive. Data-reuse schemes such as cross-conformal prediction ( Vovk 2015 ) and related leave-one-out approaches ( Barber et al. 2021 ) improve data efficiency, but their validity analyses frequently require additional arguments and may introduce slack when the resulting conformity scores are not exchangeable ( Gasparin & Ramdas 2025 ) . These considerations motivate methods that reuse labeled data while retaining sharp finite-sample guarantees. When confronted with a pool of predictors, a natural strategy is to either select a single predictor ( Liang et al. 2024 ; Bai & Jin 2024 ; Hegazy et al. 2025 ) or aggregate predictors or their uncertainty sets ( Gasparin & Ramdas 2024b ; Rivera et al. 2024 ; Alami et al. 2025 ) . However, many selection- and aggregation-based conformal approaches preserve validity by relying on disjoint data splits, extra calibration data, or conservative corrections. These limitations trace to a central requirement in classical conformal prediction: standard finite-sample guarantees rely on exchangeability of the calibration and test conformity scores ( Shafer & Vovk 2008 ; Angelopoulos et al. 2023 ; Angelopoulos et al. 2024 ) . When exchangeability is violated at the data level, for example under distribution shift ( Tibshirani et al. 2019 ; Barber et al. 2023 ) or in online prediction ( Gasparin & Ramdas 2024a ; Sale & Ramdas 2025 ) , validity is often recovered via additional assumptions or correction factors that can loosen guarantees ( Barber et al. 2023 ) . In the one-shot setting studied here, non-exchangeability instead arises algorithmically , because instance-adaptive selection or aggregation over a pool of reference-conditioned predictors breaks exchangeability at the score level, preventing a direct application

Read the complete text