Empirical Bayes Conformal Prediction for Vision and Language Models

Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh · 2026

Read Empirical Bayes Conformal Prediction for Vision and Language Models by Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23189.txt.

Introduction

Zeng and four coauthors present Bayes conformal prediction for vision and language. Conformal prediction gives free coverage but ranks from one unstable score, while average variants smooth scores. Both discard inconsistency that can flag stable picks. Their method uses r-values to turn score spread into an uncertainty score, estimating top-group odds from mean and spread. It offers closed-form and sampling estimators. R-values keep target coverage while cutting noisy false picks. Tests on classification, CLIP, and language show kept coverage and smaller sets.

Catalog record

CREATORJiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110050
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23189

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Empirical Bayes Conformal Prediction for Vision and Language Models Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh Abstract Conformal prediction (CP) gives distribution-free coverage for modern vision and language models, but it is often forced to make a ranking decision from a single unstable nonconformity score. Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate. Both options discard the inconsistency that can help identify whether a candidate is indeed stable. A weak answer can enter the conformal set even if the evidence is not strong, simply because one posterior sample or prompt phrasing made it look strong. But variability can help distinguish a stable signal from noise-driven fluctuations. We describe an empirical Bayes conformal prediction framework that uses r r -values to convert score variability into an uncertainty informed nonconformity score. The resulting r r -value estimates how likely a candidate’s latent score belongs to the top-ranked group after accounting for both its mean score and its uncertainty. It admits both a closed-form Normal-Normal empirical Bayes estimator and a nonparametric posterior-sampling estimator. Using the r r -value as the nonconformity score preserves the target conformal coverage while provably reducing the inclusion of high variance false candidates under mild regularity conditions. Across image classification, CLIP-based VLM benchmarks, and LLMs, we show that r r -value conformal prediction preserves target coverage while improving ranking stability and reducing set size when variability is informative, and reverting to CP-like behavior when variability vanishes. 1 Introduction Large language models (LLMs) and vision models such as Vision Transformers (ViTs) [ 8 , 31 , 18 ] are increasingly being used to inform decision making, e.g., in clinical deployments and finance where quantifying uncertainty and mitigating risk are essential. Bayesian neural networks (BNNs) [ 15 ] , MC-dropout [ 9 ] and Deep Ensembles (DE) [ 17 ] are important ideas in linking trustworthiness and model variability. But applying these methods directly to large pre-trained models can be expensive [ 1 ] . Conformal prediction ( {\rm CP} ) [ 29 ] provides an alternative framework where instead of estimating uncertainty inside the model, we wrap the model’s scores with a calibrated prediction set. Rather than attaching uncertainty to a single prediction, CP returns a set guaranteed to contain the true output with user-specified probability under exchangeability [ 4 ] . These properties have made CP a promising tool for both vision and language models [ 30 , 27 ] . The ranking problem in Conformal Prediction. Despite its distribution-free coverage guarantee, {\rm CP} is not designed to directly use epistemic uncertainty from model parameters or training randomness. Standard CP relies on a single model output/score. The guarantee controls whether the true label is included, but the usefulness of the set relies on how well this score orders true candidates ahead of false ones. If several candidate labels receive similar scores, small fluctuations can change their order and therefore change the conformal set. This ranking instability becomes more pronounced under model or data heterogeneity, where score variability is no longer negligible. Existing results [ 26 ] show that {\rm CP} can struggle under statistical heterogeneity, resulting in unreliable coverage. Fig. 1 shows an example. For the same input image, different model instances sampled from the same posterior distribution of the model can produce different conformal prediction sets. The takeaway is that score variability should not simply be ignored. Instead, it can provide useful information for constructing more stable and efficient conformal rankings. Figure 1 : Posterior sampled models yield different CP sets for the same image, revealing epistemic instability. This instability is not just due to posterior model sampling. In vision models, it may arise from posterior uncertainty over model parameters; in VLMs and LLMs, it may arise from prompt paraphrasing or model based evaluation of candidate responses. A high score can mean two very different things: stable evidence for a candidate or a fluctuation of an otherwise weak one (which is when high-variance candidates enter the conformal set). {\rm CP} sees only a single noisy realization, while {\rm CP}_{\text{avg}} averages realizations before calibration and smoothing this variability. Instead of eliminating variability, our goal is to check what it tells us about the reliability of a candidate’s rank. The contribution of this work is to utilize what is considered a weakness of large models, namely their inconsistency [ 6 , 22 ] , as a signal for conformal efficiency. We introduce an empirical Bayes conformal prediction framework ( {\rm CP}_{r\text{-value}} ) based on the r -value [ 11 ] by measuring how likely a candidate’s latent score belongs to the top-ranked group after accounting for both its estimated score and its uncertainty. Theoretically, under exchangeability, using the r -value as the nonconformity score preserves the target conformal coverage, while it reduces the inclusion probability of high variance false candidates and leads to smaller expected prediction sets. We estimate this quantity either with a Normal–Normal empirical Bayes model, which gives a closed-form expression and theoretical insight, or with an assumption-free posterior-sampling estimator. The nonparametric version is a Monte Carlo estimator of the same posterior tail probability defining the theoretical r -value. Across image classification, VLM, and LLM tasks, {\rm CP}_{r\text{-value}} behaves similarly to {\rm CP} or {\rm CP}_{\text{avg}} when variability is small, but produces smaller and more stable conformal sets when the variability is informative. 2 Preliminaries We briefly review conformal prediction and then show, with examples, why relying on a single pre-trained model, while neglecting model uncertainty, can impact {\rm CP} reliability. Terminology/Notations. Consider a dataset \{(x_{i},y_{i})\}_{i=1}^{n} , where x_{i} is an input such as an image, a question, or a prompt, and y_{i} is the corresponding label or response. For a given input x , let \mathcal{U}(x)=\{u_{1},\ldots,u_{K}\} denote the candidate outputs, such as class labels in image classification or candidate responses in language tasks. The model, or an external evaluator, assigns each candidate u_{j} a numerical score f(x)_{j} . For image classification, f(x)_{j} can be the logit or probability of class j ; for LLM tasks, it can be a likelihood or a quality score assigned by another model. We use the term "candidate" throughout for readability, which corresponds to the “unit” terminology in [ 11 ] . 2.1 Conformal Prediction: Mechanism and Coverage {\rm CP} constructs a prediction set by comparing test scores to a calibration threshold. Once a nonconformity score is fixed, {\rm CP} provides a distribution free coverage guarantee under exchangeability. Here, the size of the prediction set depends on how this score ranks candidate outputs. Setup. Let \{(x_{i},y_{i})\}_{i=1}^{n} be a calibration set. For each calibration example, we compute a nonconformity score S_{i}=S(x_{i},y_{i}) for the true output. For classification, one choice is S_{i}=1-\mathrm{softmax}\bigl(f(x_{i})\bigr)_{y_{i}} , where smaller scores indicate more conforming labels. For a significance level \alpha , CP sets the threshold B to be the \lceil(n+1)(1-\alpha)\rceil -th smallest calibration score. For a test input x_{\mathrm{test}} , we compute S(x_{\mathrm{test}},u_{j}) for every candidate u_{j}\in\mathcal{U}(x_{\mathrm{test}}) and include candidates whose scores are below the calibration threshold: \mathcal{C}(x_{\mathrm{test}})=\{u_{j}:S(x_{\mathrm{test}},u_{j})<B\}. In classification, the set contains distinct labels (possible responses/answer options for language tasks). Coverage. Under exchangeability between calibration and test data, the rank of the true test score among the

Read the complete text