Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction

Aditya Sarkar, Jiacheng Cheng, Shlok Mishra, Nuno Vasconcelos · 2026

Read Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction by Aditya Sarkar, Jiacheng Cheng, Shlok Mishra, Nuno Vasconcelos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22570.txt.

Introduction

Aditya Sarkar, Jiacheng Cheng, Shlok Mishra, and Nuno Vasconcelos let a predictor say no, backed by a memory of past data. Selective prediction gives a model a reject option for low-trust calls. Their 2026 preprint notes that past work sticks to closed tasks with fixed answer sets.

Their plug-and-play method adds memory to judge trust on open tasks too. A model that can abstain is safer in the wild, where odd inputs abound. The work targets settings like open visual questions. It aims for fewer bold, wrong answers. Knowing when to stay silent is a skill.

Catalog record

CREATORAditya Sarkar, Jiacheng Cheng, Shlok Mishra, Nuno Vasconcelos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22570

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction Aditya Sarkar, Jiacheng Cheng, Shlok Mishra, Nuno Vasconcelos Abstract Selective prediction aims to endow predictors with a reject option, to avoid low confidence predictions. However, existing literature has primarily focused on closed-set tasks, such as visual question answering with predefined options or fixed-category classification. This paper considers selective prediction for visual language foundation models, addressing a taxonomy of tasks ranging from closed to open set and from finite to unbounded vocabularies, as in image captioning. We seek training-free approaches of low-complexity, applicable to any foundation model and consider methods based on external vision-language model embeddings, like CLIP. This is denoted as Plug-and-Play Selective Prediction ( PaPSP ). We identify two key challenges: (1) instability of the visual-language representations , leading to high variance in image-text embeddings, and (2) poor calibration of similarity scores . To address these issues, we propose a memory augmented PaPSP ( MA-PaPSP ) model, which augments PaPSP with a retrieval dataset of image-text pairs. This is leveraged to reduce embedding variance by averaging retrieved nearest-neighbor pairs and is complemented by the use of contrastive normalization to improve score calibration. Through extensive experiments on multiple datasets, we show that MA-PaPSP outperforms PaPSP and other selective prediction baselines for selective captioning, image-text matching, and fine-grained classification. Code is publicly available at https://github.com/kingston-aditya/MA-PaPSP . 2 2 footnotetext: Corresponding author: Jiacheng Cheng. 1 Introduction The success of vision-language models (VLMs) enables a wide range of promising applications. However, these models are prone to erroneous predictions for reasons that include incorrect alignment of the two modalities, image or language ambiguities, or samples from the tails of their training distribution. This hampers their usefulness for real-world applications that require performance guarantees. As shown in Figure 1 , selective prediction (SP) ( 7 ; 6 ; 15 ; 61 ; 9 ; 17 ; 60 ; 52 ) methods address this problem by refusing to make low confidence predictions, with the goal of minimizing the prediction risk on the samples that they accept . Optimal selective predictors optimize the trade-off between risk and coverage , which is the acceptance ratio throughout a dataset. While there is a literature in SP, the problem has mostly been considered for closed-set prediction, usually with finite and small output vocabularies. Examples include classification with finite labels ( 60 ; 62 ) or visual question answering from a finite set of choices ( 57 ; 9 ) , e.g. binary questions, questions involving a finite set of attributes, etc. However, vision-language tasks like the selective captioning problem of Figure 1 require open-set prediction, where the label set is infinite. This is, in fact, the regime that most foundation models operate in. There is a need for SP methods for open set tasks. One possibility is to endow the foundation model with some prediction refusal score and train or fine-tune it to optimize the ensuing risk-coverage trade-off. However, most foundation models are large and require large-scale training to avoid overfitting ( 53 ; 22 ) . In many cases, the model may even be a proprietary black box that is impossible to retrain. In this work, we investigate an alternative solution, based on an external SP model that can be attached to any VLM, as illustrated in Figure 1 , to compute a confidence score for the predictions of the latter. SP can then be implemented by rejecting predictions of score below a threshold. This is denoted as Plug-and-Play Selective Prediction ( PaPSP ) and can be implemented by using a Large Visual Language Model (LVLM), e.g. Qwen ( 1 ) , InternVL ( 72 ) , as PaPSP model, in what is called the LLM-as-judge strategy ( 13 ; 65 ) . However, LVLMs are large and have computationally expensive inference, sometimes much heavier than the VLM which makes the original predictions. Furthermore, they are not calibrated for PaPSP and requires some type of training or adaptation, which can itself be expensive and induce overfitting. Ideally, the PaPSP model should be lightweight and flexible enough to support any task without training. Figure 1: PaPSP uses an external representation model and the CLIP score to enable selective prediction for VLM tasks like captioning without training. MA-PaPSP augments this model with an external dataset, which is leveraged to estimate proxy embeddings of greater stability and better calibrated contrastive scores. The figure shows an example where PaPSP fails but MA-PaPSP succeeds at rejecting an incorrect caption for the image shown. Also shown is the Cider-4 score between predicted and ground truth captions. In this work, we investigate the use of a visual language representation models (VLRMs), like CLIP ( 46 ) or SigLIP ( 67 ) , to implement PaPSP . This is denoted as the Selective Prediction-VLM (SP-VLM). A simple yet effective implementation, illustrated in the top branch of Figure 1 , is to project the image and text produced by the foundation model into the SP-VLM embedding and reject image-text pairs of low CLIP-score ( 25 ) . We find, however that SP-VLM embeddings also suffer from calibration shortcomings, which we hypothesize to have two main causes: 1) SP-VLM representation instability , which increases the variance of image and text embeddings, compromising the reliability of similarity scores, and 2) poor calibration of similarity score across the SP-VLM embedding. To overcome these, we propose the Memory Augmented PaPSP ( MA-PaPSP ) procedure of the lower branch of Figure 1 . The external SP-VLM is augmented with a retrieval dataset of image-text pairs. Methods that leverage this dataset are then proposed to 1) reduce the variance of the SP-VLM representation, by using the average representation of the nearest neighbors of the query image as a proxy embedding for the latter, and 2) enhance score calibration, by using contrastive scores that normalize the similarity score with respect to alternative text predictions. MA-PaPSP can be seen as a new instantiation of memory augmented approaches recently shown successful for various applications ( 41 ; 66 ; 20 ; 50 ) . Like these, it is a training-free method that instead leverages an external dataset to enhance SP-VLM functionality. MA-PaPSP has several interesting properties. First, it is a relatively lightweight solution, usually much lighter than an LVLM. Second, it requires no training of either foundation model or the SP-VLM. Third, it can implement SP over a taxonomy of foundation model tasks, including classification ( 70 ; 71 ) , image-text matching ( 39 ; 26 ) , and captioning, that range from closed to open set and from finite to unbounded prediction vocabularies. Fourth, it can be used with any foundation model. We show its effectiveness for models ranging from small VLRMs to large LVLMs. Finally, MA-PaPSP can be tailored to any application domain by simply collecting data from that domain. This usually leads to improved performance over PaPSP , as illustrated in Figure 1 . We show this for domains ranging from COCO captioning to medical imaging and that good performance can be achieved even with generic datasets, like CC12M ( 4 ) , not directly tied to the application, for both open-set and closed-set SP tasks. The contributions of this work are three-fold:

Read the complete text