A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates

Gonçalo Gomes, Bruno Martins, Chrysoula Zerva · 2025

Read A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates by Gonçalo Gomes, Bruno Martins, Chrysoula Zerva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01225.txt.

Introduction

Gonçalo Gomes, Bruno Martins, and Chrysoula Zerva present a 2025 method for grading image captions word by word, with honest doubt attached. The standard learned score, CLIPScore, gives one number for a whole caption, which hides which words are wrong and how sure the score is. The team wanted finer detail and a stated risk level.

Their method makes many versions of the score by masking parts of the input, then shapes the spread of scores with a risk control frame that fits any base model. The shape-up step pins the error rate to a level the user picks. In tests, this plain recipe matched far more complex ones at spotting wrong words. It also tied its doubt estimates to real errors more tightly, which makes the grades safer to trust.

Catalog record

CREATORGonçalo Gomes, Bruno Martins, Chrysoula Zerva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates Gonçalo Gomes, Bruno Martins, Chrysoula Zerva Abstract This study explores current limitations of learned image captioning evaluation metrics, specifically the lack of granular assessments for errors within captions, and the reliance on single-point quality estimates without considering uncertainty. To address the limitations, we propose a simple yet effective strategy for generating and calibrating distributions of CLIPScore values. Leveraging a model-agnostic conformal risk control framework, we calibrate CLIPScore values for task-specific control variables, tackling the aforementioned limitations. Experimental results demonstrate that using conformal risk control, over score distributions produced with simple methods such as input masking, can achieve competitive performance compared to more complex approaches. Our method effectively detects erroneous words, while providing formal guarantees aligned with desired risk levels. It also improves the correlation between uncertainty estimations and prediction errors, thus enhancing the overall reliability of caption evaluation metrics. 1 Introduction Image Captioning (IC) evaluation is a crucial task in vision-and-language research, aiming to assess how accurately textual descriptions represent visual contents. Reference-free metrics such as CLIPScore Hessel et al. 2021 ; Gomes et al. 2025 , which measure caption quality by computing the cosine similarity between image and text embeddings, have been shown to correlate strongly with human judgments. However, simply scoring captions is often insufficient, as these quality assessments can be hard to interpret or unreliable. In many cases, effective evaluation requires not only an overall score of caption quality, but also the detection of specific errors within the caption. Without this granular information, the assessment can seem incomplete or less useful. Beyond the lack of granularity, existing metrics provide IC quality assessments relying on single-point estimates, in the sense that a single value is used as an evaluation score, without incorporating any indication of confidence over the results. This absence of uncertainty quantification can give way to misleading scores, reducing user trust. To address these challenges, we propose a conformal risk control framework, aiming at task-specific calibrated predictions, in conjunction with a simple yet effective strategy for generating distributions over CLIPScore predictions. This provides us with a principled way to adapt IC evaluation both to fine-grained analysis for each caption, and to a broader view of performance over a dataset, allowing for user-defined criteria to determine risk. First, we improve interpretability by detecting foils, which are words in the caption that are not associated with the image. Second, we overcome the limitations of single-point evaluation by introducing well-calibrated intervals, providing a trustworthy measure of caption reliability. Experimental findings demonstrate that using conformal risk control, over the distributions produced with simple methods for expressing uncertainty, such as masking parts of the input, can achieve competitive performance on foil detection compared to more complex and specialized approaches. Conformal risk control can also provide improvements in the correlation between uncertainty estimations and prediction errors, enhancing the overall reliability of caption evaluation metrics. Furthermore, we emphasize that other existing state-of-the-art methods can also benefit from our conformal calibration framework, gaining formal guarantees over their results. The proposed methodology is model-agnostic, and our work underscores the adaptability and broad applicability of risk control, offering a compelling case for its integration into vision and language research. 2 Related Work This section reviews prior work relevant to our research, organized into three key areas: uncertainty quantification, conformal risk control, and image captioning evaluation. 2.1 Uncertainty Quantification Over-confident predictions are a widespread issue in machine learning models, leading to efforts to integrate uncertainty quantification techniques. Traditionally, uncertainty estimation in neural networks has used Bayesian approaches, where weights are treated as probability distributions instead of fixed values Mackay 1992 ; Welling and Teh 2011 ; Tran et al. 2019 . However, the high computational cost of these methods led to the adoption of alternative solutions, such as model variance approaches, to approximate uncertainty. One common method involves deep ensembles, creating multiple neural networks and calculating the empirical variance of their output as an uncertainty measure Garmash and Monz 2016 ; Kendall and Gal 2017 ; Pearce et al. 2020 ; Zhan et al. 2023 . Another popular variance method is Monte Carlo dropout Gal and Ghahramani 2016 ; Glushkova et al. 2021 , which leverages dropout regularization at test time, performing multiple stochastic forward passes and calculating the mean and variance of the outputs to approximate uncertainty. However, recent work by Ulmer and Cinà 2021 has shown that these variance-based methods can become unstable in out-of-distribution data, failing to produce accurate uncertainty estimates. 2.2 Conformal Risk Control Conformal prediction and risk control are powerful frameworks for calibrating predictive models to ensure reliable decision making, giving formal guarantees tailored to the specific risks of a given domain. As mentioned by Angelopoulos and Bates 2021 , conformal prediction provides a model-agnostic method for creating prediction sets that are guaranteed to contain the true outcome with a user-defined probability. This assurance is achieved by focusing on coverage as the risk function, ensuring that the prediction sets have the specified coverage level across various data distributions and models. Similarly, as discussed by Bates et al. 2021 , the risk control framework extends the concept of conformal prediction by incorporating broader notions of risk beyond coverage. This generalization allows the calibration of predictions to minimize specific, application-relevant risks, such as the cost of errors or misclassifications in high-stakes settings (i.e., the false positive rate). While both frameworks aim to improve predictive reliability, conformal prediction is primarily concerned with probabilistic validity, whereas risk control is the general framework that emphasizes aligning predictions with context-specific risk tolerances. 2.3 Image Captioning Evaluation Recently, image captioning evaluation shifted towards the use of reference-free metrics to assess captioning models. One of the pioneering metrics in this new approach is CLIPScore Hessel et al. 2021 , which evaluates captions without ground-truth references. Based on the Contrastive Language Image Pretraining (CLIP) model Radford et al. 2021 , CLIPScore calculates a modified cosine similarity between the image and caption representations under evaluation. This approach has shown a high correlation with human judgments, outperforming established reference-based metrics such as BLEU Papineni et al. 2002 and CIDEr Vedantam et al. 2015 . CLIPScore has become a widely adopted metric for image captioning evaluation, inspiring the development of numerous new learned CLIP-based evaluation metrics Sarto et al. 2023 ; Kim et al. 2022 ; Gomes et al. 2025 . However, scoring alone is insufficient for a comprehensive evaluation, which has led to recent studies focused on identifying foil words in captions. Shekhar et al. 2017 introduced the FOIL-it benchmark, featuring data with foil words by replacing nouns in MS-COCO Lin et al. 2014 captions with semantically similar alternatives. Building on this foundation, ALOHa Petryk et al. 2024 expanded the scope by addressing foil words involving a broader range of objects, particularly visual concepts under-represented in training data for captioning models Agrawal et al. 2019 . In terms of recent methods for detecting foil words, Rich-HF Liang et al. 2024 employs human-annotated datasets of mismatched keywords and implausible image regions, to train a multimodal language model capable of providing dense alignment feedback. In turn, Nam et al. 2024 introduced a novel approach for detecting foil words using

Read the complete text