VCRScore: Image captioning metric based on V&L Transformers, CLIP, and precision-recall

Guillermo Ruiz, Tania Ramírez, Daniela Moctezuma · 2025

Read VCRScore: Image captioning metric based on V&L Transformers, CLIP, and precision-recall by Guillermo Ruiz, Tania Ramírez, Daniela Moctezuma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09155.txt.

Introduction

Guillermo Ruiz and co-authors propose a new score for image captions, built on vision language models. Their paper from 2025 notes that old scores like BLEU and ROUGE still rule caption judging, though models have leapt ahead. Judging tools that lag the models can misread real gains.

A human labeled data set was built first, scoring how well captions match image content. Their VCRScore blends a vision language transformer, the CLIP model, and precision recall counts. Against human scores it beat the classic and newer rivals tried. The authors give the score and the labeled set as tools for fairer caption model checks.

Catalog record

CREATORGuillermo Ruiz, Tania Ramírez, Daniela Moctezuma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09155

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VCRScore: Image captioning metric based on V&L Transformers, CLIP, and precision-recall Guillermo Ruiz, Tania Ramírez, Daniela Moctezuma Abstract Image captioning has become an essential Vision & Language research task. It is about predicting the most accurate caption given a specific image or video. The research community has achieved impressive results by continuously proposing new models and approaches to improve the overall model’s performance. Nevertheless, despite increasing proposals, the performance metrics used to measure their advances have remained practically untouched through the years. A probe of that, nowadays metrics like BLEU, METEOR, CIDEr, and ROUGE are still very used, aside from more sophisticated metrics such as BertScore and ClipScore. Hence, it is essential to adjust how are measure the advances, limitations, and scopes of the new image captioning proposals, as well as to adapt new metrics to these new advanced image captioning approaches. This work proposes a new evaluation metric for the image captioning problem. To do that, first, it was generated a human-labeled dataset to assess to which degree the captions correlate with the image’s content. Taking these human scores as ground truth, we propose a new metric, and compare it with several well-known metrics, from classical to newer ones. Outperformed results were also found, and interesting insights were presented and discussed. 1 Introduction Nowadays, the Vision & language research area is continuously growing, with tasks such as Image Captioning, Visual Question-Answering, and Image generation, among others. This kind of task involves text and images, which turn it into a very challenging problem. Particularly, the Image Captioning (IC) task has been studied recently, and it can be very useful in many diverse applications, such as content retrieval, intelligent video surveillance systems applications, computer-human interfaces, etc. State-of-the-art proposals to deal with the IC task are rapidly updating, they went from being based on traditional approaches, based on probabilistic models, to the current Vision Transformers. Nevertheless, evaluation metrics have been almost untouched through the years. Proof of this even today the metrics based on n -grams matching are still very popular in the IC Task [ 28 , 8 , 41 , 35 ] . Moreover, there are more lexical-based metrics such as METEOR, ROUGE, SPICE, and CIDEr. Despite that new semantic-based metrics have been proposed in the last few years, such as BertScore, BLEURT, and CLIP, there is much space for improvement yet [ 12 ] . This improvement is related to doing better metrics to evaluate the Image Captioning task. We know that the best way to do that is human-based, manually tagging the system’s output, but this is very unpractical and highly expensive in time and money, and it is also very exhausting, so it is very necessary to have good and updated performance metrics. As in any machine learning problem, how the Image Captioning models are tested is crucial to know how much we have done, how much is necessary to do, what routes to take, and which to avoid. 1.1 Problem statement In any Natural Language Processing (NLP) tasks, such as automatic translation, image captioning, etc., comparing a set of reference sentences X and a prediction or candidate \hat{X} is usually used. This comparison is made by a metric which is a function f(X,\hat{X})\in\mathbb{R} . Theoretically, better metrics have a higher correlation with human scores. The metrics can be organized into some categories [ 44 ] : • n -gram based (such as BLEU, METEOR, ROUGE) • Edit-Distance-based (such as CIDER) • Embedding-based (such as BertScore, ClipScore) • Learned or trainable (such as Beer [ 38 ] , Blend [ 27 ] , and also our proposal) In this work, a learned metric is proposed that deals with and considers both aspects, semantic and lexical, as well as has a good correlation with humans. Our proposal is learned because we tackle the problem as a regression model where our y or dependent variable, is the score provided by humans in our dataset generation as well as another found in the literature. Several experiments were done, trying to find the best metric. 1.2 Contribution We proposed a new Image Captioning metric to evaluate the model’s output in both senses, lexical and semantical. To achieve that, it was necessary to build a new dataset through human tagging using as input 100 images from the MSCOCO dataset, the output of five state-of-the-art models, and the sixth one from the original set of captions from MS-COCO corresponding to the selected image. This overall process is described in Section 2 . Our main contributions are: • A new human-tagging dataset, corresponding to 600 image-caption score values. • An analysis of the results of the tagging process, as well as the taggers’ agreement and a comparison between models. • A metric proposal based on a model trained with data, trying to approximate the same performance as human taggers 1 1 1 available in https://github.com/ResearchInfoGeo/VCRScore . • A deep evaluation and comparison with other most used evaluation metrics. The paper is organized as follows, the section 2 details the entire procedure done by generating our dataset and its analysis and evaluation as well as the model’s performance evaluation done by humans. Section 3 describes all the components related to our metric proposal. Experimental results and their analysis are described in Section 4 , and finally, Section 5 concludes. 2 Dataset generation We generate a new dataset using a set of images from the MS-COCO dataset [ 26 ] and the caption prediction from five state-of-the-art models in addition to one caption related to the image from the MS-COCO dataset. Theoretically, this last caption is the best possible since humans did it. This produces a dataset with 600 image-captions evaluation pairs. The models used to generate these captions were diverse, from those proposed many years ago to those more recent, they are: BLIP2 [ 22 , 23 ] , OFA [ 40 ] , Convcap [ 5 ] , Meshed-Memory Transformer (M 2 ) [ 7 ] , and Show Attend And Tell (SAAT) [ 43 ] . In addition, we used the fifth reference from the MS-COCO dataset as another input for the assessment. The main idea is to build a metric that learns the relationship or score provided by humans when they evaluate more and less adequate image-caption pairs. 2.1 Tagging process As mentioned, a new dataset was generated from the knowledge of four taggers evaluating the captions from 100 images and the captions predictions from five different models. Furthermore, humans generated a correct caption from the MS-COCO dataset. That means a total of 600 pairs of image-caption to be evaluated by humans. To do that, we built a simple tool to show the images and a randomly selected caption. A screenshot of this tool can be seen in Figure 1 where it can be seen that there is an image displayed and also a caption related to it. Also, there are five score options to be selected by the human tagger, they are: 1. No error or lack of information (best possible option, value 1), 2. No error, but important info is missing (second best possible option, value 0.75) 3. It has minor errors (third best possible option, value 0.5) 4. It

Read the complete text