SPECS: Specificity-Enhanced CLIPScore for Long Image Caption Evaluation

Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva · 2025

Read SPECS: Specificity-Enhanced CLIPScore for Long Image Caption Evaluation by Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-03897.txt.

Introduction

In a 2025 research paper, Xiaofu Chen and two co-authors examine SPECS, a metric for judging long, detailed image captions without needing reference captions. Older n-gram metrics are cheap but miss semantic correctness. Similarity metrics built on embeddings correlate poorly with human judgments in practice. Large language model judges correlate well but cost too much for repeated use during model development.

SPECS modifies the CLIP model with a new training objective built around specificity. Correct details earn reward, and incorrect details earn penalty, instead of treating all content alike. The metric is reference-free: a caption is scored against its image alone. In their tests, SPECS matches open-source language-model metrics in correlation with human judgments, far more efficiently, since no large model runs for every score. The authors position it as a practical tool for checkpoint evaluation during model development, with code released.

Catalog record

CREATORXiaofu Chen, Israfel Salazar, Yova Kementchedjhieva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-03897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPECS: Specificity-Enhanced CLIPScore for Long Image Caption Evaluation Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva Abstract As interest grows in generating long, detailed image captions, standard evaluation metrics become increasingly unreliable. N-gram-based metrics though efficient, fail to capture semantic correctness. Representational Similarity (RS) metrics, designed to address this, initially saw limited use due to high computational costs, while today, despite advances in hardware, they remain unpopular due to low correlation to human judgments. Meanwhile, metrics based on large language models (LLMs) show strong correlation with human judgments, but remain too expensive for iterative use during model development. We introduce SPECS (Specificity-Enhanced CLIPScore), a reference-free RS metric tailored to long image captioning. SPECS modifies CLIP with a new objective that emphasizes specificity: rewarding correct details and penalizing incorrect ones. We show that SPECS matches the performance of open-source LLM-based metrics in correlation to human judgments, while being far more efficient. This makes it a practical alternative for iterative checkpoint evaluation during image captioning model development. Our code can be found at https://github.com/mbzuai-nlp/SPECS . 1 Introduction Image captioning has been a key topic in vision-language research, offering a controlled setting to study grounded language generation Karpathy and Fei-Fei 2015 . While early efforts focused on short, general captions Vinyals et al. 2015 , recent work has shifted toward generating long, detailed descriptions that capture fine-grained visual information Johnson et al. 2016 ; Cho et al. 2022 ; Doveh et al. 2023 ; Li et al. 2023 . This complex task requires strong visual grounding and improved cross-modal alignment Liu et al. 2024 ; Li et al. 2021 ; Xie et al. 2025 . It expands the scope of generative vision-language modeling but also magnifies a long-standing challenge: how to evaluate captions reliably and efficiently. Figure 1: Example of specificity in caption evaluation. Given an image, captions with increasing correct details (“jumper” \rightarrow “blanket” \rightarrow “blanket with fringed edges”) should receive progressively higher cosine similarity, (\cdot) , to the image. SPECS ranks these minimal pairs correctly at a high rate, reflecting its strong specificity. Automatic evaluation in captioning, as in other natural language generation tasks, has long been a challenge Otani et al. 2023 ; Wang et al. 2023 . Early metrics rely on n -gram overlap between generated captions and references. While computationally efficient, these methods fail to capture semantic similarity, often penalizing valid paraphrases and underestimating the severity of hallucinations, even in short captions Papineni et al. 2002 ; Banerjee and Lavie 2005 ; Vedantam et al. 2015 ; Lin 2004 . To address these limitations, Representational Similarity (RS) metrics use pretrained vision-language models to compare image and caption embeddings in a shared feature space Hessel et al. 2021 ; Sarto et al. 2023 , often eliminating the need for reference captions. More recently, evaluation metrics based on large language models (LLMs) have become the standard, showing strong correlation with human judgments, especially as the length of generated image captions increases Chan et al. 2023 ; Yu et al. 2024 ; Ye et al. 2025 . Each new generation of evaluation metrics brings improvements in semantic expressiveness, but often at the cost of higher computational requirements. As a result, there is frequently a mismatch between what is technically feasible and what is practical for routine model development. For instance, although CLIPScore Hessel et al. 2021 demonstrated stronger semantic alignment than traditional metrics like CIDEr Vedantam et al. 2015 , its adoption remained limited due to the high computational cost at the time of its release. A similar pattern is now emerging with LLM-based metrics: while they achieve state-of-the-art correlation with human judgments, their inference cost makes them impractical for iterative evaluation during model training and development. This tradeoff between reliability and efficiency becomes particularly problematic in the context of long image captioning, where no existing metric can strike a reasonable balance between the two. Recent studies have shown that even simple metrics like BLEU-4 outperform RS metrics like CLIPScore in terms of correlation with human judgments ( Ye et al. 2025 , sample-level Kendall’s Tau correlations of 0.27 and 0.17, respectively) . These results underscore a pressing gap: there is no metric that reliably evaluates the quality of long captions while remaining computationally practical. In this work, we introduce SPECS ( SP ecificity- E nhanced C LIP- S core), a reference-free RS metric designed for the evaluating long image captions. SPECS builds on a long-context adaptation of CLIP Zhang et al. 2024 and incorporates a new training objective that emphasizes specificity : rewarding correct details and penalizing incorrect ones. In extensive evaluations, SPECS matches the best open-source LLM-based metric Lee et al. 2024 in terms of correlation to human judgments, while being over two orders of magnitude more efficient. SPECS is a practical and scalable solution for iterative model development in long caption generation. 2 Related work Image captioning metrics are central to evaluating vision language models (VLMs). These metrics score how well a model can describe an image, in a way that aligns with human judgment. Image Caption Evaluation Methods Early work relied on n -gram matching, where metrics such as BLEU Papineni et al. 2002 , METEOR Banerjee and Lavie 2005 , ROUGE Lin 2004 , and CIDEr Vedantam et al. 2015 compared generated captions against human references using surface lexical overlap. While easy to compute, these metrics often fail when captions use different but valid phrasings, leading to low correlation with human judgment. To overcome this, later approaches explored semantic parsing, such as SPICE Anderson et al. 2016 , which evaluates scene-graph structures. CLIPScore Hessel et al. 2021 and PACScore Sarto et al. 2023 build on pre-trained vision–language models such as CLIP Radford et al. 2021 , assessing captions through representational similarity between images and their captions. More recently, large language models (LLMs) have been used as evaluators by prompting them to score alignment between captions and images: FaithScore Jing et al. 2023 extracts atomic facts from captions using an MLLM and verifying each fact against the image to measure faithfulness. CLAIR Chan et al. 2023 leverages LLMs in a zero-shot setting to score captions and explain their judgments, GPT4-Eval Liu et al. 2023 is evaluated by prompting GPT-4 to judge multimodal responses, with performance measured by relative quality scores. RLAIF-V Yu et al. 2024 evaluates captions by prompting open-source MLLMs to verify decomposed claims and score hallucination reduction. FLEUR Lee et al. 2024 prompts an open-source MLLM to directly compare an image with a candidate caption in a reference-free setting, producing both a numerical score and a natural-language rationale to align evaluation with human judgments. CAPTURE Dong et al. 2024 evaluates captions by extracting and matching fine-grained visual details. DCScore Ye et al. 2025 evaluates captions by decomposing them into primitive information units and measuring their accuracy and coverage using GPT-4. These LLM-based methods show strong performance but remain prohibitively expensive and mostly closed-source. CLIP-based adaptations Metrics based on representational similarity carry promise as they operate on the level of semantics rather than surface, and they are less costly than LLMs. However, CLIP, the common choice for base model in such metrics, is not suitable for long caption evaluation for two key reasons: it lacks compositionality and it

Read the complete text