CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models

Alexis Roger, Prateek Humane, Daniel Z. Kaplan, Kshitij Gupta, Qi Sun, George Adamopoulos, Jonathan Siu Chi Lim, Quentin Anthony, Edwin Fennell, Irina Rish, *Equal contribution, Mila - Quebec AI Institute, Université de Montréal, realiz.ai · 2025

Read CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models by Alexis Roger, Prateek Humane, Daniel Z. Kaplan, Kshitij Gupta, Qi Sun, George Adamopoulos, Jonathan Siu Chi Lim, Quentin Anthony, Edwin Fennell, Irina Rish, *Equal contribution, Mila - Quebec AI Institute, Université de Montréal, realiz.ai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09672.txt.

Introduction

Alexis Roger and co-authors build CHIRP, a fine-grained benchmark for judging vision-language model answers, in this 2025 paper. These models answer open-ended questions about images, which are hard to grade. Current grading mixes automatic scores, AI judges, and human review.

The team first built a family of models at several sizes to probe grading methods. Tests with those models exposed flaws in how answers are now scored. CHIRP was designed to catch finer differences in answer quality. It aims to show when a score reflects real skill and when it does not. Better grading should steer model builders toward answers people truly value. The bench is open for other teams to test their own models.

Catalog record

CREATORAlexis Roger, Prateek Humane, Daniel Z. Kaplan, Kshitij Gupta, Qi Sun, George Adamopoulos, Jonathan Siu Chi Lim, Quentin Anthony, Edwin Fennell, Irina Rish, *Equal contribution, Mila - Quebec AI Institute, Université de Montréal, realiz.ai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09672

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models Alexis Roger, Prateek Humane, Daniel Z. Kaplan, Kshitij Gupta, Qi Sun, George Adamopoulos, Jonathan Siu Chi Lim, Quentin Anthony, Edwin Fennell, Irina Rish, *Equal contribution, Mila - Quebec AI Institute, Université de Montréal, realiz.ai Abstract The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and benchmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We test our model on a scaling suite across vision and langauge sizes, identifiying shortcomings of current evaluation approaches. To overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to our evaluation code and the CHIRP benchmark to promote reproducibility and advance VLM research. 1 Introduction Recently, a lot of significant advances have been made in Vision-Language Models (VLMs), driven by breakthroughs in computer vision and natural language processing Chen et al. (2022) ; Li et al. (2023b) ; Liu et al. (2023b) ; Sun et al. (2023) . However, existing VLM benchmarks, often designed for specific tasks (e.g., VQAv2 Goyal et al. (2017) ), struggle to accurately reflect real-world VLM performance and capture nuanced differences between models Hsieh et al. (2024) . This is particularly evident when evaluating models with significant architectural variations, where standard benchmark scores remain similar despite noticeable differences in human-perceived model quality. To address this issue, we introduce CHIRP, a hybrid VLM benchmark that combines automated metrics’ scalability with human evaluators’ nuanced judgment. We argue that this approach is crucial for capturing the complexities of VLM behavior, which traditional benchmarks often fail to represent. To demonstrate the limitations of existing benchmarks and the efficacy of our proposed method, we use a scaling suite called Robin, a series of VLMs trained at various scales, inspired by the Pythia language model suite Biderman et al. (2023) . By systematically varying the Vision Encoder (VE) and the Large Language Model (LLM) sizes, we see that while some benchmark scores remain largely unaffected, human evaluations reveal significant differences in the models’ outputs quality. Our findings underscore the need for more robust and human-centric VLM evaluation methodologies. CHIRP paves the way for developing more reliable and informative VLM benchmarks, ultimately leading to the creation of more effective and impactful VLMs. Furthermore, we will also investigate in detail the possibility of using different AI agents as proxies to human evaluations. 2 Related Work Scaling Suites. Scaling laws have recently emerged as one of the central research areas in large foundation models Aghajanyan et al. (2023) ; Isik et al. (2024) . These laws enable performance prediction based on variations in compute time, dataset size, and model parameters, facilitating efficient resource allocation by extrapolating results from small-scale experiments. Kaplan et al. (2020) pioneered the application of scaling laws to language models, demonstrating a power-law relationships between loss and model size, dataset size, and compute time. This has led to practical applications, such as the Pythia suite Biderman et al. (2023) , which comprises of identically trained language models with varying parameter sizes, empirically verifying these scaling laws. Cherti et al. (2023) investigated the scaling laws of the CLIP vision encoders, training and comparing different sizes of the CLIP vision encoders on the same data. These models indeed verified the aforementioned scaling laws and have become a very popular suite of models. AI-based Evaluation. The advent of powerful foundation models like GPT-4V offers a new way to evaluate weaker models, moving beyond traditional, rigid metrics such as exact string matching, as done in popular benchmarks like Hudson & Manning (2019) ; Mishra et al. (2019) ; Singh et al. (2019) . Early evidence from benchmarks like MM-Vet Yu et al. (2023) and VQA tasks Agrawal et al. (2016) suggests that evaluating with stronger models offers a promising path towards more comprehensive and insightful evaluation, surpassing the limitations of static, string-based methods Ji et al. (2023) ; Lee et al. (2024) . This shift towards leveraging the semantic understanding of LLMs for evaluations promises to unlock a better evaluations, leading to a better understanding of model capabilities. Zheng et al. (2023) introduce two benchmarks, MT-Bench and Chatbot Arena, to explore the feasibility of employing LLMs as judges. Their findings indicate that advanced LLMs, such as GPT-4, closely align with human preferences, achieving over 80% of agreement Rafailov et al. (2024) . Similarly, AlpacaEval Li et al. (2023a) utilizes LLMs to assess instruction-following models. Wu & Aji (2023) focused on the bias in evaluations conducted by both human and LLM annotators, particularly noting a preference for flawed content if it avoids brevity or grammatical errors, and introduced the Multi-Elo Rating System (MERS) for more nuanced assessments. A study by Koo et al. (2023) pointed out significant biases of LLMs evaluators, with an average Rank-Biased Overlap (RBO) score of 49.6%, suggesting a misalignment between machine and human preferences. While there exists benchmarks like LLaVA-Bench Liu et al. (2023a) which elicit open-ended responses, and evaluate using LLMs, they often lack thought-provoking and diverse questions that challenge a model’s perception and contextual understanding. For example, most of the open-ended questions in LLaVA-Bench are of the form: "describe the image…". We address this shortcoming with our benchmark CHIRP. 3 CHIRP We introduce CHIRP, a new evaluation benchmark, which grades long form responses. CHIRP comprises of 104 open ended questions, evaluated by either humans or VLMs. These free form questions do not correspond to a single "correct" answer. Instead, they require models to generate flexible, creative and complex responses. Consequently, we evaluate models using a preference based rating in which two model’s responses are compared side by side. Instructions on downloading the CHIRP benchmark can be found in Appendix C . We will show in Section 5.2 that current benchmarks evaluate models with high variance, making it difficult to accurately rank models with similar performance. CHIRP addresses this issue by using significantly fewer but higher-quality prompts and pairwise evaluations to establish a more reliable ranking. 3.1 Generating the dataset We wrote questions along with image descriptions, which were then refined with the help of GPT-4 OpenAI (2023) . The image descriptions were given to Dalle-E 3 to generate the associated images. We finetuned the image descriptions and questions by hand to generate the optimal image question pairs. The questions created are classified in 8 distinct categories: descriptive analysis, inferential reasoning, contextual understanding, emotional and psychological understanding, ethical evaluations, abstract understanding, creative and subjective analysis, and visual aesthetics evaluation. Detailed descriptions and examples of these categories can be found in Appendix C.1 . Unlike many datasets that rely on pre-existing images, our approach allows us to generate images specifically tailored to our thought-provoking questions and detailed analysis. This also removes the risk of the model having seen the image in training. Moreover, we eliminate the risk of evaluating models on contaminated images as all of them were validated by hand. A sample text-image pair of the CHIRP dataset can be seen in Appendix C.2.1 . 3.2 Human based evaluations We utilize CloudResearch for large scale human evaluation of our

Read the complete text