COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark

Ishant Chintapatla †*, Kazuma Choji †*, Naaisha Agarwal †*, Andrew Lin *, Hannah You *, Charles Duong °*, Kevin Zhu °*, Sean O’Brien °*, Vasu Sharma °* · 2025

Read COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark by Ishant Chintapatla †*, Kazuma Choji †*, Naaisha Agarwal †*, Andrew Lin *, Hannah You *, Charles Duong °*, Kevin Zhu °*, Sean O’Brien °*, Vasu Sharma °* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13405.txt.

Introduction

Ishant Chintapatla and co-authors build a test of a skill most picture benchmarks skip, in a 2025 paper. Vision-language models are usually asked questions about photos. Here they must judge whether a statement about a photo is true or false. Crowded scenes are the hard case: dozens of people overlap, and fine claims are easy to get wrong. Their benchmark, COREVQA, pairs photos from a crowd dataset with 5,608 made-up true or false statements.

Even the best models scored under eighty percent on the task. Others fell as low as forty percent, near guessing for some. That gap, the authors argue, exposes a blind spot in how these models reason about busy scenes, since ordinary question answering scores looked far stronger. The benchmark gives labs a sharper tool for a skill that matters in surveillance review, sports analysis, and other crowded settings.

Catalog record

CREATORIshant Chintapatla †*, Kazuma Choji †*, Naaisha Agarwal †*, Andrew Lin *, Hannah You *, Charles Duong °*, Kevin Zhu °*, Sean O’Brien °*, Vasu Sharma °*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark Ishant Chintapatla †*, Kazuma Choji †*, Naaisha Agarwal †*, Andrew Lin *, Hannah You *, Charles Duong °*, Kevin Zhu °*, Sean O’Brien °*, Vasu Sharma °* Abstract Recently, many benchmarks and datasets have been developed to evaluate Vision-Language Models (VLMs) using visual question answering (VQA) pairs, and models have shown significant accuracy improvements. However, these benchmarks rarely test the model’s ability to accurately complete visual entailment, for instance, accepting or refuting a hypothesis based on the image. To address this, we propose COREVQA (Crowd Observations and Reasoning Entailment), a benchmark of 5608 image and synthetically generated true/false statement pairs, with images derived from the CrowdHuman dataset ( Shao et al. 2018 ) , to provoke visual entailment reasoning on challenging crowded images. Our results show that even the top-performing VLMs achieve accuracy below 80%, with other models performing substantially worse (39.98%-69.95%). This significant performance gap reveals key limitations in VLMs’ ability to reason over certain types of image–question pairs in crowded scenes. 1 1 1 Huggingface link: https://huggingface.co/datasets/COREVQA2025/COREVQA 2 2 2 Github link: https://github.com/corevqa/COREVQA † Equal Contribution * Algoverse AI Research ° Equal Senior Authorship 1 Introduction Vision language models (VLM), such as GPT-4.1 ( Achiam et al. 2023 ) and Gemini 2.5 Pro ( Cloud ) , have shown remarkable capabilities in image understanding and multimodal task completion ( Li et al. 2025 ) . As VLMs grow more sophisticated, the demand for rigorous evaluation methods that assess deep visual and textual understanding becomes increasingly critical. ( Huang et al. 2023 ; Agrawal et al. 2018 ; Goyal et al. 2017 ) . However, despite offering a wide variety of tasks, existing VLM evaluation benchmarks often fall short in assessing nuanced understanding and deep comprehension of natural situations and analysis, primarily due to simple and easy-to-understand images/questions. These limitations mean models may succeed (and continue to succeed) by exploiting superficial cues or relying on parametric knowledge, without robust visual processing. The impact is impeding VLM improvements due to a scarcity of robust multimodal reasoning assessments ( Li et al. 2025 ; Guan et al. 2024 ) . To fill this void in VLM assessment, we propose COREVQA (Crowd Observations and Reasoning Entailment Visual Question Answering)–a challenging evaluation benchmark based on images of dense human crowds in complex, natural settings. While existing crowd-based datasets only assess or focus on recognition, detection, and counting ( Wang et al. 2020a ; Wang et al. 2020b ; Sindagi et al. 2020 ; Wang et al. 2019 ; Zhang et al. 2016 ; Xie et al. 2019 ) , COREVQA requires models to integrate fine-grain visual analysis with textual logic in scenarios where visual ambiguity and seemingly trivial, easy-to-miss details are key. COREVQA aims to help researchers spot flaws and gaps in VLM understanding that will spur improvement in robustness. Our main contributions are as follows. • We propose a pipeline to synthetically generate difficult questions for specific images based on typical VLM weaknesses. • We created the first large-scale benchmark with multi-person, crowd-based images for evaluating VLM capability in busy scenarios (using our proposed pipeline). • We evaluate several state-of-the-art (SOTA) VLMs on COREVQA, revealing across-the-board struggle with nuances and fine details when dealing with images with several people overflowing with diverse shapes, colors, and sizes. 2 Related Work 2.1 Vision-Language Benchmarks Several benchmarks have become standard for evaluating core Visual-Question Answering (VQA) abilities. VQAv2 ( Goyal et al. 2017 ) is the successor to the original VQA dataset ( Antol et al. 2015 ) , which aims to assess general VQA performance through a more balanced and challenging benchmark than VQA. Though they are still used for standardized evaluation, typical VQA datasets (like OK-VQA ( Marino et al. 2019 ) and TextVQA ( Singh et al. 2019 ) ) lack complexity and difficulty. ( Goyal et al. 2017 ) . Newer datasets that analyze visual reasoning, understanding, recognition, and question answering include MMTBench ( Ying et al. 2024 ) , VCR ( Zellers et al. 2019 ) , MM-Vet ( Yu et al. 2023 ) , SEEDBench ( Li et al. 2023 ) , and NaturalBench ( Li et al. 2024 ) . Most recent datasets overlook improving evaluation quality, and focus on assessing a wider range of tasks for easy and standardized comparison with other models ( Fu et al. 2024 ) . These benchmarks include MMBench ( Liu et al. 2024b ) , MMMU ( Yue et al. 2024 ) , MMStar ( Chen et al. 2024 ) , M3GIA ( Song et al. 2024 ) . Since most popular VLM benchmarks are designed in a multiple-choice format, further research is required to assess their robustness and reliability ( Li et al. 2025 ) . Some others include NTSEBENCH ( Pandya et al. 2024 ) and VLDBENCH ( Raza et al. 2025 ) , which target specific areas of visual reasoning. Difficult, targeted datasets like HallusionBench ( Guan et al. 2024 ) have been created to evaluate key VLM weaknesses. Visual entailment benchmarks such as SNLI-VE ( Xie et al. 2019 ) , Defeasible Visual Entailment ( Zhang et al. 2025 ) , and VALSE ( Parcalabescu et al. 2021 ) have all created questions that help test the models’ ability to understand the text in regards to the image. Primary crowd-based datasets include NWPU-Crowd ( Wang et al. 2020a ) , JHU-CROWD++ ( Sindagi et al. 2020 ) , PANDA ( Wang et al. 2020b ) , and GCC ( Wang et al. 2019 ) . By using a true/false format (rather than the primary usage of multiple choice ( Li et al. 2025 ) ) we reduce confounding variables found in answer choices because models cannot pick a correct choice when only given true or false (by nature) ( Balepur et al. 2024 ) . Through our crowd-based dataset and synthetic generation we probe both visual entailment and textual comprehension, unlike many datasets mentioned that typically lack complex images (see more in A.6 ). 3 COREVQA COREVQA is a novel VQA benchmark designed to evaluate VLMs’ capabilities for detailed visual inspection and multi-step visual entailment. The benchmark features true/false statements about images that sound plausible but require careful visual grounding to verify their entailment status. 3.1 Benchmark Overview COREVQA tests two core capabilities: depth of visual entailment and precision in analyzing fine-grained visual details. Models must classify the truthfulness of complex statements about images, with statements averaging 30 words and featuring complex sentence structures. The binary classification task assesses two main tasks – Meticulous Visual Inspection: identifying subtle details in visual clutter or peripheral regions, and Complex Visual Entailment: understanding spatial relationships, making contextual inferences, and resisting plausible misdirection. The benchmark has 5,608 unique image-statement pairs. Images come from the CrowdHuman dataset ( Shao et al. 2018 ) , featuring diverse indoor and outdoor environments with groups of people. Each image is paired with one unique true/false statement generated through prompting ChatGPT for true statements and Claude for false statements. Ground truths were hand-labeled. Table 1: Key Statistics of the COREVQA Dataset Characteristic Value Dataset size 5,608 image-statement pairs true statements 1,566 (27.9%) false statements 4,042

Read the complete text