VistaQA : Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki · 2026

Read VistaQA : Benchmarking Joint Visual Question Answering and Pixel-Level Evidence by Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20676.txt.

Introduction

Mozhgan Nasr Azadani and seven colleagues present VistaQA, a benchmark that ties visual answers to pixel-level proof. Current tests check text answers or object locations alone, leaving reasoning and grounding unlinked. VistaQA holds 1,157 expert-checked samples across six task types and six visual domains, from direct perception to relational reasoning. Models must answer correctly and supply precise segmentation masks that back each answer.

Catalog record

CREATORMozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59234
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VistaQA : Benchmarking Joint Visual Question Answering and Pixel-Level Evidence Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki Abstract Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VistaQA , a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VistaQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VistaQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce Grove , a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under Grove , highlighting a substantial gap between answer accuracy and visual evidence alignment. † † footnotetext: * Equal contribution. \dagger Equal advising. \textsuperscript{} \textsuperscript{} footnotetext: Correspondence to: [email protected] Figure 1 : VistaQA jointly evaluates answer correctness and pixel-level evidence across six tasks and six domains, requiring both to be correct and preventing compensation between modalities. 1 Introduction Multimodal large language models (MLLMs) have demonstrated remarkable progress across a wide range of multimodal tasks, including visual question answering (VQA), image captioning, and compositional reasoning [ 42 , 36 , 37 , 5 , 6 ] . Despite these advances, standard evaluation protocols [ 21 , 46 , 28 ] continue to assess performance primarily through textual correctness. A model is rewarded for producing the right answer, even when that answer is not supported by the appropriate visual evidence. Conversely, a model may produce plausible but incorrect answers driven by language priors, despite contradictory visual cues (e.g., answering “two” legs for an animal that visibly has three [ 34 ] ) [ 13 , 19 , 8 ] . This creates a fundamental limitation: textual correctness alone cannot determine whether a reply is truly grounded in the image, an essential requirement for model interpretability and hallucination mitigation. For reliable deployment in real-world settings, models must be evaluated on not only what they say , but also what visual evidence supports their statements. Recent efforts toward grounding have emerged from two complementary but largely disconnected directions. On one hand, reasoning segmentation models [ 17 , 29 , 38 , 40 , 41 ] demonstrate strong pixel-level localization capabilities, but they are not optimized for generating rich, free-form textual answers and have been reported to achieve limited success on general VQA benchmarks [ 48 ] . On the other hand, state-of-the-art MLLMs [ 11 , 26 ] achieve high accuracy on complex multimodal reasoning tasks, but typically do not produce explicit visual evidence to substantiate their predictions. While more recent visual grounding approaches [ 45 , 35 , 44 , 27 ] begin to bridge this gap, their textual and visual capabilities are still largely evaluated in isolation: models are scored on answer accuracy and segmentation quality as separate metrics, usually on different datasets ( e.g. , VQA benchmarks for text and segmentation datasets for masks). Consequently, existing evaluation protocols do not enforce alignment between answers and their supporting visual evidence, leaving the coupling of reasoning and grounding an open challenge. Existing benchmarks reflect and reinforce this divide. VQA benchmarks [ 9 , 47 , 22 , 12 ] evaluate compositional reasoning through textual answers, but provide no mechanism to verify whether those answers are grounded in the correct image regions. Conversely, referring expression and reasoning segmentation benchmarks [ 17 , 43 , 30 ] emphasize localization, but treat segmentation as the prediction target rather than as evidence supporting a textual answer. More recent efforts attempt to combine answering and grounding signals; however, grounding is typically treated as a referring or auxiliary output rather than explicit evidence [ 39 , 4 ] , and evaluation protocols assess answer correctness and localization quality independently, without enforcing consistency between them [ 45 , 35 ] . In addition, these benchmarks are often limited to narrow domains or task settings and rarely account for hallucination, where models must recognize the absence of valid visual evidence. As a result, there remains no benchmark that systematically requires models to produce correct free-form answers that are explicitly supported by pixel-level visual evidence across diverse tasks and domains. To address this gap, we introduce VistaQA , a benchmark for the joint evaluation of free-form answer correctness and pixel-level evidence grounding in VQA. Each sample in VistaQA consists of a question, a reference answer, and a segmentation mask that specifies the visual evidence required to support that answer. Current state-of-the-art MLLMs are insufficiently reliable to autonomously generate complex image–question–mask triplets end-to-end, so constructing a benchmark that jointly evaluates answer correctness and pixel-level evidence grounding requires rigorous human control at every stage of the pipeline. VistaQA thus comprises 1,157 carefully curated samples spanning six task types, including identification , attribute , OCR , spatial , counting , and comparison and six visual domains, including indoor , outdoor , autonomous driving , robotics , science , and mathematics . These tasks cover the spectrum from direct perceptual recognition to compositional and relational reasoning in diverse real-world settings. These tasks and domains are illustrated in Figure 1 . Notably, VistaQA includes hallucination-aware samples, where questions are unanswerable or refer to absent entities, requiring models to correctly identify the absence of valid visual evidence. We formulate evaluation as a joint correctness problem: a prediction is considered fully correct only when both the textual answer and the corresponding evidence mask are correct. To support this setting, we introduce Grove ( GRO unded e V idence E valuation), a unified metric that jointly measures answer correctness and grounding fidelity by computing a per-sample geometric mean of smoothed text and mask scores, ensuring that both dimensions are satisfied simultaneously. Grove is designed around two key desiderata: (1) joint sensitivity , penalizing failures in either modality, and (2) graceful degradation , preserving signal under partial correctness, while remaining applicable across diverse model classes and evaluation settings. Our main contributions are as follows: • We introduce VistaQA , a comprehensive benchmark for the joint evaluation of free-form answers and pixel-level supporting evidence across six tasks and six domains, with explicit hallucination-aware scenarios. • We propose Grove , a unified evaluation metric that jointly measures answer correctness and grounding fidelity by computing a per-sample geometric mean of smoothed text and mask scores, ensuring that both dimensions are satisfied simultaneously. • We conduct comprehensive experiments across state-of-the-art baselines on VistaQA , demonstrating that even the strongest current models struggle to align answers with correct visual evidence. 2 Related work Early benchmarks for

Read the complete text