VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
Chenyu Wang, Tianle ChenH. M. Sabbir AhmadKayhan BatmanghelichWenchao LiBoston University {chyuwang, tianle, sabbir92, batman, wenchao}@bu.edu · 2026
Read VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models by Chenyu Wang, Tianle ChenH. M. Sabbir AhmadKayhan BatmanghelichWenchao LiBoston University {chyuwang, tianle, sabbir92, batman, wenchao}@bu.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09214.txt.
Introduction
Vision-language models can be confidently wrong, and knowing whether doubt comes from the image, the text, or their mismatch matters for safety. Chenyu Wang and four co-authors introduce VLM-UQBench, a benchmark for modality-specific and cross-modal uncertainty, built from 600 real VizWiz samples sorted into clean, image-uncertain, text-uncertain, and cross-modal subsets. A perturbation pipeline adds eight visual, five textual, and three cross-modal distortions.
Two new metrics track how uncertainty scores react to perturbations and how they correlate with hallucinations, applied across four vision-language models and three data sets. Findings are sobering. Methods specialize by modality and depend heavily on the underlying model. Modality-specific uncertainty often accompanies hallucinations while scores give weak risk signals, and subtle instance-level ambiguity largely escapes detection. The gap between current practice and fine-grained, modality-aware uncertainty is large.
Catalog record
| CREATOR | Chenyu Wang, Tianle ChenH. M. Sabbir AhmadKayhan BatmanghelichWenchao LiBoston University {chyuwang, tianle, sabbir92, batman, wenchao}@bu.edu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09214
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models Chenyu Wang, Tianle ChenH. M. Sabbir AhmadKayhan BatmanghelichWenchao LiBoston University {chyuwang, tianle, sabbir92, batman, wenchao}@bu.edu Abstract Uncertainty quantification (UQ) is vital for ensuring that vision–language models (VLMs) behave safely and reliably. A central challenge is to localize uncertainty to its source, determining whether it arises from the image, the text, or misalignment between the two. We introduce VLM-UQBench, a benchmark for modality-specific and cross-modal data uncertainty in VLMs, It consists of 600 real-world samples drawn from the VizWiz dataset, curated into clean, image-, text-, and cross-modal uncertainty subsets, and a scalable perturbation pipeline with 8 visual, 5 textual, and 3 cross-modal perturbations. We further propose two simple metrics that quantify the sensitivity of UQ scores to these perturbations and their correlation with hallucinations, and use them to evaluate a range of UQ methods across four VLMs and three datasets. Empirically, we find that: (i) existing UQ methods exhibit strong modality-specific specialization and substantial dependence on the underlying VLM, (ii) modality-specific uncertainty frequently co-occurs with hallucinations while current UQ scores provide only weak and inconsistent risk signals, and (iii) although UQ methods can rival reasoning-based chain-of-thought baselines on overt, group-level ambiguity, they largely fail to detect the subtle, instance-level ambiguity introduced by our perturbation pipeline. These results highlight a significant gap between current UQ practices and the fine-grained, modality-aware uncertainty required for reliable VLM deployment. 1 Introduction Figure 1 : Benchmark composition and curation workflow. We integrate diverse data sources through a three-stage pipeline of human annotation , rule-based filtering , and expert curation (upper left), yielding four VizWiz-based subsets with modality-specific uncertainty labels (Clean, Image, Text, Cross; top right). VQ-FocusAmbiguity is leveraged as a grounded case of cross-modal ambiguity, providing unambiguous vs. ambiguous text–image alignment examples (middle right). Hallucination-focused subsets are built from CLEVR scene graphs using compositional templates and rule-based generation, enabling controlled evaluation of attribute, existence, counting, and relation hallucinations (bottom left). Uncertainty quantification (UQ) is becoming a core requirement for vision–language models (VLMs) deployed in downstream tasks such as assistive question answering, everyday visual assistants, robotics, and medical or scientific image analysis 44 ; 17 ; 38 ; 45 . In these settings, VLMs must not only produce answers but also decide when to abstain, ask for clarification, or escalate to a human, since hallucinations can be costly or harmful. Effective UQ must therefore indicate both how uncertain the model is and where that uncertainty arises—for example, whether it is caused by a low-quality or ambiguous image, an under-specified or conflicting text query, or a mismatch between the two. Such source-aware UQ is crucial for taking the appropriate downstream action (retake the image, rephrase the question, or defer), yet current VLM systems provide little structured support for this modality-specific view of uncertainty. Most existing work on UQ adopts the classical epistemic–aleatoric decomposition, attributing uncertainty either to the model (epistemic) or to the data (aleatoric). Recent work in Natural Language Generation (NLG) and LLMs questions whether this view is expressive enough for modern generative systems. Recent work catalogues sources of uncertainty in NLG and proposes a two-dimensional taxonomy that separates where uncertainty comes from and how it manifests, arguing that this is more informative than the aleatoric/epistemic dichotomy 1 . In a similar spirit, subsequent work reviews the standard distinction for LLMs and shows that it misses key sources of uncertainty introduced by large-scale pre-training, alignment, and user interaction, instead distinguishing operational uncertainty (along the model lifecycle) from output uncertainty (in generated content) 2 . Further analysis of reducibility-based definitions, where epistemic uncertainty is treated as reducible and aleatoric as irreducible, shows that for interactive agents the same situation can often be made more or less “reducible” by acquiring additional information or taking actions, making a strict binary split unstable 24 . Taken together, these works suggest that the classical epistemic/aleatoric view is too coarse for modern generative models: we need uncertainty descriptions that are both source-aware and explicit about reducibility, which is especially important for VLMs, where uncertainty may originate from the image, the text, or their interaction, and where model architectures and training paradigms are even more complex. In parallel with these conceptual critiques, several recent works argue that progress in UQ requires matching progress in benchmarks and evaluation protocols, and explicitly call for more comprehensive, task-diverse benchmarks that go beyond a single scalar confidence score and standard QA datasets 2 ; 24 . In the VLM setting, existing datasets make only partial steps in this direction. UNK-VQA 11 and FocusAmbiguity 5 provide useful labels for answerability and grounded ambiguity in open-ended and free-form VQA, but the uncertainty types they cover are limited. MM-UPD 33 offers more fine-grained unsolvability categories, but is restricted to multiple-choice VQA. Besides, these benchmarks are all created at a group level, without instance-wise contrastive structure that would support controlled evaluation. MuirBench 46 moves toward such contrastive design by constructing pairwise instances, where each standard example is paired with an unanswerable variant, but it is still limited to multiple-choice VQA and focuses only on answerability versus unanswerability. CertainlyUncertain 4 provides fine-grained, instance-wise annotations for free-form VQA, but its annotations are defined within the aleatoric–epistemic dichotomy, which is not well aligned with the modality-specific uncertainty needs of VLMs. Moreover, most of these datasets were originally developed to evaluate model performance, robustness, or answerability, rather than to directly probe UQ methods. A dedicated benchmark for UQ in VLMs should instead offer fine-grained, instance-wise structure across multiple tasks, and explicitly support modality-aware uncertainty analysis that can separate image, text, and cross-modal sources. To address these gaps, we introduce VLM-UQBench, a benchmark specifically designed for modality-specific uncertainty in visual question answering. At its core, VLM-UQBench contains four subsets derived from VizWiz 12 : a clean set, an image-uncertainty set, a text-uncertainty set, and a cross-modality uncertainty set. Each example is curated with expert annotations that identify the primary source of uncertainty, giving ground-truth labels for whether uncertainty comes from the image, the text, or their interaction. To complement these real-world subsets, we incorporate a grounded cross-modality subset from VQ-FocusAmbiguity 5 , which uses segmentation-based grounding to mark all valid referents for questions with multiple plausible image regions. We adopt its ambiguous and unambiguous pairs as grounded cross-modal samples, providing explicit supervision for uncertainty arising from text–image alignment. Together, these components offer rich, modality-aware annotations that go beyond coarse epistemic/aleatoric tags. On top of the annotated subsets, VLM-UQBench includes a scalable perturbation pipeline and hallucination-focused subsets. The perturbation pipeline introduces instance-wise, fine-grained uncertainty through targeted visual, textual, and cross-modal edits, allowing us to construct contrastive pairs and probe how UQ methods respond to controlled, modality-specific perturbations without additional human labels. The hallucination subsets, built from synthetic scenes with full scene graphs, further enable systematic evaluation of how different UQ methods relate to hallucination behaviour under controlled visual and cross-modal changes. The contributions of this paper are fourfold: • Modality-specific benchmark. We formalize and release VLM-UQBench , which, as described above, provides modality-specific uncertainty labels for image, text, and cross-modal sources in free-form VQA, combining real-world data with grounded cross-modal ambiguity from VQ-FocusAmbiguity. • Scalable pipeline and paired metrics. We design a scalable perturbation pipeline that