Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
Ziqi Wen, Parsa Madinei, Miguel P. Eckstein · 2026
Read Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency by Ziqi Wen, Parsa Madinei, Miguel P. Eckstein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13047.txt.
Introduction
Wen, Madinei and Eckstein measure how vision-language models describe scenes versus humans, using counterfactual semantic saliency: the semantic shift when an object is causally removed from a scene. Against a human baseline of 16,603 responses over 307 scenes and 1,306 counterfactual variants, models over-rely on large, central and high-saliency objects, and under-rely on people. A model's size bias largely explains how far its scene descriptions diverge from humans'.
Catalog record
| CREATOR | Ziqi Wen, Parsa Madinei, Miguel P. Eckstein |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45423 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13047
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency Ziqi Wen, Parsa Madinei, Miguel P. Eckstein Abstract Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,289 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high saliency objects. In contrast, models rely less on people in the scenes than our human participants to describe the images . A model’s size bias is a primary driver explaining variations in model-human semantic divergence. Code and data will be available at https://github.com/starsky77/Counterfactual-Semantic-Saliency . Figure 1 : Counterfactual Semantic Saliency (CSS) reveals a pervasive perceptual gap between humans and vision-language models (VLMs). CSS quantifies the importance of an object by ablating it from a factual scene and measuring the semantic shift in the resulting descriptions. The object causing the maximal semantic shift is considered the most critical to scene perception. The example illustrates a mismatch between ALL 19 models and human consensus. The rightmost plot illustrates the percentage of the evaluated VLMs that failed to identify the human-consensus critical object for a given scene. 1 Introduction As deep neural networks (DNNs) achieve superior performance, verifying whether these models exhibit perceptual strategies aligned with humans has become a critical challenge. Studies in AI-human alignment have revealed a persistent divergence in these strategies: even when achieving comparable accuracy on benchmark tasks, DNNs and human observers often rely on different features. Compared with humans, models tend to show over-reliance on local textures ( 21 ; 19 ) , subtle pixel patterns ( 28 ) , other localized cues that are highly inconsistent with holistic human perception ( 16 ; 61 ) , or even ignoring the spatial relationship of visual elements ( 64 ) . While scaling parameters and training data have been shown to induce emergent alignment ( 20 ; 22 ; 37 ) , these evaluations have primarily focused on isolated, simplified tasks such as basic object classification. Consequently, understanding the perceptual gap between modern Vision Language Models (VLMs) and human observers in the context of natural scene perception, which requires integrating multiple objects, spatial contexts, and complex semantic relationships, remains largely underexplored. To investigate this alignment gap, the driving factors behind model decisions need to be understood. However, the scale, complexity, and frequently closed-source nature of modern VLMs have made many traditional "white-box" interpretability techniques impractical. Evaluating the mechanisms of these architectures requires a shift toward black-box, model-agnostic strategies. Causal intervention frameworks ( 58 ; 24 ; 51 ; 14 ) have emerged as a powerful solution; by systematically manipulating input variables, such as ablating specific visual elements, we can directly measure the downstream effects and isolate the causal weight of individual scene components. Building upon this paradigm, we propose Counterfactual Semantic Saliency (CSS), a novel, black-box, and model-agnostic methodology designed to evaluate object importance in scene perception. CSS enables us to systematically investigate Whether modern VLMs perceive and prioritize elements in complex scenes in alignment with human observers. To compare model and human semantic alignment, we constructed a human baseline using complex, multi-object images from a widely adopted human saliency dataset ( 63 ) . Using state-of-the-art generative inpainting with Nano Banana 2 ( 56 ) , we created 307 factual scenes and 1,306 high-fidelity counterfactual variants by systematically ablating 3–6 visual elements per scene. We then conducted a large-scale human psychophysics study, collecting 16,289 valid textual descriptions across all 1,613 images from 227 independent participants. Pervasive gaps between models and human scene perception were found through rigorous statistical analysis. Notably, every evaluated VLM failed to reach human-human consistency levels when predicting the most critical object for scene comprehension or when ranking overall object importance. Furthermore, we identified divergences in the underlying perceptual biases during scene viewing: whereas humans focus more on the person (Person Bias), models heavily rely on bottom-up attributes, including Center Bias, Low-level Saliency Bias, and Size Bias. Among these divergent factors, we isolate exaggerated Size Bias as the dominant factor contributing to the human-AI semantic alignment gap. The primary contributions of this work are as follows: • We introduce Counterfactual Semantic Saliency (CSS), a black-box and model-agnostic methodology to systematically quantify the importance of objects in scene perception, overcoming the limitations of traditional white-box interpretability methods. • We establish a human baseline by conducting a large-scale human psychophysics study, yielding a comprehensive benchmark dataset comprising 16,289 valid responses across 307 factual scenes and 1,306 counterfactual variants from 227 independent participants. • We provide the first systematic quantification of the semantic alignment gap between modern VLMs and human observers in scene comprehension, revealing a pervasive divergence in the preference over low-level attributes and top-down priors between VLMs and humans. • Through robust statistical analysis, we discover the major driving factor of this perceptual gap, demonstrating that an exaggerated Size Bias in modern architectures contribute the most to their divergence from human consensus. 2 Related Works Historically, understanding the decision-making processes of visual models like CNNs relied heavily on white-box methods, ranging from gradient-based ( 49 ) and activation-based ( 4 ) internal analyses to early black-box probing ( 45 ; 41 ) . As Vision Transformers become the dominant vision encoder architecture, visualizing attention maps provides cues about the features that contribute to a model’s output ( 9 ; 36 ; 31 ) . However, the immense complexity of modern VLMs makes these straightforward techniques no longer sufficient to fully capture the true causal features driving a model’s generation ( 30 ; 10 ) . While black-box analytical methods have long existed, previous approaches to complex visual tasks have faced significant obstacles. When altering image inputs for causal intervention, researchers either need to ensure that the modified image quality does not introduce confounding artifacts ( 51 ) , or they have to collect naturally occurring image pairs that were visually similar but semantically distinct ( 57 ) . Although traditional inpainting models could rapidly remove objects from scenes ( 38 ; 47 ; 52 ; 67 ) , the visible traces left behind by these modifications were non-negligible. However, recent advancements in generative AI have made high-quality, flawless, and fine-grained image editing viable ( 44 ; 56 ; 59 ; 62 ) . SOTA closed-source models are capable of generating images that are nearly indistinguishable to human observers, and the maturation of image-to-image generation has enabled highly controllable editing ( 18 ; 48 ; 32 ) . This technological leap serves as the foundational premise for our work. Counterfactual Semantic Saliency (CSS) leverages the most advanced generative models available (Nano Banana 2) to produce high-fidelity counterfactual images, allowing us to systematically verify whether