Attention, Please! PixelSHAP Reveals What Vision-Language Models Actually Focus On
Roni Goldshmidt · 2025
Read Attention, Please! PixelSHAP Reveals What Vision-Language Models Actually Focus On by Roni Goldshmidt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06670.txt.
Introduction
Roni Goldshmidt introduces PixelSHAP, a tool that shows which parts of an image drive a vision-language model's answer. His 2025 paper adapts Shapley analysis, a credit-assignment idea from game theory, to structured visual objects. Knowing what a model looks at matters for trust and debugging in high-stakes uses.
PixelSHAP blurs or removes image objects one at a time and measures how the answer shifts. It needs no access to model internals, so it works with commercial systems too. Tests in self-driving scenes showed the tool exposing what the model truly used. Weaknesses include sensitivity to how objects are segmented and trouble when objects overlap. The code is open source.
Catalog record
| CREATOR | Roni Goldshmidt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06670
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attention, Please! PixelSHAP Reveals What Vision-Language Models Actually Focus On Roni Goldshmidt Abstract Abstract Interpretability in Vision-Language Models (VLMs) is essential for building trust, enabling debugging, and supporting critical decision-making in high-stakes applications. We introduce PixelSHAP , a powerful model-agnostic framework that extends Shapley-based analysis to structured visual entities. While TokenSHAP [ 7 ] successfully applies to text prompts, PixelSHAP advances this approach for vision-based reasoning by systematically analyzing image objects and precisely quantifying their influence on a VLM’s response. PixelSHAP operates solely on input-output pairs without requiring access to model internals, ensuring seamless compatibility with both open-source and commercial models. The framework supports diverse embedding-based similarity metrics and achieves remarkable computational efficiency through innovative optimization techniques derived from TokenSHAP [ 7 ] . We demonstrate PixelSHAP’s exceptional ability to enhance interpretability of complex visual scenes, with potential applications in critical domains such as autonomous driving. Our open-source implementation provides researchers and practitioners with a robust tool for advancing explainable AI in vision-language systems. 1 Introduction Interpretability in Vision-Language Models (VLMs) is essential for trust, debugging, and decision-making, particularly in high-stakes applications such as autonomous driving. Existing techniques primarily focus on token-level or feature-level analysis [ 7 , 18 , 19 ] , but fail to capture spatial object relevance , leaving it unclear which specific objects influenced a model’s response. For example, if a VLM generates the caption ”A pedestrian is crossing the street” for an image with multiple pedestrians, traditional methods cannot identify which specific pedestrian the model focused on. To address this limitation, we introduce PixelSHAP , the first object-level interpretability framework for generative VLMs that extends Shapley-based analysis to structured visual entities. Unlike previous approaches that work only with classification models or require access to model internals, PixelSHAP operates solely on input-output pairs from models that generate open-ended text responses. While TokenSHAP [ 7 ] successfully applies Shapley value estimation to textual prompts, PixelSHAP adapts this concept to visual content by defining image objects as structured feature groups , systematically perturbing them, and quantifying their contribution to a VLM’s response. Figure 1 : Pixel SHAP simulation of a car crash. The left shows the scene, prompt, and GPT-4o response, and the right highlights object importance. PixelSHAP significantly improves computational efficiency by shifting from pixel-level to object-level perturbations. By leveraging segmentation models to generate object masks and applying Shapley-based importance estimation, our approach evaluates only a limited number of meaningful objects rather than thousands of individual pixels, making it computationally tractable while preserving high interpretability fidelity. Contributions Our work introduces PixelSHAP and makes the following key contributions: • Object-Level Explainability for VLMs: We propose PixelSHAP, the first model-agnostic interpretability framework for text-generative VLMs, integrating segmentation-based perturbations with Shapley value estimation. • Visual Attribution: PixelSHAP provides a method for visualizing object contributions, allowing users to analyze model decisions through heatmaps and object masks. • Improved Trustworthiness: By identifying which objects influenced a model’s output, PixelSHAP enhances transparency in vision-language decision-making. • Challenges and Trade-offs: We discuss the computational complexity and sensitivity to segmentation quality associated with object-level perturbation methods. The remainder of this paper is structured as follows: Section 2 reviews existing interpretability methods. Section 3 details the problem statement. Section 4 explains the PixelSHAP algorithm. Section 5 describes our dataset and masking strategies. Section 6 presents empirical evaluations, Section 7 shows qualitative examples, Section 8 discusses implications and future directions, and Section 9 concludes our work. 2 Related Work Interpretability in vision-language models (VLMs) has been studied through various approaches including perturbation-based analysis, gradient-based saliency maps, and Shapley-value estimations. We categorize existing methods into general interpretability, Shapley-based explanations, and vision-specific approaches. 2.1 General Interpretability Methods Feature importance methods like LIME [ 18 ] and Anchors [ 19 ] provide locally interpretable explanations by perturbing inputs, while Grad-CAM [ 20 ] visualizes class-specific saliency maps for convolutional networks. More advanced techniques include RISE [ 14 ] , which uses randomized input sampling, and XRAI [ 22 ] , which groups important pixels into meaningful regions. However, these methods typically don’t account for structured objects within images. 2.2 Shapley-Based Interpretability Shapley values from cooperative game theory [ 21 ] have become valuable for machine learning explainability due to their axiomatic properties. TokenSHAP [ 7 ] applied this approach to language models, quantifying individual token contributions, but doesn’t extend naturally to vision content. FastSHAP [ 8 ] made Shapley estimation computationally feasible for large models, while MM-SHAP [ 13 ] focused on attributing importance between visual and textual modalities using image patches rather than semantically meaningful objects, limiting its utility for visual interpretability. 2.3 Interpretability in Vision-Language Models VLM interpretability remains challenging due to complex multimodal interactions. Methods like Score-CAM [ 23 ] and CLIP-Explainer [ 2 ] generate attention maps to visualize text-image associations. Newer approaches leverage GroundingDINO [ 11 ] and SAM [ 9 ] for object recognition and segmentation. For black-box commercial VLMs, occlusion-based techniques [ 24 ] and gradient-free approaches like D-RISE [ 15 ] and Meaningful Perturbation [ 4 ] have been adapted to identify relevant image regions. 2.4 Object-Level Explainability and Our Contribution Despite progress in interpretability, methods specifically designed for object-level explainability in text-generative VLMs are lacking. The increasing use of segmentation-based explainability [ 5 ] and multimodal reasoning frameworks [ 16 ] highlights this need. While XAtten [ 6 ] proposed a cross-modal attention-based interpretability framework, it remains model-dependent, requiring access to model internals. We introduce PixelSHAP , extending TokenSHAP [ 7 ] to enable object-centric attribution using segmentation models. By systematically masking objects and evaluating their influence on VLM-generated text, PixelSHAP provides a structured understanding of vision-language interactions without requiring access to model internals. Unlike gradient-based methods that rely on activation maps, our approach directly quantifies object importance through systematic perturbation of visual entities, making it the first method to integrate segmentation-based object attribution with Shapley value estimation for text-generative VLMs. 3 Problem Statement Figure 2 : Overview of the PixelSHAP framework. The method systematically perturbs object groups, queries a vision-language model (VLM), and computes Shapley values to quantify object importance. The core challenge in interpreting Vision-Language Models (VLMs) that generate free-form text lies in isolating the contribution of specific visual elements to the model’s output. Unlike classification models that produce discrete probabilities over predefined categories, generative VLMs produce natural language that lacks a direct quantitative link to input features. This introduces a fundamental attribution problem: how do we determine which parts of an image most influenced a particular textual response? Formally, given an image I containing multiple objects \{O_{1},O_{2},...,O_{n}\} , a text prompt p , and a VLM response R=f(I,p) , our goal is to attribute a relative importance score \phi_{i} to each object O_{i} that accurately reflects its influence on R . This attribution problem is challenging for several reasons: 1. Non-Discrete Outputs: Unlike classification models that output probability distributions, text-generative VLMs produce sequences where influence cannot be directly mapped to class probabilities. 2. Semantic Ambiguity: The influence of visual elements on text generation often has semantic dimensions that pixel-level perturbations cannot capture. 3. Black-Box Constraints: Many commercial VLMs provide no access to internal representations, gradients, or attention weights. 4. Computational Complexity: Naive pixel-level perturbation approaches would require an