Tinted Frames: Question Framing Blinds Vision-Language Models
Wan-Cyuan Fan, Jiayun Luo †, Declan Kutscher †, Leonid Sigal, Ritwik Gupta · 2026
Read Tinted Frames: Question Framing Blinds Vision-Language Models by Wan-Cyuan Fan, Jiayun Luo †, Declan Kutscher †, Leonid Sigal, Ritwik Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19203.txt.
Introduction
Fan and co-authors show that the wording of a question can blind vision-language models to the image. Even when two wordings need the same visual work, closed forms draw far less attention to the picture. Multiple choice and yes-no prompts shift focus away from key regions. The team traces most of the accuracy loss to this misplaced attention. They add a light prompt-tuning fix with learned tokens. It restores the stronger gaze seen with open prompts and lifts scores across forms.
Catalog record
| CREATOR | Wan-Cyuan Fan, Jiayun Luo †, Declan Kutscher †, Leonid Sigal, Ritwik Gupta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60859 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tinted Frames: Question Framing Blinds Vision-Language Models Wan-Cyuan Fan, Jiayun Luo †, Declan Kutscher †, Leonid Sigal, Ritwik Gupta Abstract Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings. † † footnotetext: † Equal contribution. Corresponding author [email protected] . ⋆ Project Page 1 Introduction Complex, multi-modal reasoning tasks have been the driving force behind contemporary vision-language model (VLM) development. As these models tackle increasingly difficult real-world datasets, it is critical that their reasoning and responses are appropriately grounded in visual evidence. However, despite their impressive performance on simple benchmarks, recent research has revealed that the visual capability of these systems is largely a function of text priors and biases. VLMs are “blind” and exhibit distinct failures in visual grounding, raising fundamental questions about whether they are reasoning over the image or merely leveraging powerful language priors to generate plausible answers. Recent work characterizes these issues as a problem of visual disengagement and structural bias. Studies [ 40 , 32 ] have shown that VLMs assign little attention to visual tokens, generating responses driven primarily by textual context rather than visual evidence. This lack of attention is not uniformly distributed. Models frequently allocate disproportionately high attention weights to visual attention sink tokens [ 21 , 29 , 20 ] , semantically meaningless background tokens, while also exhibiting severe spatial biases. For instance, artifacts from positional encodings (i.e., RoPE [ 37 ] ) and causal attention mechanisms can create effective “blind spots” [ 51 , 39 , 42 ] , resulting in the neglect of specific image regions regardless of their semantic importance. However, these analyses have typically been done holistically, averaging observations across heterogeneous benchmarks. This perspective implies that such blindness is a static, inherent flaw of the model architecture. While there is ample evidence that prompting impacts model accuracy [ 15 , 34 , 25 ] , there is little evidence that suggests that visual perception process itself is impacted by this, or that a simple question framing can induce such behavior. Figure 1 : VLM grounding changes as a function of question framing. Attention maps reveal that while the model actively attends to the target object (the chair) during open-ended generation, it exhibits disengagement and misallocation when the same question is posed as a Yes/No or MCQ task. Note that we employ attention rollout rather than averaging attention weights across layers and token. By recursively rolling out attention matrices, we trace information propagation from inputs of early layers to output embeddings. Qwen2.5-VL-7B is used for the visualization. The top 3 tokens with the highest attention are highlighted in red boxes, and the minimum and maximum values of the linear colormap are set to the same value for all images. In this work, we demonstrate that VLMs are selectively blind. They decide how much to look at an image based on the textual framing of the question—such as open-ended, Yes/No, or multiple-choice—despite different framings requiring the same visual concepts to answer correctly. We study this phenomenon mechanistically, utilizing attention rollout [ 2 ] to estimate the visual information propagation and visualize the attention map where the model attends when making output decisions. We hypothesize that alternative framings impact model performance indirectly via deviations in visual attention. Since open-ended, Yes/No, and multiple-choice question (MCQ) are the three dominant framings used to evaluate VLMs across major benchmarks, this framing-dependent behavior has direct implications for how we assess model capabilities. We conduct a three-part analysis. First we establish and quantify the impact of framing on the accuracy by introducing cross-framing inconsistency as a diagnostic measure. By posing semantically equivalent questions across all formats, we find that models which correctly answer open-ended questions frequently fail their constrained counterparts, especially for tasks involving object grounding. Second, we find that framing impacts both amount and distribution of attention. We find that constrained framings trigger a shift in visual attention strategy, reducing overall attention on the image and redirecting attention away from task-relevant regions. Finally, through intervention on attention, we confirm that indeed the impact of the framing on the accuracy is induced by the shift in visual attention. Armed with these findings, we propose a lightweight prompt-tuning mitigation strategy that learns a small set of soft tokens to realign the visual attention of constrained framings to match the robust patterns of open-ended setting. Our mitigation strategy restores visual grounding and yields consistent improvements across multiple models and benchmarks without modifying model weights. 2 Related Work Vision-language models (VLMs) [ 22 , 5 , 48 , 30 , 38 , 1 , 50 ] have rapidly advanced from simple captioning to complex multimodal reasoning, but how do we know they truly understand what they see? Since contemporary VLMs output free-form language, it is difficult to disentangle visual understanding and reasoning from linguistic shortcuts [ 26 ] . Answering this question requires looking beyond output accuracy, into whether these models truly ground their reasoning in what they see, and what factors might cause that grounding to break down. Visual Grounding in VLMs. Visual grounding, the ability to localize textual concepts within an image, has been long-standing goal in computer vision. Classical architectures such as object detectors [ 33 , 7 , 17 ] are explicitly trained to produce spatial localizations. More recently, vision transformers have been shown to develop interpretable attention patterns that correlate with object boundaries [ 8 ] . VLMs embed these vision encoders but are trained end-to-end for language generation. Therefore, spatial grounding is an implicit learning task as opposed to a primary objective. Yet, recent works [ 21 , 13 ] have shown that this implicit grounding is often unreliable. VLMs can produce correct answers while attending to irrelevant regions, suggesting that strong benchmark performance does not guarantee genuine visual understanding. In this work, we establish a mechanistic link between question framing, visual attention, and output quality. Visual Disengagement and Bias in VLMs. A growing amount of work documents visual shortcomings of VLMs. Studies [ 49 , 40 ] have shown that these models often allocate much lower attention to visual content than to textual ones, potentially generating responses driven by language priors rather than visual evidence. Others [ 21 , 29 ] have identified that models disproportionately attend to semantically meaningless visual tokens when performing visual reasoning tasks, further diluting visual engagement on the area of interest. Beyond