Analyzing the Sensitivity of Vision Language Models in Visual Question Answering

Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal · 2025

Read Analyzing the Sensitivity of Vision Language Models in Visual Question Answering by Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21335.txt.

Introduction

Monika Shah and co-authors probe how vision-language models handle questions that break the normal rules of polite talk. People cope well when a speaker says too much, too little, or drifts off point, though it costs extra thought. The team ask whether AI models that answer questions about images cope as well, in their 2025 study.

Extra words were added to hand-written questions from a standard image question set. Three top models were tested on the altered questions. Scores fell steadily as the extra wording piled up, unlike the grace people show. The authors say this gap offers a useful way to map where such models still fall short of human talk skills.

Catalog record

CREATORMonika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analyzing the Sensitivity of Vision Language Models in Visual Question Answering Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal Abstract We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of conversation proposed by Grice. Specifically, even when Grice’s maxims of conversation are flouted, humans typically do not have much difficulty in understanding the conversation even though it requires more cognitive effort. Here, we study if VLMs are capable of handling violations to Grice’s maxims in a manner that is similar to humans. Specifically, we add modifiers to human-crafted questions and analyze the response of VLMs to these modifiers. We use three state-of-the-art VLMs in our study, namely, GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-Flash on questions from the VQA v2.0 dataset. Our initial results seem to indicate that the performance of VLMs consistently diminish with the addition of modifiers which indicates our approach as a promising direction to understand the limitations of VLMs. 1 Introduction Vision Language Models (VLMs) Team et al. 2024 ; Liu et al. 2023 ; Hurst et al. 2024 that unify Large Language Models with computer vision have made significant advances in multimodal tasks such as image captioning Yang et al. 2019 ; Cornia et al. 2020 ; Wang et al. 2022 and visual question answering (VQA) Antol et al. 2015 . However, we are just beginning to understand the reasoning capabilities and more importantly, the limitations of these models Campbell et al. 2024 . In this work, inspired by theories from cognitive science, we understand the behavior of VLMs in VQA when we increase the cognitive load in comprehending questions . Specifically, in Grice’s classical theory of cooperative principles Grice 1975 , it is known that humans acting cooperatively in a conversation typically need to follow a set of rules commonly known as Grice’s maxims . These maxims make conversation more effective and ensure efficient communication. However, it is known from previous studies that even when these maxims are violated, humans can comprehend conversation easily Davies 2000 . However, violations to Grice’s maxims places greater cognitive burden on the listener Jacquet et al. 2018 . In this work, we study how VLMs react when Grice’s maxims are violated. Specifically, we treat VQA as a single utterance conversation where a human is asking the AI model a question to which the AI model responds with an answer. We introduce modifiers into human-crafted questions that adds greater detail to a question. At the same time, these details typically tend to violate Grice’s maxims since they were not deemed to be essential when a human crafted the original question. While an AI model could benefit from the added information, processing modifiers will increase the reasoning required to answer the question. We add two types of modifiers, namely, visual and relational modifiers. The visual modifiers add more detail related to visual properties such as color, shape, etc., while relational modifiers add details related to spatial relationships. We use VLMs to generate a modified question with either visual or relational modifiers. Next, we verify if the modified question changes human perception. That is, if humans can answer the modified question with an answer that is equivalent to the answer to an unmodified question, this implies that the modifier does not alter the question. Therefore, we would expect a VLM to be able to do a similar type of reasoning. We evaluate this on three state-of-the-art VLMs, GPT-4o OpenAI 2024 , Gemini-1.5-Flash Team et al. 2024 and Claude-3.5-Sonnet Anthropic 2024 on the VQA v2.0 dataset. That is, we generate modified questions from each of these VLMs and evaluate the responses of each VLM to the modified questions. Our initial results seem to indicate that VLMs are sensitive to modifications to questions. In particular, we find that there is a consistent performance degradation in the presence of modifiers. In particular, when modifiers are added through Gemini-1.5-Flash, the performance degradation is more significant in all 3 VLMs. 2 Related Work Following the original VQA task Antol et al. 2015 , several improved datasets for VQA have been developed Goyal et al. 2017 ; Selvaraju et al. 2020 ; Tan and Bansal 2019 to evaluate VQA systems. More recently, the trend has shifted towards incorporating LLMs within the evaluation process. For instance, Zhou et al. 2023 uses ChatGPT to automatically evaluate model outputs on a Likert scale. The work in Mañas et al. 2024 leverages LLMs to evaluate answers. Specifically, it formulates VQA as an answer-rating task where the LLM (Flan-T5 Chung et al. 2024 , Vicuna-v1.3 Chiang et al. 2023 and GPT-3.5-Turbo) is instructed to score the correctness of a candidate answer given a set of reference answers. The work in Britton et al. 2022 is related to our approach where it adds question modifiers to VQA and analyzes its effect on LXMERT Tan and Bansal 2019 . However, there has not been a significant amount of work that relates the reasoning of VLMs in VQA grounded in principles of human cognition which is the direction we follow in this work. 3 Pragmatics in Visual Question Answering Figure 1: Original question of the green is the question that satisfies the Grice’s maxim and the questions with modifiers that violates the Grice’s maxim. Grice’s classical theory of cooperative principles in pragmatics is widely used to characterize human conversation. Specifically, Grice developed principles that explain effective conversation between participants assuming that the participants have a common goal of understanding each other and therefore act cooperatively. These principles are summarized in four maxims, namely, the maxim of quality, quantity, relation and manner. The maximum of quality suggests that speakers should be as truthful as possible and only say what they believe to be true based on evidence. The maxim of quantity suggests that the right amount of information must be provided in a conversation, i.e., one should not add too much or too little information. The maxim of relation suggests that a speaker should stay relevant to the topic and the maxim of manner suggests the need to avoid ambiguity and focus on clarity. While Grice’s maxims characterize effective conversation, violation of Grice’s maxims does not mean that the conversation is incomprehensible Davies 2000 . Specifically, since participants are assumed to be acting cooperatively, if a speaker violates a maxim, then the burden of understanding falls on the listener. That is, the listener is expected to work harder (cognitively) to comprehend the intention behind utterances that violate the maxims. We use this principle as a way to understand the limitations of VLMs. Specifically, we think of the VQA task as a conversation that involves a single utterance between two participants, i.e., one participant is a human who asks a question to the AI model and the other participant is the AI model that needs to generate an answer. In cases where the human participant flouts Grice’s maxims, there is an increased burden of understanding on the AI model to produce an answer that the human can agree on. Figure 2: Illustrating our workflow. We generate modified questions from human-crafted questions using a

Read the complete text