How Well Can Vision-Language Models Understand Humans’ Intention? An Open-ended Theory of Mind Question Evaluation Benchmark
Ximing Wen, Mallika Mainali, Anik Sen · 2025
Read How Well Can Vision-Language Models Understand Humans’ Intention? An Open-ended Theory of Mind Question Evaluation Benchmark by Ximing Wen, Mallika Mainali, Anik Sen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22093.txt.
Introduction
Ximing Wen, Mallika Mainali, and Anik Sen test whether vision-language models can read human intentions from images. Their 2025 paper builds an open-ended benchmark for theory of mind, the skill of inferring beliefs and mental states. Thirty annotated images anchor the question set.
Four models of varied size take the test. The largest model leads overall, while one smaller model nearly matches it. Models stumble on tangled scenes like bullying or cheating, where intent hides under social nuance. Smaller models sometimes land the right intent from the wrong visual cues, a lucky hit that masks weak grounding. The authors release the dataset publicly. Their findings warn that fluent answers can hide shallow social reading in current models.
Catalog record
| CREATOR | Ximing Wen, Mallika Mainali, Anik Sen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22093
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Well Can Vision-Language Models Understand Humans’ Intention? An Open-ended Theory of Mind Question Evaluation Benchmark Ximing Wen, Mallika Mainali, Anik Sen Abstract Vision Language Models (VLMs) have demonstrated strong reasoning capabilities in Visual Question Answering (VQA) tasks; however, their ability to perform Theory of Mind (ToM) tasks, such as inferring human intentions, beliefs, and mental states, remains underexplored. We propose an open-ended question framework to evaluate VLMs’ performance across diverse categories of ToM tasks. We curated and annotated a benchmark dataset of 30 images and evaluated the performance of four VLMs of varying sizes. Our results show that the GPT-4 model outperformed all the others, with only one smaller model, GPT-4o-mini, achieving comparable performance. We observed that VLMs often struggle to infer intentions in complex scenarios such as bullying or cheating. Our findings reveal that smaller models can sometimes infer correct intentions despite relying on incorrect visual cues. The dataset is available at https://github.com/ximingwen/ToM-AAAI25-Multimodal . Introduction Understanding human intentions through visual cues is a fundamental aspect of social intelligence, allowing effective communication, collaboration, and interaction [ 2 ] . This capability, often referred to as the Theory of Mind (ToM), involves the ability to infer the beliefs, desires, and intentions of others based on observable behaviors and environmental contexts [ 9 , 7 , 12 ] . Recent advances in VLMs have demonstrated impressive abilities in multimodal reasoning, combining visual and textual information to perform complex tasks [ 5 , 10 , 13 ] . However, their capability to perform ToM-like reasoning, specifically in interpreting intentions from visual cues, remains underexplored. For example, Etesam et al. 2023 only investigate the emotional component of ToM, instead of exploring more broad categories such as intentions, religions, etc. Jin et al. 2024 frame the ToM task as a binary choice question, without requiring VLMs to engage in open-ended reasoning. Consequently, this approach may not fully capture the VLMs’ capability to perform ToM tasks. To further highlight, ToM tasks present unique challenges for VLMs, requiring both visual feature extraction and contextual reasoning to infer hidden mental states. Thus, our study, which evaluates VLM performance on ToM tasks through an open-ended question framework, is pivotal to assessing VLMs’ capacity for advanced multimodal understanding and social intelligence. Open-ended Question Framework In this study, we aim to investigate the capability of VLMs to perform ToM tasks by testing their ability to interpret intentions based on visual cues in images. To fully evaluate whether VLMs truly understand humans’ intentions, we proposed an open-ended question framework composing the following three research questions: • Q1: How effectively can VLMs identify human intentions in visual scenarios? • Q2: Can VLMs recognize accurate visual cues and use them to perform ToM tasks? • Q3: Can VLMs comprehend human intentions sufficiently to make reasonable future inferences? Q1 focuses on inferring individuals’ mental states and intentions, a core ToM skill. Q2 examines the model’s ability to identify and articulate visual cues, linking observations to inferred mental states. Q3 evaluates predictive reasoning asking the model to infer potential future actions or events based on the scene. By designing tasks that require inferring the purpose or mental state of individuals depicted in diverse scenarios, we seek to evaluate the extent to which models align with human-like reasoning in visual intention understanding. Our findings contribute to research on VLMs by highlighting their strengths and limitations in approximating human cognition, paving the way for socially aware AI advancements. Data Development Data Collection We defined 30 scenarios based on two intention categories (emotion-based and action-based) and images sourced from platforms including iStock, Shutterstock, Unsplash, and Pexels under appropriate licenses to ensure copyright compliance. We only included images that conveyed clear intentions with measurable visual cues, such as facial expressions, body language, interaction with objects, and eye gaze that indicated the mental states and intentions of the individuals. Each image underwent a comprehensive review to ensure suitability for research objectives and images with ambiguous cues were excluded. The final dataset provides diverse and suitable content for research. An overview of the dataset is shown in Figure 1 . Figure 1 : An overview of the three samples used for this work to evaluate Theory of Mind (ToM) capabilities of three vision language models (VLMs). This preview shows the intention generated using two VLMs: LLaVA (7B) and GPT-4, along with the human-annotated intention. Data Annotation Each author annotated a subset of 10 images, providing detailed descriptions in three categories: intention, visual cues, and future inference. A third-party evaluator reviewed and validated the 30 annotations to ensure consistency and accuracy, confirming that they accurately captured intentional actions, visual cues, and potential future inferences. Experimental Design Task We designed a structured prompt to generate responses from VLMs that aligned with the objectives of this study. The prompt is as follows: "Based on the given image, answer the following in one sentence each: (1) What do you think is the intention, mental state, feeling or belief of each person in the image? (2) What visual cues in the image helped you determine what people might be thinking or feeling? (3) Can you infer what might happen next?" VLMs were expected to extract the intentions from the image, recognize visual cues that support the inferred intention, and generate plausible future scenario descriptions consistent with the context of the image. Models We evaluated the performance of four VLMs - GPT-4 [ 1 ] , GPT-4o-mini (8B parameters) [ 11 ] , Deepseek v1 (7B parameters) [ 3 ] , and LLaVA (7B parameters) [ 8 ] -in inferring ToM components from images. Evaluation Model responses were manually compared with human annotations using a scoring system based on keyword relevance and accuracy. A score of 1 was given for responses with correct or synonymous keywords that accurately described the context. Partially correct responses were given a score of 0.5. Smaller models, such as DeepSeek, often identified intentions correctly, but struggled with scenario details, such as misidentifying gender or objects. While object recognition errors were ignored for the ‘intention’ category, these inaccuracies were given a score of 0.5 in the ‘visual cues’ category. Responses without relevant keywords were given a score of 0. Result and Discussion We assessed the performance of VLMs across three ToM tasks using our metric. The results, presented in Table 1 , display the accuracy scores out of 30 for each category. Accuracy across three ToM tasks Among the four models tested, GPT-4 performed the best on all tasks. Despite being a smaller model, GPT-4o-mini had comparable scores. In contrast, LLaVA-7B achieved significantly lower scores, indicating its limited ability to interpret subtle visual cues and make accurate inferences. Deepseek v1-7B outperformed LLaVA-7B but underperformed compared to GPT-based models. VLM Intention Visual Cue Future Inf. GPT4 27 27 28 GPT4o-mini 27.5 27 27.5 LLaVA-7B 7.5 8.5 7 Deepseek-7B 17 16.5 16 Table 1 : Performance of VLMs’ responses for inferring intention, visual cues, and future inference What types of human intentions can VLMs recognize? We found that GPT-based models could identify a range of human intentions, such as determination, care, frustration, compassion, praying, and bullying. Deepseek v1-7B could recognize some intentions, but struggled with subtle ones such as emotional distress, frustration, and bullying. LLaVa-7B