Can Vision Language Models Infer Human Gaze Direction? A Controlled Study
Zory Zhang Brown University, Pinyuan Feng Columbia University Bingyang Wang Emory University Tianwei Zhao Johns Hopkins University Suyang Yu University of Washington Qingying Gao Johns Hopkins University, Hokin Deng Carnegie Mellon University, Ziqiao Ma University of Michigan, Yijiang Li University · 2025
Read Can Vision Language Models Infer Human Gaze Direction? A Controlled Study by Zory Zhang Brown University, Pinyuan Feng Columbia University Bingyang Wang Emory University Tianwei Zhao Johns Hopkins University Suyang Yu University of Washington Qingying Gao Johns Hopkins University, Hokin Deng Carnegie Mellon University, Ziqiao Ma University of Michigan, Yijiang Li University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05412.txt.
Introduction
Zory Zhang and co-authors test whether vision-language models can tell where a person is looking, in a 2025 paper. Gaze is a basic social cue, and the team photographed real scenes for the test: 1,360 images of a person gazing at one of several objects on a table. Crucially, they sometimes pointed the person's head at the target, sometimes at a decoy, and sometimes left it free.
The models lagged far behind human judges. The gap was not about image resolution or naming the objects. The models were mostly reading head direction instead of the eyes, so a turned head fooled them. A small fine-tuning experiment suggested the bias comes from training data rather than the architecture itself. Fixing gaze reading would help machines join conversations more naturally.
Catalog record
| CREATOR | Zory Zhang Brown University, Pinyuan Feng Columbia University Bingyang Wang Emory University Tianwei Zhao Johns Hopkins University Suyang Yu University of Washington Qingying Gao Johns Hopkins University, Hokin Deng Carnegie Mellon University, Ziqiao Ma University of Michigan, Yijiang Li University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05412
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Vision Language Models Infer Human Gaze Direction? A Controlled Study Zory Zhang Brown University, Pinyuan Feng Columbia University Bingyang Wang Emory University Tianwei Zhao Johns Hopkins University Suyang Yu University of Washington Qingying Gao Johns Hopkins University, Hokin Deng Carnegie Mellon University, Ziqiao Ma University of Michigan, Yijiang Li University of California, San Diego, Dezhi Luo University of MichiganGrowAI https://growing-ai-like-a-child.github.io/ Project Page | Data | Code Abstract Gaze-referential inference—the ability to infer what others are looking at—is a critical component of a theory of mind that underpins natural human-AI interaction. In a controlled study, we evaluated this skill across 111 Vision Language Models (VLMs) using photos taken with manipulated difficulty and variability, comparing performance with that of human participants (N = 65), and analyzed behaviors using mixed-effects models. We found that 94 of the 111 VLMs failed to do better than random guessing, while humans achieved near-ceiling accuracy. VLMs even respond with each choice almost equally frequently. Are they randomly guessing? Although most VLMs struggle, when we zoom in on five of the top-tier VLMs with above-chance performance, we find that their performance declined with increasing task difficulty but varied only slightly across different prompts and scene objects. These behavioral features cannot be explained by considering them as random guessers. Instead, they likely use a combination of heuristics and guessing such that their performance is subject to the task difficulty but robust to perceptual variations. This suggests that VLMs, lacking gaze inference capability, have yet to become technologies that can naturally interact with humans, but the potential remains. Figure 1 : (a) The gaze referential inference task. (b) 99.9% confidence intervals of the accuracy means are depicted. A random-guessing machine achieves an accuracy of around 42%. A performance gap exists between top-tier Vision Language Models and humans. 1 Introduction Theory of Mind (ToM) is our ability to attribute mental states (e.g., intention, desire, belief, and goals) to ourselves and others ( Premack and Woodruff, 1978 ) . It underpins our capacity to understand others, engage in natural interactions, and collaborate effectively. As recent advances in language modeling are being rapidly integrated into every aspect of human society, their ability to interact meaningfully with humans depends on possessing a machine form of ToM ( Ma et al., 2023 ) . While early exploration in text-based technologies has yielded intriguing findings (e.g., Kosinski, 2024 ), recent advances in generative Vision-Language Models (VLMs, e.g., Gemini et al., 2023 ; Liu et al., 2023 ), which process both visual and textual inputs, offer promising hosts for a more capable ToM with natural interactions. A key challenge today is uncovering the extent to which VLMs have a ToM, and if not much, to determine whether they are on the right track of improving ToM as scaling continues. However, most evaluations are not adequate in furthering understandings of the inference behind the scores (e.g., Kosinski, 2024 , but see Pi et al., 2025 ), and VLMs’ use of superficial shortcuts is hard to control without access to their training data, echoing Sap et al., 2022 . Interpreting gaze, a skill humans are uncannily good at, can illuminate this challenge, as it serves as one of the basic elements that more complex ToM builds on. Gaze is a window into other people’s unconscious knowledge: their visual focus ( Posner and Petersen, 1990 ) , conversational referents ( Prasov and Chai, 2008 ; Qian et al., 2023 ) , linguistic knowledge ( Golinkoff et al., 1987 ) , intention to speak during conversation turn takings ( Kendrick et al., 2023 ) , intended motor actions ( Land, 2006 ) , and immediate intentions and desire in general ( Mikulincer et al., 2014 ) . Gaze is also one of the starting points where infants begin to generalize their experience (with open and closed eyes) to other social agents, which allows them to begin understanding others who are “like them” ( Mikulincer et al., 2014 ) . All of these are critical ingredients of ToM. More broadly, gaze inference likely bootstraps our later social learning, cognitive development ( Csibra and Gergely, 2009 ) , and possibly language acquisition ( Brooks and Meltzoff, 2005 , but also see Sander et al., 2024 ), all of which furthers our grasp of ToM. Considering that gaze inference constitutes the content and helps the development of a ToM, the pressing question is whether VLMs can pick up gaze cues like humans do. If they cannot, they are equipped with a weaker ToM that will leave them a harder time understanding humans and interacting and collaborating with humans. Investigating this will help build a more nuanced understanding of machine ToM in VLMs. Instead of building a general benchmark of gaze understanding, we aim to characterize VLMs’ behavioral patterns through a controlled study to constrain hypotheses of their underlying inference. We formulate the basic task as inferring the object that falls along gaze direction (“line of sight,” as shown in Fig. 1 ), one that more sophisticated gaze understanding builds on. We controlled variables that might affect the task difficulty and potential sources of variability in performance. We took 900 photos of a scene with an actor (as the gazer) by a table and objects on the table, covering multiple photo-shooting views, gazers, proximities between objects, and the combinations of objects with different visual contrast in the scene. See Fig. 2 for examples of the pictures. Then, we asked VLMs and humans (N=65) to choose which object the person in the photo is looking at from a set of options and inferred behavioral patterns from their performance. Out of 111 VLMs, 94 performed about as well as if they had guessed randomly without looking at the images. Even worse, they responded with every possible option almost equally often, which is consistent with the account that they chose randomly from the given options. On the other hand, human participants showed near-ceiling accuracy, as depicted in Fig. 1 . Still, as we took a closer look at five chosen top-level VLMs, we found patterns that nevertheless cannot be explained by labeling them as mere “random guessers” or “approximate retrievers” ( Kambhampati, 2024 ) , which will predict either a chance-level performance, or their performance only varies with the superficial familiarity of the context relative to their “database.” Instead, they performed well above chance (all p<.002 ), and their performance declined with increasing task difficulty but varied only slightly with the specific prompt phrasing and the specific objects present in the scene. These findings suggest that the underlying inference of the top-tier models is imperfect but approaches the gaze inference task in a way that is affected by the difficulty (rather than familiarity), and the inference is robust to perceptual variations, such as object visual features, sizes, and background distractors. We hypothesized that VLMs may rely solely on head direction without eye direction to perform this task—an observation that opens avenues for further investigation. 2 Related Work Gaze Estimation and Following It is helpful to distinguish gaze direction estimation and gaze following. Estimation involves inferring the direction of gaze, while gaze following is the phenomenon that infants look where someone else is looking ( Mikulincer et al.,