Visuospatial Perspective Taking in Multimodal Language Models

Jonathan Prunty 1 Seraphina Zhang 1,2 Patrick Quinn 1 Jianxun Lian 3 Xing Xie 3 Lucy Cheke 2 1 Leverhulme Centre for the Future of Intelligence, University of Cambridge 2 Department of Psychology, University of Cambridge 3 Microsoft Research Asia · 2026

Read Visuospatial Perspective Taking in Multimodal Language Models by Jonathan Prunty 1 Seraphina Zhang 1,2 Patrick Quinn 1 Jianxun Lian 3 Xing Xie 3 Lucy Cheke 2 1 Leverhulme Centre for the Future of Intelligence, University of Cambridge 2 Department of Psychology, University of Cambridge 3 Microsoft Research Asia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23510.txt.

Introduction

Prunty and five co-authors test whether multimodal models can take another person's viewpoint. They adapt two classic human tasks for machines. The Director Task checks viewpoint use in a talk-based setting. The Rotating Figure Task probes viewpoint shifts across angles. Models do fine at some levels but fail at Level 2. That level asks them to drop their own view and adopt someone else's. The gap matters for teamwork settings, where partners see scenes from unlike spots.

Catalog record

CREATORJonathan Prunty 1 Seraphina Zhang 1,2 Patrick Quinn 1 Jianxun Lian 3 Xing Xie 3 Lucy Cheke 2 1 Leverhulme Centre for the Future of Intelligence, University of Cambridge 2 Department of Psychology, University of Cambridge 3 Microsoft Research Asia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90696
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visuospatial Perspective Taking in Multimodal Language Models Jonathan Prunty 1 Seraphina Zhang 1,2 Patrick Quinn 1 Jianxun Lian 3 Xing Xie 3 Lucy Cheke 2 1 Leverhulme Centre for the Future of Intelligence, University of Cambridge 2 Department of Psychology, University of Cambridge 3 Microsoft Research Asia Abstract As multimodal language models (MLMs) are increasingly used in social and collaborative settings, it is crucial to evaluate their perspective-taking abilities. Existing benchmarks largely rely on text-based vignettes or static scene understanding, leaving visuospatial perspective-taking (VPT) underexplored. We adapt two evaluation tasks from human studies: the Director Task, assessing VPT in a referential communication paradigm, and the Rotating Figure Task, probing perspective-taking across angular disparities. Across tasks, MLMs show pronounced deficits in Level 2 VPT, which requires inhibiting one’s own perspective to adopt another’s. These results expose critical limitations in current MLMs’ ability to represent and reason about alternative perspectives, with implications for their use in collaborative contexts. Keywords Artificial Intelligence \cdot Social Cognition \cdot Theory of Mind \cdot Spatial Cognition \cdot Language Understanding 1 Introduction Generative models are becoming embedded within a wide range of social domains. They are rapidly taking on roles as teachers, colleagues, therapists, friends, and romantic partners [ 48 , 7 ] . Given this widespread adoption, it is important to prioritise the development of reliable methods for understanding the strengths and limitations of their social cognition, so as to be able to predict safe and reliable contexts of use, and avoid dangerous or costly failure cases [ 34 , 9 , 5 ] . One core feature of human social cognition is Theory of Mind (ToM): the capacity to reason about, explain, and predict others’ behaviour based on their beliefs, intentions, and emotions [ 41 , 3 , 55 ] . Given its foundational role in human social interactions [ 15 , 16 ] , there is growing debate about whether and to what extent such capabilities exist in artificial systems, and how they should be measured [ 23 ] . While standardised ToM tests exist within cognitive science, many require participants to read short stories and reason about the mental states of characters [ 21 , 4 ] . These vignette-style tasks are easily adaptable for language models, but may not reliably assess ToM capabilities in such systems. LLMs have encountered many similar story-based examples in their training corpora and can often achieve high scores by exploiting shallow associations rather than understanding underlying causal factors [ 35 , 22 ] . Moreover, the condensed format of these stories removes much of the inferential challenge present in real social situations. Consequently, performance on vignette tasks does not always generalise to novel, naturalistic or atypical scenarios [ 53 , 27 , 47 ] . The advent of multimodal capabilities within language models enables evaluations using images and other sensory input, particularly paradigms from developmental and comparative psychology [ 13 , 54 ] . While these often test earlier-developing ToM capabilities than adult-targeted vignette tasks, they are not inherently “simpler” – indeed, they likely represent ToM challenges as experienced in real life more faithfully. In natural scenarios, our reasoning about others’ minds is informed by the context in which we are situated. Relevant information is not explicitly stated in a narrative (“Elsa saw the cat”) but must be inferred from behaviour and context (e.g., perceptual input showing Elsa’s eyes pointing toward a cat with no occluder between them). Visuospatial perspective taking (VPT) – the ability to represent what someone sees and how they see it – is among the most studied and well-understood socio-cognitive capabilities in developmental psychology. It is therefore an excellent candidate for assessing AI systems if we aim to move beyond narrow linguistic framings of Theory of Mind and generate more robust predictions about real-world AI performance. 1.1 Visuospatial perspective taking The ability to inhibit our own viewpoint and represent a scene from another’s perspective is a core component of human social interactions, and is related to a range of social behaviours such as shared attention, referential communication and joint action [ 16 , 52 , 6 , 44 ] . For instance, by adopting someone’s viewpoint, we can understand what they might know about their surroundings, what they might want, or what they might be referring to when speaking. Developmental psychologists have shown that this capability can be divided into two levels [ 11 , 12 , 28 , 43 ] as children can determine whether a person is seeing something (Level 1) earlier in development than they can determine how it might appear to them (Level 2). These two capability levels can also be distinguished in adults. That is, Level 1 perspective taking judgements about whether a person can see something or not are made rapidly and spontaneously through relatively simple line-of-sight computations [ 45 , 38 ] . By contrast, Level 2 perspective taking, involving judgements about how something would appear to another person, is effortful and requires inhibiting one’s own egocentric perspective and mentally rotating visual information to adopt the other person’s viewpoint [ 50 , 51 , 49 , 26 , 8 , 36 ] . Neuroimaging evidence also suggests the two levels are distinct, with Level 2 (but not Level 1) VPT tasks activating the right temporo-parietal junction (rTPJ) – a brain region involved in distinguishing between self and other, inhibiting the egocentric perspective, and attributing mental states to others [ 33 , 39 , 46 ] . As well as making visual judgements about others’ perspectives, VPT also involves making spatial judgments about where something is located from that person’s viewpoint [ 50 , 25 , 43 ] . For spatial content, judging whether something is in front or behind someone is straightforward to solve by following their line of sight (Level 1), but judging whether something is to the left or right of someone is more effortful if they do not share our spatial perspective, requiring egocentric inhibition and mental rotation (Level 2). Both visual and spatial components of VPT are used in combination when interpreting referential communication [ 26 , 10 ] . For instance, when disambiguating the phrase “Can you pass me the jar on the left?”, identifying which jar the speaker is referring to depends on whether you share the same visual perspective (as they would not be referring to a jar that only you can see) and also whether you share the same spatial perspective (if the speaker was standing opposite, their left would be your right). 1.2 Visuospatial perspective taking in multimodal language models Recent advances have enabled language models to process multiple input modalities [ 2 , 24 , 32 ] , supporting their deployment in interactive multimodal settings such as instruction following, embodied reasoning, and collaborative task execution (see 57 for a recent survey). Accordingly, growing attention has focused on whether multimodal language models (MLMs) can support visuospatial perspective taking (VPT). Early investigations adapted Piaget’s Three Mountains Task [ 40 ] to assess Level 1 and Level 2 VPT [ 31 , 18 ] . While foundational, these benchmarks emphasise recognition of static scene layouts rather than the dynamic egocentric-to-allocentric transformations required for real-world interaction. More recent work has introduced

Read the complete text