Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models
Bridget Leonard, Scott O. Murray · 2026
Read Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models by Bridget Leonard, Scott O. Murray (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16378.txt.
Introduction
Bridget Leonard and Scott Murray use mind based tokens to fix self view bias in joint models. Their 2026 preprint deals with models that join pics and words. Such models do well on sense tasks that join the two. They fail on space tasks that need the view of an other. That fail is a self view bias that will not fade with scale.
Tokens drawn from mind study aim to break that bias. The tokens push the model to hold an other view in mind. Gains would show that form cues can fix space thought. Space thought counts for robots and aid tools in real rooms. The work tests a low cost fix path.
Catalog record
| CREATOR | Bridget Leonard, Scott O. Murray |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models Bridget Leonard, Scott O. Murray Abstract Multimodal language models (MLMs) perform well on semantic vision–language tasks but fail at spatial reasoning that requires adopting another agent’s visual perspective. These errors reflect a persistent egocentric bias and raise questions about whether current models support allocentric reasoning. Inspired by human spatial cognition, we introduce perspective tokens , specialized embeddings that encode orientation through either (1) embodied body-keypoint cues or (2) abstract representations supporting mental rotation. Integrating these tokens into LLaVA-1.5-13B yields performance on level-2 visual perspective-taking tasks. Across synthetic and naturalistic benchmarks (Isle Bricks V2, COCO, 3DSRBench), perspective tokens improve accuracy, with rotation-based tokens generalizing to non-human reference agents. Representational analyses reveal that fine-tuning enhances latent orientation sensitivity already present in the base model, suggesting that MLMs contain precursors of allocentric reasoning but lack appropriate internal structure. Overall, embedding cognitively grounded spatial structure directly into token space provides a lightweight, model-agnostic mechanism for perspective-taking and more human-like spatial reasoning. Keywords Perspective-Taking \cdot Multimodal Models \cdot Spatial Reasoning \cdot Egocentric Bias Introduction Spatial reasoning is a core process in human cognition. Engaging spatial reasoning often involves making perspective-based decisions, such as judging how others perceive the world or how our own perception changes as we move through space. A critical component of such reasoning is visual perspective-taking (VPT) — the ability to simulate a visual perspective other than one’s own — which plays a central role in social interaction and spatial navigation. If multimodal language models (MLMs) are to support perspective-based spatial reasoning, they must likewise be able to represent and operate over non-egocentric viewpoints. However, despite achieving near-human, and often superhuman, performance on many vision-language tasks, MLMs consistently fail at perspective-based reasoning, instead defaulting to an egocentric perspective ( 22 ; 23 ; 9 ; 4 ; 3 ) . This failure is particularly striking given that an egocentric bias is a well-established feature of human cognition, but one that humans can systematically overcome through development and experience. Understanding how and when humans overcome this bias offers a unique opportunity to investigate MLM capabilities through the lens of cognitive development. In the human developmental literature, VPT is divided into two levels that emerge sequentially. While level 1 VPT, understanding what another person can or cannot see, develops around two years of age ( 13 ) , understanding how a scene appears from another’s viewpoint, or level 2 VPT, begins to develop at age four ( 15 ) and continues throughout middle childhood ( 19 ) and even into adolescence and young adulthood ( 2 ) . This protracted developmental trajectory suggests that level 2 VPT depends on cognitive processes that are qualitatively distinct from those supporting level 1 VPT, and that these processes require extended experience to mature. If humans, with their rich embodied experience and social learning, need years to develop this skill, it is perhaps unsurprising that MLMs trained primarily on static image-text pairs struggle with perspective-taking. Understanding the cognitive mechanisms humans employ to overcome egocentric bias may provide insight into what is fundamentally missing in current MLMs, and suggest pathways for developing these capabilities. Researchers have explored several approaches to enable VPT in MLMs. One proposed solution has been to augment MLMs with external computer vision systems for spatial reasoning. For example, 8 developed Abstract Perspective Change (APC) in which vision foundation models (e.g., depth, pose, orientation estimation) parse perspective-based prompts to guide reasoning in language models. While effective, such multi-model approaches are computationally expensive and, critically, do not address whether spatial reasoning can emerge within the internal representations of MLMs themselves. An alternative line of work suggests that visual grounding can be strengthened through explicit token design. LLaVA Aurora ( 1 ) introduces perception tokens that encode intermediate visual representations, such as depth maps and bounding box coordinates, directly into the language model’s token space. By expanding the tokenizer and fine-tuning the multimodal projector and language embeddings, this approach allows models to reason over structured visual information internally, improving performance on perception-heavy tasks without relying on external tools. These findings raise a compelling question: if perception tokens can enhance low-level visual understanding, could similarly structured tokens help MLMs acquire the higher-order spatial reasoning skills required for perspective-taking? In this study, we leveraged insights from human visual perspective-taking (VPT) to test whether perspective-based reasoning can emerge within the internal representations of a multimodal language model. Specifically, we focused on LLaVA 1.5 13B ( 11 ) , a relatively small, transformer-based, open-source MLM. In the human cognitive literature, level 2 VPT relies on cognitive processes that are qualitatively distinct from those involved in level 1 VPT. Behaviorally, response times in level 2 VPT tasks increase systematically with the degree of misalignment between the observer and the reference frame, whereas the degree of alignment has little effect on level 1 judgments ( 17 ) . Neuroimaging studies further show that level 2 VPT engages a distributed network including regions associated with theory of mind, internal body representations, and mental rotation ( 5 ) . Together, these findings suggest that humans overcome egocentric bias in level 2 VPT by temporarily embodying another perspective and transforming spatial representations accordingly. We used these mechanistic insights as a guiding framework to probe whether analogous internal operations could be induced in a transformer-based MLM, rather than relying on external spatial reasoning modules. Inspired by VPT as an embodied process ( 7 ) , we developed two complementary approaches grounded in distinct cognitive mechanisms observed in human perspective-taking. The first approach focused on perspective embodiment, using tokens derived from body-related information to support reasoning about whether a reference was in an aligned (same perspective as viewer) or unaligned (opposite perspective as viewer) state. This approach allowed us to examine whether explicit sensitivity to perspective alignment could support allocentric reasoning within the model. Observations that some degree of alignment sensitivity was present even in the base model motivated a second approach that treats VPT as a form of mental rotation. In this framework, humans construct abstract scene representations that can be transformed to reconcile differing viewpoints ( 18 ) . Consequently, our second approach incorporated spatial information about both reference and query objects to enable direct transformation of scene representations based on the reference’s orientation. Together, these approaches provide a mechanistically grounded test of whether human-inspired cognitive operations can be instantiated within MLM representations, offering a principled framework for comparing artificial and human perspective-taking. Results Figure 1: Overview of evaluation and training data. a , Stimuli from the perspective-taking benchmark systematically vary the reference avatar’s orientation across degrees of alignment (0°–360°). b , Token construction methods: embodiment tokens encode body keypoint coordinates and calculated yaw orientation (above); rotation tokens encode object bounding box coordinates and azimuth labels from OrientAnything (below). c , Curriculum learning structure showing progression from atomic token generation to chain-of-thought reasoning to direct answer formats. The training schedule increases the proportion of chain-of-thought and direct items by 10% each epoch. d , Representative items from each evaluation benchmark (Perspective-Taking, Isle Bricks V2, COCO validation, 3DSRBench). In our analysis, we primarily evaluated our models on a perspective-taking benchmark that systematically varies the reference’s alignment with the viewer’s perspective