Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation
Phillip Y. Lee Jihyeon Je Chanho Park Mikaela Angelina Uy, Leonidas Guibas Minhyuk Sung · 2025
Read Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation by Phillip Y. Lee Jihyeon Je Chanho Park Mikaela Angelina Uy, Leonidas Guibas Minhyuk Sung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17207.txt.
Introduction
Phillip Lee and five co-authors teach vision-language models to take other viewpoints in twenty twenty-five. Taking another's perspective is a key part of human visual understanding. It matters when people work with autonomous agents in shared spaces. Modern vision-language models, the authors note, are weak at this and lean hard toward their own viewpoint.
Humans shift perspective through mental imagery, using abstract inner pictures of a scene. The authors build a framework, Abstract Perspective Change, on that idea. The model forms an abstract image of the scene and then simulates how it would look from elsewhere. Readers get a method aimed at closing a clear gap between model and human perception.
Catalog record
| CREATOR | Phillip Y. Lee Jihyeon Je Chanho Park Mikaela Angelina Uy, Leonidas Guibas Minhyuk Sung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation Phillip Y. Lee Jihyeon Je Chanho Park Mikaela Angelina Uy, Leonidas Guibas Minhyuk Sung Abstract We present a framework for perspective-aware reasoning in vision-language models (VLMs) through mental imagery simulation. Perspective-taking—the ability to perceive an environment or situation from an alternative viewpoint—is a key benchmark for human-level visual understanding, essential for environmental interaction and collaboration with autonomous agents. Despite advancements in spatial reasoning within VLMs, recent research has shown that modern VLMs significantly lack perspective-aware reasoning capabilities and exhibit a strong bias toward egocentric interpretations. To bridge the gap between VLMs and human perception, we focus on the role of mental imagery, where humans perceive the world through abstracted representations that facilitate perspective shifts. Motivated by this, we propose a framework for perspective-aware reasoning, named Abstract Perspective Change ( APC ), that effectively leverages vision foundation models, such as object detection, segmentation, and orientation estimation, to construct scene abstractions and enable perspective changes. Our experiments on synthetic and real-image benchmarks, compared with various VLMs, demonstrate significant improvements in perspective-aware reasoning with our framework, further outperforming fine-tuned spatial reasoning models and novel-view-synthesis-based approaches. Our project page is at https://apc-vlm.github.io/ . Figure 1 : We introduce Abstract Perspective Change ( APC ) , a framework that empowers VLMs to adopt arbitrary perspectives for spatial reasoning. As demonstrated by the examples above, APC significantly enhances VLM’s ability to imagine a scene from alternative viewpoints , overcoming the inherent egocentric bias that constrains the spatial reasoning of existing VLMs to the camera’s viewpoint. 1 1 footnotetext: Correspondence: Phillip Y. Lee ([email protected]) and Minhyuk Sung ([email protected]) 1 Introduction Vision-language models (VLMs) have made remarkable progress, positioning themselves as a crucial backbone for general-purpose physical AI agents. The growing research efforts to improve VLMs’ spatial reasoning capabilities [ 74 , 12 , 10 , 53 ] reflect this potential. Early VLMs were limited to basic tasks such as visual question answering (VQA) and image captioning [ 41 , 40 , 1 ] . However, recent advancements have enabled them to perform complex visual reasoning [ 30 , 73 , 46 , 2 , 19 , 29 ] and extract spatial properties, including spatial relationships, relative sizes, and distances [ 37 , 74 ] . Further techniques such as instruction-tuning and vision-centric adapters, have expanded their capabilities, allowing depth-aware [ 10 , 7 ] and region-aware [ 12 , 27 ] spatial reasoning. Despite these advances, progress remains largely confined to egocentric spatial reasoning, and even the latest VLMs struggle with allocentric reasoning—answering questions from perspectives other than the camera’s (Fig. 2 ). Allocentric reasoning is crucial for high-level planning, environmental interaction, and collaboration with autonomous agents [ 22 , 81 , 85 ] . Moreover, it serves as a key benchmark for human-level spatial understanding. However, as analyzed by Zhang et al. [90] , most VLMs exhibit a strong bias toward an egocentric perspective. Even when explicitly prompted to adopt an allocentric viewpoint, VLMs often revert to egocentric interpretations [ 43 , 25 , 90 , 89 ] . Recent efforts to enhance spatial reasoning remain focused on improving egocentric reasoning [ 10 , 12 , 53 ] , leaving allocentric reasoning largely unaddressed. To bridge the gap between VLMs and human perspective reasoning, we ask: What cognitive process allows humans to effortlessly shift perspectives? Unlike current VLMs, humans seamlessly form internal representations of the physical world, making perspective reasoning an intuitive and natural process. The mechanism of creating internal representations, known as mental imagery [ 56 , 23 , 59 , 35 , 67 ] , plays a fundamental role in cognition, enabling us to simulate visual, spatial, and conceptual scenarios. This ability allows for abstraction beyond immediate perception, facilitating sophisticated spatial reasoning tasks such as mentally rotating objects, inferring occlusions, and envisioning alternative viewpoints [ 14 , 20 , 6 ] . A key aspect of mental imagery is that it is not simply the process of visualizing a clear image from different perspectives; rather, it involves forming an abstract representation of a scene that encodes essential spatial information and can be reinterpreted from a new perspective. From a computational standpoint, such an abstract representation is particularly advantageous, as equipping VLMs with the imaginative capability to generate novel views remains extremely challenging. In contrast, constructing an abstract representation requires significantly less computation and can be achieved procedurally. Figure 2 : Egocentric vs. Allocentric. While VLMs perform well when questions are asked from an egocentric ( i.e . camera’s) perspective, they struggle when the same questions are posed from an allocentric perspective, showing a strong bias toward egocentric reasoning. Inspired by this, we introduce a novel framework for adapting perspectives in VLMs by simulating the mental imagery process and modifying the perspective in the given prompt. Our goal is to leverage the strengths of both VLMs and recent vision foundation models, such as object detection [ 50 , 9 , 93 ] , image segmentation [ 33 , 62 ] , and orientation estimation [ 77 ] . The proposed framework takes an image and a perspective-based question as input and operates through three key stages. First, by simulating the mental imagery process, it builds an abstract representation of the scene in the input image. The VLM parses the prompt to identify objects in the image, while vision foundation models extract the center and orientation information of each object in 3D space. Second, the VLM analyzes the prompt to determine the reference object from whose perspective the question is asked, and transforms the abstraction to be aligned with that perspective. Finally, a new prompt is generated by reinterpreting the scene from the reference object’s perspective. We explore different formats for rendering the scene abstraction: (1) a text-based representation, where objects are described using numerical 3D coordinates, and (2) an image-based representation, where objects are visualized as colored boxes corresponding to the original image. The newly generated prompt is fed to the VLM to obtain the final answer. Note that our framework is not designed for a specific type of allocentric question. We leverage the egocentric reasoning capabilities of VLMs by performing allocentric-to-egocentric prompt conversion through scene abstraction, removing the perspective-related barrier in the question while preserving its original intent in the new prompt. In our experiments on COMFORT++ [ 90 ] and 3DSRBench [ 54 ] , our method achieves robust spatial reasoning across a variety of tasks and perspectives. In constrast, baseline VLMs and previous frameworks for spatial reasoning often struggle with even simple viewpoint shifts, reconfirming a notable bias toward the camera’s perspective. These results highlight how our abstraction-based representation significantly enhances the spatial reasoning capabilities of VLMs beyond their default egocentric perspectives. 2 Related Work 2.1 Spatial Reasoning with VLMs Building on the remarkable advancements of vision-language models (VLMs) [ 46 , 47 , 37 , 2 , 19 ] , recent studies have adapted VLMs for real-world spatial reasoning. Numerous evaluations revealed that VLMs struggle on even elementary spatial-perception tasks [ 61 , 72 , 75 , 60 , 24 ] and higher-level spatial reasoning based on images or videos [ 32 , 45 , 68 , 70 , 83 , 38