Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Mahtab Bigverdi Linjie Li Weikai Huang Yiming Liu Jaemin Cho, Jieyu Zhang Tuhin Kundu Chris Dangjoo Kim Zelun Luo, Linda G. Shapiro Ranjay Krishna · 2026
Read Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models by Mahtab Bigverdi Linjie Li Weikai Huang Yiming Liu Jaemin Cho, Jieyu Zhang Tuhin Kundu Chris Dangjoo Kim Zelun Luo, Linda G. Shapiro Ranjay Krishna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03988.txt.
Introduction
Bigverdi and co-authors target spatial reasoning in vision-language models. Such models fail when key facts are not in plain view. A task may need a view from an unseen angle. It may need a path traced through blocked space. Partial views must merge into one clear spatial picture.
Their Imaginative Perception Tokens stand for what the model would see under other layouts. They stay tied to the observed input. The team built about 20,000 examples across three tasks. Those tasks are perspective taking, path tracing, and multi-view counting. With the BAGEL model, token training beat text chain-of-thought training. Counting accuracy rose by 3.4 points on one task. Gains grew when token and label training were combined.
Catalog record
| CREATOR | Mahtab Bigverdi Linjie Li Weikai Huang Yiming Liu Jaemin Cho, Jieyu Zhang Tuhin Kundu Chris Dangjoo Kim Zelun Luo, Linda G. Shapiro Ranjay Krishna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models Mahtab Bigverdi Linjie Li Weikai Huang Yiming Liu Jaemin Cho, Jieyu Zhang Tuhin Kundu Chris Dangjoo Kim Zelun Luo, Linda G. Shapiro Ranjay Krishna Abstract Vision-language models (VLMs) excel at many tasks, yet continue to struggle with spatial reasoning—problems where the key information is not directly observable in the input. Many spatial questions require imaginative perception : simulating an unseen viewpoint, tracing a trajectory through an occluded space, or integrating partial views into a coherent spatial map. Humans naturally support this kind of reasoning through imagination. Prior work has introduced intermediate visual representations (e.g., visual thoughts, depth, or box tokens), but these intermediates often refine structure already visible rather than predicting the missing spatial structure implied by the evidence. We introduce Imaginative Perception Tokens (IPT) , intermediate perceptual representations that externalize what a VLM would perceive under an alternative spatial configuration while remaining consistent with the observed input. To study this capability, we formulate three tasks that require imaginative perception: Perspective Taking (PET) , Path Tracing (PT) , and Multiview Counting (MVC) . For each task, we construct datasets of ∼ \sim 20K examples spanning simulated and real-world settings, paired with ground-truth intermediate imaginations, final answers, and curated evaluation benchmarks. Using the unified VLM BAGEL 12 as our backbone, IPT supervision improves spatial reasoning across several settings and often outperforms textual chain-of-thought training, even when no image is generated at inference time. For example, on MVC, IPT improves accuracy by 3.4% and achieves performance competitive with strong closed-source models on Path Tracing. We also find that mixed training with IPT and label-only data can further improve performance. In contrast, textual chain-of-thought can be detrimental on these tasks, substantially degrading performance in some cases, highlighting a modality mismatch when forcing spatial computation through language. Overall, IPT provides a principled supervision signal for reasoning over unobserved structure, yielding stronger spatial generalization and a more interpretable intermediate aligned with the underlying geometry of the task. Code will be released at the project page . † † footnotetext: * Equal contribution. Figure 1 : Overview of the three spatial imagination tasks. The left columns show training examples with ground-truth imaginative perception; the right columns show evaluation examples. 1 Introduction Spatial reasoning still still remains a persistent challenge for vision-language models (VLMs) 10 ; 8 ; 1 . Many spatial questions require reasoning about how objects relate within a three-dimensional environment, how these relationships change under viewpoint transformations, or how information from multiple partial observations should be integrated into a coherent scene representation 43 ; 18 for vision-language models (VLMs) 10 ; 8 ; 1 . While current models can often recognize objects and attributes, they frequently struggle when reasoning requires manipulating spatial structure, such as predicting how a scene would appear from another viewpoint 22 ; 24 or aggregating information across multiple views 39 . A key reason for this difficulty is that many spatial reasoning problems cannot be solved by analyzing the input alone. Instead, they require constructing a spatial representation that is not directly observed. Humans naturally address such problems through imagination: when asked what lies to the left after moving to a new position, or how many objects exist in a room seen from several viewpoints, we mentally simulate the scene from unseen perspectives or integrate partial observations into a unified spatial map 35 ; 43 ; 44 . In other words, spatial reasoning often depends on imagining missing spatial structure that proceed despite incomplete observations. Existing approaches provide only partial solutions. Recent work teaches models to generate intermediate visual thoughts alongside language 15 ; 21 ; 17 , while others introduce structured perceptual intermediates, such as depth maps or bounding boxes represented as tokens 2 ; 40 ; 28 . Although these methods demonstrate that intermediate visual representations can support reasoning, they primarily operate over information already present in the input observation, refining visible structures or extracting perceptual attributes. However, as discussed above, many spatial reasoning problems arise precisely because the required spatial information is not directly observable, and therefore requires imagination. To address this gap, we propose Imaginative Perceptual Tokens for VLMs. When VLMs are trained with them, they enable intermediate reasoning steps that represent novel spatial views. Unlike standard perceptual intermediates that describe structures visible in the input, imaginative representations correspond to what the model would perceive if it were observing the input from a different spatial configuration, such as from an unseen viewpoint or after integrating multiple partial observations into one. At the same time, they are not unconstrained imagination: the predicted percept must remain consistent with the observed scene. These tokens externalize the model’s prediction of what would be perceived given incomplete spatial evidence. To study this capability, we propose three spatial reasoning tasks that fundamentally require imaginative perception. (1) Perspective Taking requires predicting how a scene would appear from a new viewpoint given a single first-person observation (“If you move to the marked position and turn left, will the chair appear on your left or right?”); (2) Path Tracing requires inferring what an agent would see along a navigation path based on a top-down view (“If you walk along the marked path, which object will you see on your side?”); Finally, (3) Multiview Counting requires integrating multiple partial observations into a top-down view to determine the number of objects present in the scene. These tasks would be made easy when correctly predicting what would be perceived in a different spatial configuration. For each task we construct a dataset of approximately 20k examples each drawn from both real-world and synthetic simulated environments, with ground-truth intermediate spatial imaginations paired with final answers. Each dataset is accompanied by a human-filtered benchmark for evaluation. Together these constitute the first datasets designed explicitly to train and evaluate visually-grounded intermediate spatial reasoning in models. Empirically, we find that training with imaginative perceptual supervision can improve performance on these spatial reasoning tasks compared to answer-only supervision, and often compares favorably to textual chain-of-thought approaches. These improvements can persist even when the model does not explicitly generate intermediate images at inference time, suggesting that such supervision may help models develop stronger internal spatial representations. At the same time, we observe that the benefits vary across tasks and settings, indicating that imagination quality and task structure both play important roles. Overall, our results suggest that supervising models with intermediate perceptual predictions offers a useful direction for improving spatial reasoning, particularly in settings where the required structure is not directly observable from the input. 2 Related Works Evaluation of VLMs’ spatial reasoning. A growing body of benchmarks has established that spatial reasoning remains a persistent weakness of modern vision-language models. Early datasets target brittleness in basic spatial predicates: SpatialSense 36 reduces language priors through adversarial crowdsourcing, while VSR 23 scales relation types in a caption-verification format, and What’sUp 18 uses minimal-pair testing to reveal systematic failures on left/right and above/below distinctions. More recent work shifts from 2D relations to viewpoint and 3D structure. 3DSRBench 24 shows that models fail under modest changes in perspective, depth, and occlusion, and ViewSpatial-Bench 22 identifies a “perspective gap”: models often succeed in camera-centered views but break when asked to