PanoWorld: Towards Spatial Supersensing in 360 ∘ Panorama World
Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen, [1ex] Zhejiang University · 2026
Read PanoWorld: Towards Spatial Supersensing in 360 ∘ Panorama World by Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen, [1ex] Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13169.txt.
Introduction
Wang and seven co-authors study how multimodal models handle a 360-degree panorama as a whole, viewer-centred space, instead of splitting it into flat views. They define key skills including meaning anchors, spherical localisation, frame shifts and depth-aware 3D sense, and build a setup turning mixed-source shots into geometry-aware training. Their PanoWorld model injects spherical geometry into the visual stream via Spherical Spatial Cross-Attention. A new test benchmark, PanoSpace-Bench, shows PanoWorld clearly beats closed and open-source rivals.
Catalog record
| CREATOR | Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen, [1ex] Zhejiang University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65817 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PanoWorld: Towards Spatial Supersensing in 360 ∘ Panorama World Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen, [1ex] Zhejiang University Abstract Multimodal large language models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360 ∘ panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H ∗ Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released at https://wcpcp.github.io/PanoWorld . Figure 1 : Existing MLLMs reason over fragmented local views, making it difficult to associate spatial cues in 360 ∘ . We introduce pano-native supersensing, which teaches VLMs to perceive and reason directly over 360 ∘ panoramas, providing a unified full-surround representation for downstream tasks such as human-centric visual search, omnidirectional 3D spatial reasoning, and panoramic navigation. 1 Introduction Recent multimodal large language models (MLLMs) have made substantial progress in perspective-image visual understanding, yet robust spatial reasoning remains challenging [ 39 , 36 , 48 , 34 ] . A key limitation of this paradigm is that it inherits the limited instantaneous field of view of human-like perception, whereas tasks such as human-centric visual search, navigation, and immersive scene understanding benefit from full-surround environmental awareness. 360 ∘ panoramic sensing therefore offers a form of supersensing, expanding spatial perception from local views to the entire observer-centered environment. Despite this potential, current approaches often address full-surround reasoning through sequential perspective exploration, as highlighted by recent efforts [ 52 ] , where a continuous panorama is decomposed into local perspective views to simulate human-like exploration of the surrounding 3D environment. This naturally raises the question of whether 360 ∘ spatial reasoning can be modeled more directly and efficiently from panoramic representations themselves, which encode globally consistent observer-centered scenes, including wrap-around continuity and viewpoint-dependent spatial relations. However, directly transferring existing perspective models to panoramic understanding is challenging, since panoramic and perspective images exhibit fundamental representation gaps, including geometric distortion, non-uniform spatial sampling, and boundary discontinuities [ 24 ] . Although some existing approaches in other panoramic tasks utilize perspective-transfer pipelines through projection and stitching [ 3 , 42 , 12 , 30 , 62 ] , recent progress suggests that panorama-specific models trained on large-scale panoramic data can achieve stronger performance [ 25 , 12 ] , which highlights the importance of large-scale panoramic supervision for learning pano-native representations. These observations suggest that panoramic MLLMs should likewise move beyond perspective transfer toward panorama-specific modeling. Moreover, to enable a unified MLLM capable of handling diverse spatial reasoning tasks, it is essential to systematically characterize the capabilities. However, existing efforts on panoramic MLLMs are typically organized around individual tasks or benchmarks [ 63 , 57 , 49 , 10 ] , and thus remain fragmented and incomplete, lacking both a systematic understanding of the required capabilities and a unified benchmark to evaluate them. To address these limitations, we propose a unified pano-native spatial learning framework for MLLMs that learns an observer-centered representation of the 360 ∘ surrounding environment. We begin by introducing a capability-based formulation of panoramic spatial understanding, decomposing it into key components such as spherical localization, relative direction reasoning, viewpoint transformation, 3D spatial relations, and global scene topology. Building on this formulation, we develop a metadata-driven pipeline for scalable panoramic data construction and build a comprehensive benchmark that evaluates these capabilities beyond conventional VQA-style metrics. As summarized in Table 1 , the resulting resource covers 570K ERP panoramas and provides a combination of depth-aware signals, entity-level metadata, scalable annotation, and verified graph supervision not jointly available in prior panoramic resources. We further introduce a pano-aware MLLM model PanoWorld with Spherical Spatial Cross-Attention, enabling the model to align visual features with the underlying geometry of panoramic inputs. Extensive experiments show that our approach achieves strong performance on the proposed benchmark and generalizes effectively to existing 360 ∘ reasoning benchmark H ∗ Bench [ 52 ] and VLN benchmark R2R-CE Val-Unseen [ 21 ] , substantially outperforming existing methods. In summary, our main contributions are as follows: • We systematically formulate panoramic spatial reasoning in MLLMs as a capability-structured problem, and derive a taxonomy including spherical localization, relative direction reasoning, viewpoint transformation, 3D spatial relations, and global scene topology. • Based on this formulation, we develop a scalable metadata-driven pipeline for large-scale panoramic data construction, together with a comprehensive benchmark that systematically evaluates the defined spatial reasoning capabilities. • We propose a pano-aware MLLM model PanoWorld with spherical spatial cross-attention for geometry-consistent panoramic understanding. • Extensive experiments demonstrate that the proposed model has achieved competitive performance on the proposed benchmark and effective transfer to existing 360 ∘ reasoning and VLN benchmarks, with substantial gains over prior methods. 2 Related Work Table 1 : Comparison with representative panoramic resources. We report the number of underlying panoramic images rather than only QA counts. ✓: available; ✗: unavailable; \triangle : partially available. Resource Number View Scene Depth / 3D Entity metadata Scalable annotation QA / Captions Verified graph PanoCity [ 15 ] 120K Pano Outdoor ✓ ✗ \triangle ✗ ✗ Pano-AVQA [ 53 ] 5.4K Video Mixed \triangle \triangle ✗ ✓ ✗ Dense360 [ 63 ] 160K Pano Mixed ✗ ✓ ✓ ✓ \triangle OmniVQA [ 57 ] 1.2K Pano Indoor ✗ \triangle \triangle ✓ ✗ ODI-Bench [ 49 ] 2K Pano Mixed ✗ \triangle ✗ ✓ ✗ CFpano [ 56 ] 2.7K Multi-Pano Mixed ✗ \triangle ✗ ✓ ✗ PanoVQA [ 11 ] 44.6K Pano Outdoor ✗ \triangle \triangle ✓ ✗ OSR-Bench [ 10 ] 4.1K Pano Indoor \triangle \triangle ✗ ✓ ✗ PanoEnv [ 26 ] 595 Multi-Persp. Mixed ✓ \triangle ✗ ✓ ✗ PanoWorld 570K Pano Mixed ✓ ✓ ✓ ✓ ✓ Pano-Native Panoramic Designing. To bridge gaps [ 24 ] between panoramic and perspective understanding, recent studies have emphasized the need for panorama-specific design. Existing efforts mainly address this problem from two perspectives: data and models. On the data side, recent works construct task-specific datasets and benchmarks for perception [ 25 , 14 , 60 ] , and MLLM understanding [ 63 , 57 , 49 , 26 ] On the model side, prior studies introduce pano-aware designs, including distortion map [