Unified Panoramic Geometry Estimation via Multi-View Foundation Models

Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek · 2026

Read Unified Panoramic Geometry Estimation via Multi-View Foundation Models by Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26368.txt.

Introduction

Vukasin Bozic and six colleagues lift 3D foundation models built for flat photos into the panorama world with PaGeR. From a trained transformer for 3D rebuilding, they tune one model to read flat and wraparound images alike. One pass yields two scales of depth, surface normals, and sky masks. Mixing both image kinds in training keeps the old 3D know-how intact. Indoor and outdoor tests report top scores and strong results on unseen scenes.

Catalog record

CREATORVukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54447
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Panoramic Geometry Estimation via Multi-View Foundation Models Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek Abstract Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360 ∘ 360^{\circ} scene from a single panoramic image. In this work, we introduce PaGeR ( Pa noramic Ge ometry R econstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360 ∘ 360^{\circ} scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here . Input Point Cloud Depth Input Depth / Normal Figure 1 : Different geometric modalities predicted by PaGeR. Given only a single monocular panoramic image, our framework simultaneously reconstructs highly detailed scale-invariant depth, absolute metric depth, surface normals, and sky segmentation masks across both indoor and outdoor environments. Spatial measurements are presented in meters. 1 Introduction Sensing and understanding the 3D structure of the surrounding world is important in many applications, ranging from virtual and augmented reality to autonomous driving and robotics. Scene depth and surface normals are two central geometric properties in that context: together, they describe the position and the local surface orientation at any point of the scene, providing a complete representation that supports graphics tasks like rendering and relighting as well as high-level perception tasks like spatial reasoning and path planning. A particularly attractive, but also heavily ill-posed variant is to recover depth or surface normals from a single RGB image [ 11 , 19 ] , obviating the need for multi-view capture and camera pose estimation. Early attempts relied on limited datasets and convolutional backbones [ 5 , 8 ] , but large-scale data collection [ 36 ] and advances in neural architectures, most notably vision transformers and denoising diffusion models [ 50 , 19 ] , have greatly advanced monocular geometry estimation. Most recently, this trend has converged with learning-based multi-view reconstruction, leading to foundation feed-forward models [ 43 , 25 ] capable of zero-shot, dense 3D reconstruction. From their massive training datasets, captured under diverse imaging conditions, these models acquire not only an understanding of multi-view geometry but also an elaborate prior of the world’s 3D surface structure, which supports detailed, dense depth estimation from single views. Yet, these models are designed for perspective images such that every image covers only a limited field of view, and many viewpoints must be aggregated and fused to build up spatial context and perceive the complete environment. Panoramic images, by construction, provide a full 360° view around the camera location, offering rich global context for holistic 3D understanding. However, high-quality panoramic datasets with metrically accurate depth and surface normals, needed to train panoramic reconstruction models, are laborious to collect and remain scarce. As a result, existing models tend to overfit to comparatively small datasets and struggle to generalize to unseen scenes. Another limitation is that existing models commonly represent panoramic images in equirectangular projection, which introduces serious geometric distortions. On the one hand, this means an extremely uneven sampling of the ray space (and, after unwarping, of the 3D environment). On the other hand, and perhaps more importantly, it means that one cannot easily employ transfer learning from models trained with perspective images. We take a different route and explore cubemaps as our panorama representation. Rather than designing custom architectures applicable specifically to panoramas, we use this parametrization to repurpose state-of-the-art perspective foundation models for the panoramic domain. The cubemap representation has been used to adapt pre-trained, diffusion-based image generators to 360° imagery [ 17 ] . For our purposes, we prefer to build on top of deterministic, feed-forward foundation models. Besides avoiding the computational overhead of diffusion and the practical limitations of operating in a compressed latent space, models like DA3 [ 25 ] are already designed for (perspective) multi-view input. They offer a natural synergy with the cubemap format, as their geometric prior includes the integration of multiple viewing directions and is fundamentally stronger than previous, purely monocular schemes. To properly anchor the spatial context, we explicitly condition the architecture on camera parameters and introduce targeted modifications of the decoder to ensure distortion-free and seamless reconstruction across face boundaries. Furthermore, we propose to use a mixed training regime with both synthetic panoramas and real perspective imagery. This strategy allows the network to adapt to the 360° setting while remaining firmly grounded in real-world image statistics, thus preventing overfitting to peculiarities of synthetic data and preserving the prior of the pre-trained foundation model. Taken together, we introduce PaGeR , a unified geometry estimation framework for panoramic (and perspective) images. Its latent representation, inherited from the foundation geometry model, is holistic and allows for simultaneous decoding of multiple scene properties. We exploit this property and equip the backbone with multiple, coupled task heads: Scale-Invariant (SI) depth, metric scale estimation, surface normals, and sky segmentation. That design allows the network to jointly reason about related geometric properties and to efficiently extract a comprehensive 3D representation in a single forward pass (see Fig. 1 ). By reparameterizing panoramic geometry as a structured multi-view problem, we achieve high-resolution, metrically accurate predictions that set a new state of the art for several benchmarks. Furthermore, to address the lack of benchmark data for rigorous evaluation in long-range, outdoor scenarios, we curate and introduce ZüriPano, a novel dataset of real-world outdoor panoramas with associated high-accuracy LiDAR scans. In summary, our contributions are: • A novel strategy to adapt foundation geometry models to panorama geometry. Our scheme is built around the cubemap representation consisting of six perspective images, and combines it with a hybrid training strategy to seamlessly transfer 3D scene priors to the 360° panorama setting, while sidestepping degradations caused by equirectangular distortion. • PaGeR, a unified panoramic geometry estimation model , featuring a shared transformer backbone (adopted from DA3) and specialized task heads to enable holistic reconstruction in a single forward pass. • Zero-shot generalization to unseen indoor and outdoor scenes, outperforming methods limited to a specific setting; and the new ZüriPano benchmark for zero-shot evaluation. 2 Method This section outlines the geometric preliminaries of panoramic representations and our perspective backbone architecture. We then introduce the panoramic adaptation layers and the hybrid training strategy designed to bridge the perspective and spherical domains. Finally, we detail the unified multi-task architecture and formalize the loss objectives for each geometric modality. 2.1 Preliminaries Panoramic Image Representations.

Read the complete text