H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors

Chenxing Jiang, Zhe Tong, Pusen Gao, Peize Liu, Yang Xu, Chuan Fang, Ping Tan, Shaojie Shen · 2026

Read H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors by Chenxing Jiang, Zhe Tong, Pusen Gao, Peize Liu, Yang Xu, Chuan Fang, Ping Tan, Shaojie Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14963.txt.

Introduction

Chenxing Jiang and seven others present H-OmniStereo, a system for full-surround depth from top-bottom panoramic images. Vertical epipolar lines let perspective stereo methods carry over, but such datasets are scarce and monocular cues warp under spherical distortion. The team trains on over 2.8 million synthetic pairs of this kind and adds a heading-aligned surface-normal estimator for stable geometric cues. Tests show it beats rivals out of domain and transfers to consumer cameras.

Catalog record

CREATORChenxing Jiang, Zhe Tong, Pusen Gao, Peize Liu, Yang Xu, Chuan Fang, Ping Tan, Shaojie Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37640
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors Chenxing Jiang, Zhe Tong, Pusen Gao, Peize Liu, Yang Xu, Chuan Fang, Ping Tan, Shaojie Shen Abstract Stereo matching on top-bottom equirectangular images provides an effective framework for full-surround perception, as vertically aligned epipolar lines enable the use of advanced perspective stereo architectures that are largely driven by large-scale datasets and monocular priors. However, the performance of such adaptations is severely limited by the scarcity of omnidirectional stereo datasets and the degradation of perspective monocular priors under spherical distortions. To address these challenges, we propose H-OmniStereo, a zero-shot omnidirectional stereo matching framework. First, we construct high-quality synthetic dataset comprising over 2.8 million top-bottom equirectangular stereo pairs to scale up training. Second, we introduce an equirectangular monocular normal estimator, specifically operating in a heading-aligned coordinate system. Beyond providing distortion-robust and cross-view-consistent geometric priors for establishing reliable correspondences in stereo matching, this design boosts training efficiency and accommodates train-test FoV mismatches. Extensive experiments show that our approach achieves higher accuracy than existing methods on out-of-domain datasets and successfully generalizes to real-world consumer camera setups using a single model. The model and dataset will be released online 1 1 1 https://github.com/JIANG-CX/H-OmniStereo . I Introduction Omnidirectional (360°) images offer a wide field of view for comprehensive environmental perception. As shown in Fig. 1 , omnidirectional stereo pairs captured by top-bottom panoramic rigs exhibit vertically aligned epipolar lines under equirectangular projection. Exploiting this property makes stereo matching on top-bottom equirectangular image pairs a cost-effective solution for full-surround 3D perception [ 38 , 30 , 5 ] . Such capability is crucial for applications like robot navigation, AR/VR and environmental monitoring [ 41 ] . Stereo matching is a long-standing problem in computer vision. In recent years, substantial progress has been made in stereo matching for perspective images [ 33 , 4 , 32 ] . These advances have been largely driven by large-scale stereo datasets and monocular depth foundation models [ 37 ] . Trained on massive monocular datasets, these models generate dense features that serve as strong geometric priors for establishing stereo correspondences. However, these improvements do not readily transfer to omnidirectional stereo matching. A major reason is the scarcity of large-scale and diverse equirectangular stereo datasets, as summarized in TABLE. I . Consequently, existing omnidirectional stereo matching methods [ 15 , 30 , 38 , 5 ] are typically trained and evaluated in in-domain settings, resulting in limited zero-shot generalization. Moreover, when equirectangular images are subjected to different camera poses and spatial transformations, their inherent spherical distortions hinder the generalization of monocular depth priors learned from perspective images [ 1 ] . Although recent equirectangular monocular depth foundation models [ 1 , 14 , 17 , 8 ] are better adapted to equirectangular imagery, they predict the distance to the sphere center. As a result, the same 3D point can be assigned different depth values across two views. Because the features driving the depth prediction inherit this inconsistency, they are ill-suited to act as priors for establishing reliable correspondences in stereo matching. Fig. 1: Left: Spherical disparity under top-bottom camera pairs. Right: Dense 3D reconstruction and trajectory estimation (red line) achieved by integrating our proposed single-pair omnidirectional stereo matching model into a visual odometry pipeline. By processing sequential top-bottom equirectangular stereo pairs, our method enables complete environment recovery. The ceiling of the point cloud is cropped for better visualization. To address these challenges, we propose H-OmniStereo, an omnidirectional stereo matching framework that leverages a large-scale synthetic stereo dataset and omnidirectional monocular normal priors in a heading-aligned coordinate system to achieve strong accuracy and zero-shot generalization. Specifically, to scale up training for top-bottom equirectangular stereo, we build a large-scale, high-quality synthetic dataset of over 2.8M image pairs. It spans over 1K scenes and 800K diverse 3D assets, with stereo images captured under highly randomized camera parameters. Overall, compared to existing datasets [ 38 , 42 , 15 , 30 ] , our dataset is substantially more diverse and approximately 70 times as large as the largest prior dataset, providing a strong foundation for training omnidirectional stereo matching models with zero-shot generalization. Furthermore, to provide monocular priors that are robust to spherical distortions and geometrically consistent across top-bottom views, we train an equirectangular monocular normal estimator in a heading-aligned coordinate system. In standard normal estimation, surface normals are defined in a fixed camera coordinate system. While this maintains consistency across top and bottom views, it ignores the fact that heading rotations in equirectangular images preserve local patterns. Consequently, the model is unnecessarily burdened with distinguishing among spatially varying normals for identical local patterns, making it less accurate and more susceptible to train-test FoV mismatches of the input image. To address this, we define normals within a longitude-varying, heading-aligned coordinate system, as shown in Fig. 2 . By adapting each pixel’s coordinate system to its longitude, identical local patterns correspond to the same heading-aligned normal, regardless of heading. Moreover, pixels along the same longitude share the same coordinate system to ensure consistency across top-bottom pairs. Therefore, this representation improves learning efficiency and naturally accommodates varying inference FoVs and random cropping during stereo training. Additionally, since normal estimation is agnostic to the absolute scale of the scene and objects, our stereo dataset can also serve as accurate normal supervision. In total, we train the normal estimator on 8.4M equirectangular images, including an additional 5.6M monocular images collected from existing datasets [ 40 , 31 , 6 ] . In summary, the contributions of this work are as follows: • We construct a high-quality synthetic dataset for top-bottom omnidirectional stereo matching, comprising over 2.8M image pairs with substantially greater scale and diversity than previous datasets. • We propose a training-efficient equirectangular monocular normal estimator that specifically operates within a heading-aligned coordinate system. This estimator provides distortion-robust and cross-view-consistent geometric priors, facilitating reliable correspondence establishment in omnidirectional stereo matching. • Extensive experiments demonstrate that our method outperforms existing approaches on out-of-domain datasets and successfully generalizes to real-world consumer cameras using a single model. TABLE I: Comparison of top-bottom omnidirectional stereo datasets. Properties Deep360 [ 15 ] 360SD-Net [ 30 ] 3D60 [ 12 ] Helvipad [ 38 , 5 ] Ours Scenarios Flying Objects ✗ ✗ ✗ ✗ ✓ Indoor ✗ ✓ ✓ ✓ ✓ Outdoor ✓ ✗ ✗ ✓ ✓ GT Depth ✓ ✓ ✓ ✓ ✓ Normal ✗ ✗ ✓ ✗ ✓ Precise Calibration ✓ ✓ ✓ ✗ ✓ Camera Params Finite set Single fixed Single fixed Single fixed Arbitrary Stereo Pairs 18K 4K 25K 40K 2800K Diversity Low High High Medium High Source Type Synthetic Synthetic Synthetic Real-world Synthetic Resolution 1024{\times}512 1024{\times}512 512{\times}256 1920{\times}512 1024{\times}512 II Related Works II-A Perspective Stereo Matching Deep learning has significantly improved stereo matching accuracy and generalization, beginning with early CNNs that replaced handcrafted features [ 13 ] . Subsequently, a prominent line of work has emerged that estimates disparity through cost-volume construction and aggregation using 3D CNNs [ 2 ] . While these methods achieve high accuracy, they suffer from substantial memory and computational overhead. To address these inefficiencies, RAFT-style iterative frameworks [ 18 ] have been developed to recurrently refine disparities from learned correlations. These designs offer greater flexibility across

Read the complete text