REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion

Xuewei Li, Xinghan Bao, Zhimin Chen, Xi Li · 2026

Read REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion by Xuewei Li, Xinghan Bao, Zhimin Chen, Xi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16788.txt.

Introduction

Xuewei Li, Xinghan Bao, Zhimin Chen, and Xi Li sort wide-view street shots into named zones. A full round view shows roads, cars, folks, and sky at once. Tools must tag each point in that wide span. Depth cues help split near from far in the shot. Past tools used depth in raw or plain forms. Their 2026 preprint builds a depth code fit for round views.

Their code reshapes depth to match the round warp of the shot. A fusion step then joins depth with the color view. Sharper tags aid self-drive cars and guide bots. Such tools need the full round scene read right.

Catalog record

CREATORXuewei Li, Xinghan Bao, Zhimin Chen, Xi Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16788

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion Xuewei Li, Xinghan Bao, Zhimin Chen, Xi Li Abstract As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to give complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or using the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and S pherical-dynamic Multi-Modal F usion (SMMF). REL is made up of R ectified Depth, E levation-Gained Vertical Inclination Angle, and L ateral Orientation Angle, which fully represents 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, which uses different fusion strategies to match the different regions in panoramic images. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmark, Stanford2D3D Panoramic datasets. It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance. 1 Introduction (a) ERP projection. (b) Cylindrical coordinate. (c) Comparison of HHA and REL. Figure 1 : Overview of ERP projection, cylindrical coordinate and comparison of HHA and REL. \hat{T}_{\theta,p} is the tangent line to circle O_{p} at point p . Circle O_{p} lies in the plane perpendicular to gravity, passes through point p , and its center lies on the cylindrical axis L_{cyl} of \rho\theta z . In (c), we compared how different structure information are represented in REL and HHA. There has been a great growing trend of practical applications based on 360^{\circ} cameras in recent years, including holistic sensing in autonomous vehicles 6 ; 21 ; 11 , immersive viewing in augmented reality and virtual reality devices 29 ; 30 ; 1 ; 39 , etc. Panoramic images with a complete Field of View (FoV, which is 360^{\circ}\times 180^{\circ} ) deliver complete scene perception in many real-world scenarios, thus drawing increasing attention in the research community in computer vision. Panoramic semantic segmentation (PASS) is essential for omnidirectional scene understanding, as it gives pixel-wise analysis for panoramic images and offers a dense prediction technical route acquiring complete perception of surrounding scenes 33 . Most existing PASS approaches use equirectangular projection (ERP) 25 ; 34 ; 10 to convert original 360^{\circ} data to 2D panoramic images. In brief, the original spherical information is first projected onto the side surface of a cylinder, and then the cylinder side surface is expanded as shown in Fig. 1(a) . Compared with RGB-only input, PASS strategies with RGB and depth input gain better performance, because depth information D includes 3D location and shape (typically represented by surface normal) of each pixel beneficial to dense prediction like PASS. HHA 13 is widely used to explicitly model the 3D location and surface normal of each pixel in camera cartesian coordinate with horizontal disparity (first H, H_{1} ), height above ground (second H, H_{2} ), and the angle between surface normal and gravity direction (last A, A_{1} ), which gain considerable improvement compared with directly using original D . 3D location has 3 degrees of freedom when surface normal direction has 2 degrees of freedom. As shown in Fig. 1(c) , HHA representation utilizes 3 different channels ( H_{1} , H_{2} , and A_{1} ) and the pixel location in the image to represent 4 degrees of freedom information. However, the representation of the second degree of freedom for the surface normal direction is missing. Furthermore, the HHA calculation depends on the camera posture and intrinsics (e.g., the focal length), which is not comfortable enough to deal with image-only data. Facing this limitation and inspired by the cylinder used in ERP, we propose REL representation and directly use cylindrical coordinate \rho\theta z to represent the 3D location and surface normal of each pixel. REL representation is composed of R ectified Depth (ReD), E levation-Gained Vertical Inclination Angle (EGVIA), and L ateral Orientation Angle (LOA). Firstly, as shown in Fig. 1(c) , for the point p in 3D space that corresponds to a certain pixel, we calculate the plane distance \rho , the plane angle \theta , and the height z (the same as H_{2} ) to directly represent the 3D location. Secondly, we calculate the angle between surface normal and gravity direction and lateral orientation angle which is the angle between surface normal and tangent line \hat{T}_{\theta,p} . Also, REL representation does not need camera posture and intrinsics. Furthermore, we find roughly collinearity between H_{2} and A_{1} in panoramic images (discussed in Fig. 5 in detail), we fuse H_{2} and A_{1} as our EGVIA, set planar distance \rho as our ReD, use lateral orientation angle as LOA, and gain our REL representation. It has 3 channels and is the same as HHA for a relatively fair comparison. In addition, due to the different local situations (e.g., distortion, location, etc.) of different regions in the panoramic image caused by ERP, we propose our Spherical-dynamic Multi-Modal Fusion (SMMF) to effectively fuse the RGB and REL information for different regions of panoramic images by different ways. Instead of achieving regions from input images, SMMF uses overlapping regions sampled on the cylinder side surface to reduce the breakage to the scene structure caused by cylinder side surface expansion. It not only allows different fusion ways for different regions, but also extracts semantic features across the left and right edges of a panoramic images. Using REL representation and SMMF jointly, we build up REL-SF4PASS, and our contributions are summarized as follows: • We propose our 3-channel REL representation to represent the 3D location and shape of each pixel based on cylindrical coordinate. • Our Spherical-dynamic Multi-Modal Fusion (SMMF) enriches the diversity of fusion for different regions in panoramic images and reduce the impact of cylinder side surface expansion by ERP projection. • We evaluate REL-SF4PASS on popular Stanford2D3D Panoramic datasets, and gain the 2.35% average mIoU improvement on all 3 folds. Also, it reduces the performance variance by about 70% in SGA validation. 2 Related Work The two most related fields are panoramic semantic segmentation and RGB-D semantic segmentation. Figure 2 : The overview of REL-SF4PASS. Firstly, REL representation effectively represents the depth information by using ReD, EGVIA, and LOA, which contains the 3D location and surface normal direction. Secondly, Our SMMF to get image regions from cylinder side surface and uses a Gate Network to independently determine the fusion used in each region. 2.1 Panoramic Semantic Segmentation PAnoramic Semantic Segmentation (PASS) is the task that does semantic segmentation on panoramic images. In recent years, many models have been developed for PASS. Deng et al. 7 built up the first semantic segmentation framework for big FoV (wide-angle / fish-eye) images. Furthermore, Yang et al. 32 used attention connections to make PASS more efficient, and DS-PASS was proposed. Common PASS solutions can be divided into two main fields: geometry-aware strategies and distortion-aware ones. For geometry-aware strategies 23 ; 18 ; 20 ; 19 ,

Read the complete text