Exploring Non-Local Spatial-Angular Correlations with a Hybrid Mamba-Transformer Framework for Light Field Super-Resolution
Haosong Liu, Haosong Liu, Graduate Student Member, IEEE, Xiancheng Zhu, Graduate Student Member, IEEE · 2025
Read Exploring Non-Local Spatial-Angular Correlations with a Hybrid Mamba-Transformer Framework for Light Field Super-Resolution by Haosong Liu, Haosong Liu, Graduate Student Member, IEEE, Xiancheng Zhu, Graduate Student Member, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04824.txt.
Introduction
Light field images record both position and angle of light, and raising their resolution requires using links across both. Haosong Liu, lead author with five co-authors, propose a hybrid Mamba-Transformer framework for light field super-resolution in a 2025 paper. Existing Mamba scanning strategies extract features inefficiently from such complex data.
Their design adds a subspace scanning strategy and blocks that model spatial-angular features in two stages. A parallel branch then refines epipolar features, the geometry linking views, using both Mamba and transformer parts. The framework aims for fuller use of non-local correlations at lower computational cost.
Catalog record
| CREATOR | Haosong Liu, Haosong Liu, Graduate Student Member, IEEE, Xiancheng Zhu, Graduate Student Member, IEEE |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Non-Local Spatial-Angular Correlations with a Hybrid Mamba-Transformer Framework for Light Field Super-Resolution Thanks: This work was supported in part by the Key Science and Technology Project of Fujian Province under Grant 2024HZ022007, in part by the Key Program of Natural Science Foundation of Fujian Province under Grant 2023J02022, in part by the Natural Science Foundation for Outstanding Young Scholars of Fujian Province under Grant 2022J06023, in part by the Natural Science Foundation of Fujian Province under Grant 2022J01294, in part by the Natural Science Foundation of Zhejiang Province under Grant LZ24F030010, and in part by the High-level Talent Team Project of Quanzhou City under Grant 2023CT001 ( Corresponding author: Huanqiang Zeng ). Thanks: Haosong Liu is with the School of Information Science and Engineering, Huaqiao University, Xiamen 361021, China (e-mail: [email protected]). Thanks: Xiancheng Zhu is with the School of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China (e-mail: [email protected]). Thanks: Huanqiang Zeng is with the School of Engineering, Huaqiao University, Quanzhou 362021, China, and also with the School of Optoelectronic and Communication Engineering, Xiamen University of Technology, Xiamen 361024, China (e-mail: [email protected]). Thanks: Jianqing Zhu is with the School of Engineering, Huaqiao University, Quanzhou 362021, China (e-mail: [email protected]). Thanks: Jiuwen Cao is with the Artificial Intelligence Institute, Hangzhou Dianzi University, Hangzhou 310018, China (e-mail: [email protected]). Thanks: Junhui Hou is with the Department of Computer Science, City University of Hong Kong, Hong Kong (e-mail: [email protected]). Thanks: Haosong Liu and Xiancheng Zhu contributed equally to this work. Haosong Liu, Haosong Liu, Graduate Student Member, IEEE, Xiancheng Zhu, Graduate Student Member, IEEE Abstract Recently, Mamba-based methods, with its advantage in long-range information modeling and linear complexity, have shown great potential in optimizing both computational cost and performance of light field image super-resolution (LFSR). However, current multi-directional scanning strategies lead to inefficient and redundant feature extraction when applied to complex LF data. To overcome this challenge, we propose a Subspace Simple Scanning (Sub-SS) strategy, based on which we design the Subspace Simple Mamba Block (SSMB) to achieve more efficient and precise feature extraction. Furthermore, we propose a dual-stage modeling strategy to address the limitation of state space in preserving spatial-angular and disparity information, thereby enabling a more comprehensive exploration of non-local spatial-angular correlations. Specifically, in stage I, we introduce the Spatial-Angular Residual Subspace Mamba Block (SA-RSMB) for shallow spatial-angular feature extraction; in stage II, we use a dual-branch parallel structure combining the Epipolar Plane Mamba Block (EPMB) and Epipolar Plane Transformer Block (EPTB) for deep epipolar feature refinement. Building upon meticulously designed modules and strategies, we introduce a hybrid Mamba-Transformer framework, termed LFMT. LFMT integrates the strengths of Mamba and Transformer models for LFSR, enabling comprehensive information exploration across spatial, angular, and epipolar-plane domains. Experimental results demonstrate that LFMT significantly outperforms current state-of-the-art methods in LFSR, achieving substantial improvements in performance while maintaining low computational complexity on both real-word and synthetic LF datasets. The code is available at https://github.com/hsliu01/LFMT . \svgsetup inkscapepath=i/svg-inkscape/ \svgpath svg/ I Introduction Light Fields (LFs) capture both the intensities (spatial information) and directions (angular information) of light rays, providing richer information than traditional images [ 1 ] . In recent years, the rise of portable LF cameras, such as Lytro and Raytrix, has accelerated the application of LF images, demonstrating great potential in areas such as post-capture refocusing [ 2 , 3 ] , depth estimation [ 4 , 5 , 6 , 7 , 8 ] , 3D reconstruction [ 9 ] , and virtual reality [ 10 , 11 ] . However, despite enhancing angular resolution through microlens arrays, these cameras inevitably compromise spatial resolution [ 1 ] , thereby restricting their practical applicability. Therefore, LFSR algorithms have been widely studied to improve the spatial resolution of LF images. In recent years, Convolutional Neural Networks (CNNs) have been widely applied to LFSR, aiming to learn complex spatial-angular information or correlations within LF images. A variety of feature extraction strategies have been proposed to alleviate the spatial-angular entanglement in 4D LF images. Some methods [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] extract features through several convolutions in single subspace to learn spatial or angular information, while others [ 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] utilize combined convolutions across multiple subspaces to extract and fuse features, thereby learning spatial-angular correlations. However, these methods fail to fully leverage the complete information in all subspaces, and the local receptive field of CNNs restricts the interaction in individual subspace or across different subspaces, resulting in inadequate exploration of spatial-angular correlations. Transformer models, known for their ability to capture long-range dependencies, have been increasingly applied to LFSR. Enhancing long-range interactions improves the modeling of non-local spatial-angular correlations. For example, networks such as LFT [ 29 ] , DPT [ 30 ] and LF-DET [ 31 ] extract long-range information from spatial and angular domains, while EPIT [ 32 ] alternately extract the horizontal and vertical structural information from epipolar plane domain. However, due to the quadratic complexity of self-attention computation, Transformer-based methods face challenges in constructing deeper networks to extract high-frequency information in all domains. This limitation hinders their ability to effectively and comprehensively explore non-local spatial-angular correlations in LF images. Therefore, effectively utilizing the complete information in all domains to model non-local spatial-angular correlations remains a significant challenge. Recently, State Space Models (SSMs), particularly Mamba [ 33 ] , have emerged as a powerful competitor to Transformer models, maintaining linear computational complexity while enabling long-range information modeling. Pure Mamba-based methods (e.g., MLFSR [ 34 ] and LFMamba [ 35 ] ) adopt the multi-directional scanning strategy to model non-local spatial-angular correlations. However, they overlook the inherent complexity and redundancy of LF data, resulting in limited efficiency and insufficient capability to capture the fine-grained spatial-angular structures required for high-fidelity LFSR. In contrast, pure Transformer models excel at global context modeling and can better restore spatial-angular structures close to the original LF. Nevertheless, their reliance on unfolding 4D LF data into 1D sequences for modeling incurs prohibitive computational costs, thereby constraining overall performance. As illustrated in Fig. 1 (b) and Fig. 1 (c), the coupling of LF subspace scanning may lead to computational redundancy and degrade performance in modeling non-local spatial–angular correlations. To overcome this challenge, we design the Subspace Simple Mamba Block (SSMB) as a basic component for feature extraction in LF subspaces, which adopts the Subspace Simple Scanning (Sub-SS) strategy to mitigate the inefficiency and coupling (refer to Sec. III-C for details). Additionally, to further enhance the utilization of feature space information, we introduce a symmetric branch composed of convolution and SiLU activation to compensate for the information loss caused by the Sub-SS sequential constraints. Fig. 1: (a) The 4D LF can be decoupled into 2D subspaces, including the spatial, angular, and epipolar plane domains. For illustration, the LF images on the left is shown with u=v=2 and h=w=3 , where different colors represent different viewpoints, and different letters indicate different pixels in the same viewpoint. (b) The coupling in bidirectional scanning [ 36 ] : when bidirectional scanning is performed in spatial and epipolar plane domains, the correlations