Omni-Fusion of Spatial and Spectral for Hyperspectral Image Segmentation
Qing Zhang, Guoquan Pei, Yan Wang (✉) · 2025
Read Omni-Fusion of Spatial and Spectral for Hyperspectral Image Segmentation by Qing Zhang, Guoquan Pei, Yan Wang (✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06606.txt.
Introduction
Qing Zhang, Guoquan Pei, and Yan Wang build a neural network that reads medical hyperspectral images by fusing what they show in space and in spectrum, in a 2025 paper. Such images, used in pathology, record a full light spectrum at every pixel, revealing tissue chemistry invisible to normal cameras. The richness is also the problem: the data are high-dimensional and spectrally redundant, so models struggle to use both dimensions well.
Their network, Omni-Fuse, mixes the dimensions repeatedly. A cross-dimensional module sharpens spatial and spectral features by letting each attend to the other. A spectral-guided step picks the spatial features most relevant to the spectrum as queries, and a two-stage decoder steers attention toward them. Despite the many attention blocks, the model stays efficient. On two microscopic hyperspectral datasets it beat state-of-the-art segmenters by over 5.73 points of Dice score. Code is public.
Catalog record
| CREATOR | Qing Zhang, Guoquan Pei, Yan Wang (✉) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06606
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Omni-Fusion of Spatial and Spectral for Hyperspectral Image Segmentation Qing Zhang, Guoquan Pei, Yan Wang (✉) Abstract Medical Hyperspectral Imaging (MHSI) has emerged as a promising tool for enhanced disease diagnosis, particularly in computational pathology, offering rich spectral information that aids in identifying subtle biochemical properties of tissues. Despite these advantages, effectively fusing both spatial-dimensional and spectral-dimensional information from MHSIs remains challenging due to its high dimensionality and spectral redundancy inherent characteristics. To solve the above challenges, we propose a novel spatial-spectral omni-fusion network for hyperspectral image segmentation, named as Omni-Fuse. Here, we introduce abundant cross-dimensional feature fusion operations, including (1) a cross-dimensional enhancement module that refines both spatial and spectral features through bidirectional attention mechanisms; (2) a spectral-guided spatial query selection to select the most spectral-related spatial feature as the query; and (3) a two-stage cross-dimensional decoder which dynamically guide the model’s attention towards the selected spatial query. Despite of numerous attention blocks, Omni-Fuse remains efficient in execution. Experiments on two microscopic hyperspectral image datasets show that our approach can significantly improve the segmentation performance compared with the state-of-the-art methods, with over 5.73% improvement in DSC. Code available at: https://github.com/DeepMed-Lab-ECNU/Omni-Fuse. † † footnotetext: Q. Zhang and G. Pei—Denotes equal contribution. 1 Introduction Hyperspectral Imaging (HSI) is an emerging technology in pathological diagnosis, reflecting the physiological and biochemical properties of different tissue components of tissues in a non-invasice manner [ 2 , 1 ] . As shown in Fig. 1 (a), each single-band image in a microscopic hyperspectral image (MHSI) represents the reflectance spectrum information within a specific wavelength range, with different wavelengths highlighting various pathological regions. Therefore, each pixel has a corresponding spectral curve, ensuring that the granularity of both spatial and spectral features is aligned [ 11 , 23 ] . Mining the correlations and complementary information between spectral and spatial data enhance the model’s ability to understand and analyze complex pathological scenes. MHSIs contain asymmetric information in spectral and spatial dimensions. The key challenge for MHSI segmentation is how to effectively align this cross-dimensional information. Recent attempts either learn spectral and spatial features in a sequential or a parallel manner [ 6 ] , and try to fuse features in the encoder. Since the former explicitly processes one dimension at each stage [ 21 , 19 , 20 ] , it is hard to enable the two dimensional features to simultaneously interact with each other. The latter always designs a dual stream to extract spectral and spatial features, respectively, and then followed by post-fusion, either in the late stage of the encoder or in the decoder [ 22 , 16 ] . Spectral and spatial features do not fully guide each other in this way, leading to the loss of complementary information. Though FL [ 6 ] integrates spatial and spectral features for a spatiospectral balance, they simply apply deformable cross-attention twice between the encoder and the decoder. We summarize the network architecture design in Fig. 1 (b) and (c). It shows that none of the above networks provides abundant multi-dimensional feature fusion, thereby leading to sub-optimal results. Figure 1 : (a) Microscopic hyperspectral image (MHSI) of Multi-Dimensional Choledoch Dataset (MDC) dataset. Positive and Negative regions have different spectral curves and single band images at various wavelengths can highlight different texture information. (b)-(d) Structures for spatial-spectral feature fusion. In this paper, we are the first to achieve abundant multi-dimensional feature fusion for MHSI segmentation. Similar with parallel fusion, we separately extract primary spectral and spatial features by a dual stream. Afterwards, we design our spatial-spectral omni-fusion, which makes fusion omnipresent. Our fusion lies in the feature enhancer ( “Cross-dimensional analysis” in Fig. 1 ), the query selection (“Spectral-guided spatial selection” in Fig. 1 ) and the decoder (“Cross-dimension decoder” in Fig. 1 ). For feature enhancer, the primary features mutually enhance each other through two designed bidirectional cross-dimensional analysis blocks, extracting the most important and relevant spatial-spectral feature (see “Cross-Dimensional Feature Enhancement” in Fig. 2 ). The enhanced spatial features are further compacted according to their correlation with the enhanced spectral feature (see “Spectral-Guided Spatial Query Selection” in Fig. 2 ). For decoder, we design a coarse-to-fine learning strategy to extract fine-grained localized features. Specifically, we propose a spectral-spatial decoder for coarse segmentation, using enhanced spectral and spatial features to dynamically guide the model’s attention towards the selected spatial query (see “Spatial-Spectral Decoder” in Fig. 2 ). Then, we refine the coarse mask by incorporating with the selected spatial query (see “Mask Refinement” in Fig. 2 ). Through iterative interactions, spatial and spectral features are deeply and fully fused in this paper. Our contributions are as follows: • We propose an extensive Omni feature fusion network (Omni-Fuse) for microscopic hyperspectral image segmentation, introducing a cross-dimensional enhancement to bidirectionally integrate spatial-spectral features. • We design a two-stage spatial-spectral decoder to coarsely-then-finely segment tumor regions with the assistance of SAM decoder. • Comprehensive experiments are conducted on a public MDC Dataset and a private GPCC Dataset, proving the effectiveness and efficiency of the proposed multi-modal feature fusion based segmentation method. Figure 2 : Framework architecture of Omni-Fuse. The MHSI is primarily processed by a dual steam, followed by a cross-dimensional attention based feature enhancer. And the enhanced spatial features are further compacted by a spectral-guided spatial selection. Subsequently, the compacted features are decoded cooperate with enhanced spatial and spectral features to generate a coarse mask. Finally, the coarse mask is further refined by mask attention mechanism. 2 Method 2.1 Problem Definition and Network Structure Given a pathological microscopic hyperspectral image dataset \mathcal{D}=\{\mathbf{X},\mathbf{Y}\} where \mathbf{X}\in\mathbb{R}^{H\times W\times S} represents the image with spatial resolution of H\times W and spectral band number of S , our goal is to predict its pixel-level segmentation map of H\times W . Fig. 2 illustrates the architecture of the proposed abundant Omni feature Fus ion Network ( Omni-Fuse ) for MHSI segmentation, integrating a cross-dimension feature enhancement module and a two-stage spatial-spectral coarse-to-fine decoder, to predict pixel-level label \hat{\mathbf{Y}}=\{0,1\}\in\mathbb{R}^{H\times W} . We first extract primary features through a simple dual stream, followed by a novel cross-dimensional feature enhancement module to extract the most important and relevant spatial-spectral features. And the spatial feature is compacted by a spectral-guided spatial feature selection block. Finally, a two-stage spatial-spectral decoder is designed for prediction generation from coarse to fine. 2.2 Primary Spatial-Spectral Feature Extraction Due to the fact that microscopic hyperspectral image is a three-dimensional data cube, its data size is significantly larger compared to traditional RGB or grayscale image. In hyperspectral pathology analysis tasks, how to efficiently extract and fuse these multi-dimensional features is a key challenge. To address this issue, we adopt a dual-branch architecture to separately extract spatial information and spectral information. For spatial feature extraction, we apply a Convolutional Neural Network (CNN) to capture local spatial feature F_{\text{CNN}}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C} , where C is the channel number, followed by Swin-Transformer encoder to capture long-range dependencies in the spatial domain, refining the spatial feature to F_{\text{swin}}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C} . For Spectral feature extraction, inspired by MDN [ 12 ] , we employ a bi-directional mamba based block specifically designed for capturing long-distance dependencies across the spectral domain, exporting X_{\text{spec}}\in\mathbb{R}^{B\times H\times W\times S\times L_{\text{spec}}} , where B is