Hyperspectral Mamba for Hyperspectral Object Tracking
Long Gao, Yunhe Zhang, Yan Jiang, Weiying Xie, Yunsong Li · 2025
Read Hyperspectral Mamba for Hyperspectral Object Tracking by Long Gao, Yunhe Zhang, Yan Jiang, Weiying Xie, Yunsong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08265.txt.
Introduction
Tracking an object in hyperspectral images, which record many narrow color bands, promises robustness when ordinary cameras fail. Long Gao and 4 co-authors propose HyMamba in a 2025 research paper. Earlier trackers converted the data to false-color images or fused modes in ways that lost spectral detail. Temporal links and interactions across network depth were also underused.
HyMamba unifies spectral, cross-depth, and temporal modeling with state space modules, an efficient sequence modeling family. A spectral integration module refines features progressively across depth and time. Inside it, a dedicated module scans in three directions to learn space and spectrum together. Tests on seven benchmark datasets put HyMamba at state-of-the-art level. Rich material signatures, it turns out, help most when the tracker actually keeps them intact.
Catalog record
| CREATOR | Long Gao, Yunhe Zhang, Yan Jiang, Weiying Xie, Yunsong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hyperspectral Mamba for Hyperspectral Object Tracking Long Gao, Yunhe Zhang, Yan Jiang, Weiying Xie, Yunsong Li Abstract Hyperspectral object tracking holds great promise due to the rich spectral information and fine-grained material distinctions in hyperspectral images, which are beneficial in challenging scenarios. While existing hyperspectral trackers have made progress by either transforming hyperspectral data into false-color images or incorporating modality fusion strategies, they often fail to capture the intrinsic spectral information, temporal dependencies, and cross-depth interactions. To address these limitations, a new hyperspectral object tracking network equipped with Mamba (HyMamba), is proposed. It unifies spectral, cross-depth, and temporal modeling through state space modules (SSMs). The core of HyMamba lies in the Spectral State Integration (SSI) module, which enables progressive refinement and propagation of spectral features with cross-depth and temporal spectral information. Embedded within each SSI, the Hyperspectral Mamba (HSM) module is introduced to learn spatial and spectral information synchronously via three directional scanning SSMs. Based on SSI and HSM, HyMamba constructs joint features from false-color and hyperspectral inputs, and enhances them through interaction with original spectral features extracted from raw hyperspectral images. Extensive experiments conducted on seven benchmark datasets demonstrate that HyMamba achieves state-of-the-art performance. For instance, it achieves 73.0% of the AUC score and 96.3% of the DP@20 score on the HOTC2020 dataset. The code will be released at https://github.com/lgao001/HyMamba. I Introduction Hyperspectral object tracking has emerged as a promising field in visual object tracking, owing to the rich spectral cues encoded within hyperspectral (HS) images, which reveal intrinsic material properties of the target [ 1 ] . While conventional trackers based on RGB videos have achieved notable advancements in recent years, their performance deteriorates under adverse conditions such as background clutter, occlusion, and deformation [ 2 , 3 ] . In contrast, HS data offers substantially higher spectral resolution, enabling more precise discrimination between objects of similar appearance but differing in spectral composition [ 4 ] . This advantage has increasingly attracted attention to HS object tracking, especially in complex scenarios where conventional RGB trackers are prone to failure. Most HS trackers are developed based on deep learning methods, and their performance is significantly hindered by the scarcity of large-scale, well-annotated HS video datasets. Acquiring such data is often prohibitively costly, which makes training robust models from scratch infeasible [ 5 ] . To circumvent this limitation, a prevalent strategy is to leverage powerful RGB-based tracking networks pretrained on extensive RGB datasets and subsequently adapt them to the HS domain through fine-tuning [ 6 , 7 ] . In this paradigm, HS images are typically converted into formats compatible with RGB networks, either by selecting a subset of spectral bands to form a single three-channel false-color image [ 8 , 9 ] , or by regrouping the bands into multiple false-color images that preserve more spectral diversity [ 10 , 11 ] . These transformed images are then fed into pretrained networks, enabling knowledge transfer from the RGB domain while partially retaining the unique spectral cues of the HS data [ 12 , 13 , 14 ] . Recent advances have further explored transformer-based tracking architectures in this context, owing to their superior capacity for modeling long-range dependencies. Specifically, transformer-based RGB tracking networks are applied to extract or enhance features from band regrouping images [ 5 , 6 ] . Superb performance is achieved by the transformer-based HS trackers due to the utilization of more spectral information. Despite the progress, the development of existing HS object tracking frameworks is restricted by two crucial issues. Firstly, band selection or regrouping strategies inherently compromise spectral integrity by discarding fine-grained inter-band correlations and material-specific signatures [ 4 , 13 ] . Specifically, the band selection strategy compresses the abundant spectral information into three-channel false-color images, leading to the loss of spectral information. Furthermore, the band regrouping strategy breaks the correlation information between bands in HS images. Secondly, current HS tracking frameworks fail to propagate the spectral features across network depths and consecutive frames. Using multi-level deep features is critical for complex scenarios, e.g., deformation and background clutter [ 14 ] . However, existing methods lack cross-depth interaction information learning, and the layer-wise spectral semantics remain isolated. Moreover, the temporal coherence of spectral signatures is overlooked, as most trackers process frames independently, neglecting how spectral continuity could stabilize tracking during occlusions or fast motion. Recently, Mamba shows promise for sequential modeling [ 15 ] , yet existing applications in object tracking focus on intra-frame refinement [ 16 ] or inter-frame dynamics [ 17 ] , failing to unify the cross-depth and temporal dependencies. Specifically, for the HS object tracking task, modeling spectral information across network depths and consecutive frames with a unified Mamba-based module has not been explored. Moreover, traditional Mamba-based methods scan tokens in spatial domain and lack spectral-specific adaptation for HS object tracking task [ 18 , 19 ] . Specifically, the scanning strategies in existing Mamba run forward or backward, which fails to synchronize spatial scanning with spectral correlation mining, leading to suboptimal spectral-spatial fusion. The resulting gap between general state space modules (SSMs) and HS demands necessitates a structured scanning strategy, which explicitly integrates forward-backward spatial modeling with spectral channel-wise processing. To overcome these challenges, a new Mamba-based HS object tracking framework, HyMamba, is introduced, which encodes both intra-frame cross-depth spectral information and inter-frame temporal spectral information with the proposed methods. Firstly, the spectral state integration (SSI) module is designed to facilitate the joint modeling of cross-depth and temporal spectral information. SSI, which consists of a Mamba module, a joint augment (JA) module, and a spectral augment (SA) module, is deployed between transformer layers. The Mamba module is utilized to learn the semantic information from the unconverted HS image, avoiding spectral information loss in existing methods. Meanwhile, SSI iteratively updates a recurrent spectral hidden state across transformer layers and propagates temporal information across frames, achieving intra-frame cross-depth spectral information and inter-frame temporal spectral information learning. JA and SA are applied to augment the spectral information learned from the unconverted HS image and the feature extracted by the transformer layer in bidirectional manner. Secondly, hyperspectral Mamba (HSM) is proposed to further empower SSI. To learn spatial and spectral information synchronously, HSM scans the feature extracted from the unconverted HS image along forward, backward, and channel directions. Moreover, HSM augments the information learned via three-directional scanning into the features extracted by the transformer layer. Thirdly, based on SSI and HSM, HyMamba is introduced to learn cross-depth and temporal spectral information, resulting in more discriminative and temporally consistent HS tracking. HyMamba directly models cross-depth and temporal spectral information from HS images, overcoming the crucial limitations of existing HS object tracking methods. To validate the efficacy of HyMamba, we conduct comprehensive experiments on seven HS tracking benchmarks, HOTC2020 [ 1 ] , VIS2023 [ 20 ] , NIR2023 [ 20 ] , RedNIR2023 [ 20 ] , VIS2024 [ 21 ] , NIR2024 [ 21 ] , and RedNIR2024 [ 21 ] . These datasets span a diverse range of spectral domains and scene complexities. Across all benchmarks, HyMamba consistently achieves state-of-the-art (SOTA) performance, significantly outperforming both RGB-based and existing HS trackers in terms of accuracy and precision. The core contributions of