SSFT: A Lightweight Spectral–Spatial Fusion Transformer for Generic Hyperspectral Classification
Alexander Musiat, Nikolas Ebert, Oliver Wasenmüller · 2026
Read SSFT: A Lightweight Spectral–Spatial Fusion Transformer for Generic Hyperspectral Classification by Alexander Musiat, Nikolas Ebert, Oliver Wasenmüller (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15828.txt.
Introduction
Musiat, Ebert, and Wasenmüller propose SSFT, a compact transformer that sorts hyperspectral images, which record many wavelengths per pixel, into material classes. It splits learning into spectral and spatial paths joined by cross-attention, so wavelength patterns and shapes inform each other. Tested on a mixed benchmark spanning Earth observation, fruit condition, and fine material recognition, it ranked first while using under 2 percent of the parameters of the previous leader.
Catalog record
| CREATOR | Alexander Musiat, Nikolas Ebert, Oliver Wasenmüller |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38229 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SSFT: A Lightweight Spectral–Spatial Fusion Transformer for Generic Hyperspectral Classification Alexander Musiat, Nikolas Ebert, Oliver Wasenmüller Abstract Hyperspectral imaging enables fine-grained recognition of materials by capturing rich spectral signatures, but learning robust classifiers is challenging due to high dimensionality, spectral redundancy, limited labeled data, and strong domain shifts. Beyond earth observation, labeled HSI data is often scarce and imbalanced, motivating compact models for generic hyperspectral classification across diverse acquisition regimes. We propose the lightweight Spectral-Spatial Fusion Transformer (SSFT), which factorizes representation learning into spectral and spatial pathways and integrates them via cross-attention to capture complementary wavelength-dependent and structural information. We evaluate our SSFT on the challenging HSI-Benchmark, a heterogeneous multi-dataset benchmark covering earth observation, fruit condition assessment, and fine-grained material recognition. SSFT achieves state-of-the-art overall performance, ranking first while using less than 2% of the parameters of the previous leading method. We further evaluate transfer to the substantially larger SpectralEarth benchmark under the official protocol, where SSFT remains competitive despite its compact size. Ablation studies show that both spectral and spatial pathways are crucial, with spatial modeling contributing most, and that SSFT remains robust without data augmentation. Figure 1 : To demonstrate the general applicability of SSFT, we evaluate on the heterogeneous sub-datasets of the HSI-Benchmark 13 , covering earth observation, fruit condition assessment, and fine-grained debris material recognition. The figure shows representative image patches and key dataset characteristics for each domain. We further compare the top-performing methods 36 ; 44 ; 45 per domain in terms of accuracy (bars, left axis) and model size (lollipops, right axis). Across this diverse multi-domain protocol, SSFT achieves state-of-the-art overall performance, ranking first on fruit and debris while remaining competitive on earth observation despite its compact architecture using less than 2% of the parameters of the previous leading method. 1 Introduction Hyperspectral imaging (HSI) enables fine-grained discrimination of materials and physical states by capturing rich spectral signatures for each pixel across dozens to hundreds of narrow bands. This makes HSI a powerful sensing modality for applications such as earth observation, agriculture, environmental monitoring, and quality assessment. At the same time, HSI datasets are inherently diverse: measurements vary across sensors and optics, spectral coverage and band spacing, spatial resolution, illumination conditions, and calibration and preprocessing, creating substantial domain shifts across application regimes. Moreover, beyond earth observation, annotated HSI datasets are typically small and class-imbalanced, limiting supervision and amplifying overfitting in real-world deployments. Most modern HSI classifiers build on deep backbones adapted from RGB image recognition 1 . Early work 29 ; 21 largely relied on 3D convolutions to jointly model spectral-spatial correlations, while later approaches employed 2D CNNs with spectral mixing 50 , attention mechanisms 18 , or transformer encoders 19 ; 22 to capture wider context. This progression has improved representational power, but it has also shifted the field toward increasingly large and computationally expensive architectures. While such models can perform well when labeled data is abundant and train/test conditions are closely matched, these assumptions often do not hold in HSI. In diverse low-data regimes, high-capacity models may instead exploit dataset-specific statistics or sensor and preprocessing artifacts rather than transferable spectral-spatial cues. Moreover, augmentation strategies commonly used in RGB vision do not transfer directly to HSI, since arbitrary spectral perturbations can violate acquisition physics and introduce unrealistic shortcuts. To address these challenges, we propose the Spectral-Spatial Fusion Transformer (SSFT), a compact architecture tailored to heterogeneous, low-supervision HSI regimes. SSFT processes spectral signatures and spatial structure in separate lightweight branches and integrates them through cross-attention, enabling controlled interaction between complementary cues without resorting to large backbones. With only 516k parameters, SSFT is orders of magnitude smaller than common CNN and transformer baselines 36 ; 47 , yet achieves the best overall score on HSI-Benchmark 13 (see Fig. 1 ), ranking first on Fruit and Debris while remaining competitive on earth observation. We further evaluate transfer to the substantially larger SpectralEarth benchmark 3 under the official protocol, where SSFT remains competitive despite its compact size. Finally, we complement the main results with controlled branch ablations, auxiliary supervision experiments, and an analysis of data augmentation, showing that both branches are important and that robustness in HSI cannot be assumed from RGB training recipes. 2 Related Works 2.1 Hyperspectral Image Classification Hyperspectral images are naturally represented as a spectral-spatial data cube, where each pixel is not a three-channel color vector but a densely sampled reflectance spectrum measured across dozens to hundreds of narrow wavelengths. These spectra often carry subtle absorption patterns tied to material composition and physical condition, but they are also highly structured: neighboring bands are strongly correlated, signal-to-noise varies across the spectrum, and measurements depend on acquisition factors such as optics, illumination, and calibration. As a result, hyperspectral classification is not merely a higher-dimensional version of RGB recognition; it is a data-efficient learning problem that must extract transferable spectral cues while exploiting spatial context, typically under limited and costly supervision 26 ; 15 ; 1 . A large fraction of widely used hyperspectral benchmarks stems from satellite earth observation, where a small set of canonical scenes has historically driven progress and where more recent releases move toward larger-scale, multi-scene evaluation 17 ; 8 ; 3 . However, beyond earth observation, many close-range hyperspectral applications in areas such as agriculture, food quality inspection, or environmental sensing rely on datasets that are typically smaller, more class-imbalanced, and collected under varied acquisition setups, which amplifies domain shift and makes results harder to compare across studies 14 ; 34 . To better reflect these real-world conditions, recent efforts have introduced multi-domain benchmarks that combine datasets from distinct application regimes and enforce standardized protocols, enabling systematic assessment of transfer across domains rather than performance tuned to a single sensor or scene 13 ; 6 . Deep hyperspectral classifiers can be grouped by where they mix information in the cube. Spectral models treat each pixel spectrum as a 1D signal and learn cross-band patterns along the wavelength axis 49 ; 37 . They are data-efficient and preserve fine spectral detail, but largely ignore spatial context that is often essential for disambiguating classes. Spatial CNN pipelines instead apply 2D convolutions on spatial patches, handling the spectral dimension either via projected cube representations ( e.g . , band selection or PCA channels) or via learned spectral mixing ( e.g . , 1{\times}1 convolutions) before 2D processing, thereby leveraging texture and local structure 48 ; 35 . 3D CNNs avoid this trade-off by convolving directly over spectral–spatial patches 52 ; 24 ; 12 , but their dense joint processing is computationally heavy and can overfit small benchmarks. Attention-based models provide an alternative route to capture non-local spectral-spatial dependencies 22 ; 4 . However, applying self-attention to high-dimensional hyperspectral tokens can be similarly costly, and vanilla transformer backbones are often too parameter- and compute-intensive for small, heterogeneous HSI regimes. Several recent works therefore combine spectral and spatial modeling more explicitly through dedicated fusion mechanisms. For example, spectral-spatial transformer architectures have been proposed to jointly encode both modalities and enhance cross-dimensional interaction 30 ; 42 , while lightweight fusion networks combine separate spectral and spatial feature extractors followed by feature fusion to improve efficiency 5 . Across these model families, the recurring tension is between expressive