Cosine-Normalized Attention for Hyperspectral Image Classification
Muhammad Ahmad, Manuel Mazzara · 2026
Read Cosine-Normalized Attention for Hyperspectral Image Classification by Muhammad Ahmad, Manuel Mazzara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01763.txt.
Introduction
Ahmad and Mazzara revisit attention scoring for hyperspectral classification, where dot-product similarity mixes magnitude with direction. Their cosine-normalized formula aligns scores with the angular structure of spectral patterns. Query and key vectors project onto a unit hypersphere, and squared cosine similarity stresses angles while damping size effects. In a spatial Transformer tested with few labels, the method beats recent Transformer- and Mamba-based models on three benchmarks despite a backbone. A careful test confirms cosine scoring as a reliable bias for learning.
Catalog record
| CREATOR | Muhammad Ahmad, Manuel Mazzara |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30358 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01763
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cosine-Normalized Attention for Hyperspectral Image Classification Muhammad Ahmad, Manuel Mazzara Abstract Transformer-based methods have improved hyperspectral image classification (HSIC) by modeling long-range spatial–spectral dependencies; however, their attention mechanisms typically rely on dot-product similarity, which mixes feature magnitude and orientation and may be suboptimal for hyperspectral data. This work revisits attention scoring from a geometric perspective and introduces a cosine-normalized attention formulation that aligns similarity computation with the angular structure of hyperspectral signatures. By projecting query and key embeddings onto a unit hypersphere and applying a squared cosine similarity, the proposed method emphasizes angular relationships while reducing sensitivity to magnitude variations. The formulation is integrated into a spatial–spectral Transformer and evaluated under extremely limited supervision. Experiments on three benchmark datasets demonstrate that the proposed approach consistently achieves top-tier performance, outperforming several recent Transformer- and Mamba-based models despite using a lightweight backbone. In addition, a controlled analysis of multiple attention score functions shows that cosine-based scoring provides a reliable inductive bias for hyperspectral representation learning. I Introduction Hyperspectral imaging (HSI) acquires dense spectral signatures for each spatial location in a scene, typically spanning tens to hundreds of contiguous spectral bands [ 2 ] . Owing to this fine spectral resolution, HSI enables detailed characterization of surface materials and supports a wide range of remote sensing applications, including land-cover mapping, environmental monitoring, precision agriculture, and mineral exploration [ 6 , 1 ] . Among these tasks, pixel-wise HSI classification (HSIC) remains one of the most fundamental and widely studied problems, where each pixel is assigned a semantic category based on its spectral and spatial context [ 30 , 21 ] . Despite substantial progress over the past years, HSIC continues to be challenging because hyperspectral data are high dimensional, exhibit strong inter-band redundancy, and are often accompanied by only a limited number of labeled samples [ 43 , 18 ] . \ell_{2} normalize Cosine Normalized Unit hypersphere \theta \tilde{\mathbf{q}} \tilde{\mathbf{k}} Similarity depends only on angle \tilde{\mathbf{q}}^{\top}\tilde{\mathbf{k}}=\cos\theta (\tilde{\mathbf{q}}^{\top}\tilde{\mathbf{k}})^{2}=\cos^{2}\theta \|\tilde{\mathbf{q}}\|_{2}=\|\tilde{\mathbf{k}}\|_{2}=1 Fig. 1 : Standard dot-product attention mixes vector magnitude and angular alignment through \mathbf{q}^{\top}\mathbf{k} . While after \ell_{2} normalization, the embeddings lie on a unit hypersphere and the score depends only on angular similarity. The proposed squared cosine score further sharpens the distinction between strongly aligned and weakly aligned token pairs. Deep learning methods, particularly convolutional neural networks (CNNs), have significantly advanced HSIC by learning hierarchical spectral-spatial representations directly from data [ 23 , 9 , 45 ] . Their success stems from the ability to jointly exploit local neighborhood structure and spectral variation. Nevertheless, CNNs are inherently constrained by local receptive fields, which makes modeling long-range dependencies more difficult without substantially increasing network depth or complexity [ 17 ] . This limitation becomes particularly relevant in HSI, where nonlocal interactions across spatial regions and spectral patterns may carry important semantic information. Recent efforts have therefore explored more expressive architectures for capturing such broader contextual relationships, including efficient spectral-spatial designs and knowledge-distillation-based formulations [ 32 , 15 ] . Transformer architectures offer a natural alternative because self-attention explicitly models interactions between tokens across the entire input [ 38 ] . This capability has motivated a growing number of Transformer-based and hybrid models for HSIC. Recent representative examples include MSST [ 41 ] , S2CIFT [ 40 ] , and S2CAT [ 34 ] , which strengthen spectral-spatial interaction through tailored tokenization and fusion strategies. Other studies have investigated wavelet-assisted and differential attention formulations, such as WaveFormer [ 8 ] and DiffFormer [ 13 ] . In parallel, state-space and Mamba-inspired models have emerged as powerful alternatives for long-range dependency modeling, including SSAM [ 28 ] , differential Mamba-style attention [ 31 ] , DBMLLA [ 27 ] , WDM [ 42 ] , and morphological Mamba variants [ 4 ] . Further developments, such as GraphMamba [ 39 ] , spiking Mamba [ 3 ] , PolicyMamba [ 16 ] , byte latent Mamba with knowledge distillation [ 15 ] , graph-tokenized Mamba [ 12 ] , and domain-adaptive Mamba [ 25 ] , continue to demonstrate the growing importance of sequence modeling paradigms in hyperspectral analysis. Despite their architectural diversity, most of these methods still rely on scaled dot-product attention and closely related inner-product-based similarity measures [ 44 , 10 , 33 , 14 ] . In this formulation, the interaction between query and key tokens is governed by their inner product in the embedding space, meaning that both vector magnitude and orientation affect the final score. While this is computationally convenient and effective in many settings, it implicitly assumes a Euclidean feature geometry that may not be the most appropriate for hyperspectral data. In remote sensing practice, spectral similarity is often evaluated using angular measures because the direction of a spectral signature is frequently more informative than its absolute magnitude. This is the central idea behind metrics such as the spectral angle mapper (SAM), which are known to be more robust to illumination changes and sensor-induced intensity variations. Similar observations have also been discussed in recent hyperspectral modeling work, where angular consistency has been linked to more discriminative spectral representation learning. This mismatch between conventional dot-product attention and the angular nature of hyperspectral signatures motivates a closer examination of attention scoring itself. In many existing models, the attention module is adopted as a standard component, while the geometry induced by the similarity function receives comparatively less focus. However, the score function determines how token relationships are measured, amplified, and propagated through the network. For hyperspectral data, this choice may be especially consequential because spectral magnitude can vary due to acquisition conditions, atmospheric effects, and local illumination, even when the underlying material identity remains unchanged. A similarity function that is less sensitive to such magnitude variations could therefore offer a more suitable inductive bias for HSIC. Another practical challenge in hyperspectral learning is the limited availability of labeled samples [ 26 , 5 ] . Pixel-level annotation of HSIs is expensive, labor-intensive, and often requires expert knowledge, which results in training sets where only a small fraction of pixels are labeled [ 36 , 19 , 29 , 11 , 35 ] . Under such label-scarce conditions, highly expressive models with large parameter counts can overfit easily and may struggle to generalize across scenes or sampling protocols [ 7 , 20 ] . This challenge is particularly relevant for attention-based architectures, whose performance depends not only on backbone design but also on how effectively token relationships are encoded when supervision is extremely limited [ 24 , 22 ] . Consequently, there is strong motivation to design attention mechanisms that are robust, lightweight, and better aligned with hyperspectral feature geometry [ 37 ] . To address this issue, this work revisits attention scoring from a geometric perspective and investigate the role of similarity functions in hyperspectral Transformers. As illustrated in Fig. 1 , we introduce a cosine-normalized attention formulation in which query and key embeddings are first projected onto a unit hypersphere and their similarity is then computed using squared cosine similarity. By normalizing the embeddings, the resulting score emphasizes angular agreement while suppressing the effect of feature magnitude. In this way, the attention mechanism becomes more consistent with the geometry of hyperspectral signatures, where