Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Juan Ignacio Bustos Gorostegui, Maria Elena Buemi, [0.5em] Univ. of Buenos Aires. Faculty of Exact, Natural Sciences. Dept. of Computer Science (DC) · 2026
Read Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies by Juan Ignacio Bustos Gorostegui, Maria Elena Buemi, [0.5em] Univ. of Buenos Aires. Faculty of Exact, Natural Sciences. Dept. of Computer Science (DC) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24302.txt.
Introduction
Bustos and Buemi propose a cross-modal architecture for egocentric action recognition fusing RGB video with hand-skeleton data in a Mamba frame. Erratic camera motion, frequent hand occlusion, and drifting visuals make first-person action recognition hard. Their design pairs a VideoMamba visual module with a skeleton encoder of Mamba blocks, plus a fusion module. The central contribution is four CLS token mixing strategies, Naive, Average, Weighted, and Context-based. On H2O, Average wins, beating VideoMamba by over 10 percent Top-1 accuracy in Tiny and 2 percent in Small.
Catalog record
| CREATOR | Juan Ignacio Bustos Gorostegui, Maria Elena Buemi, [0.5em] Univ. of Buenos Aires. Faculty of Exact, Natural Sciences. Dept. of Computer Science (DC) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14511 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies Juan Ignacio Bustos Gorostegui, Maria Elena Buemi, [0.5em] Univ. of Buenos Aires. Faculty of Exact, Natural Sciences. Dept. of Computer Science (DC) Abstract Egocentric action recognition is a challenging task due to erratic camera motion, frequent hand occlusion, and the difficulty of maintaining consistent visual representations over time. In this work, we propose a cross-modal architecture that combines RGB video and temporal hand skeleton data within a unified Mamba-based framework, exploiting the linear time complexity of State Space Models (SSMs). Our architecture consists of three components: a VideoMamba module for visual feature extraction, a skeleton encoder built on a stack of Mamba blocks, and a fusion module that integrates both modalities into a single representation. A central contribution of this work is the design and evaluation of four Class (CLS) token mixing strategies for multimodal fusion: Naive, Average, Weighted and Context-based. These strategies differ in how the pretrained unimodal CLS tokens, which role is to act as information sinks concentrating learned representations, are leveraged to initialize the mixed CLS token used for final classification. We evaluate all strategies on the H2O dataset. Experimental results show that the Average strategy achieves the best performance, yielding gains of over 10% Top-1 accuracy in the Tiny configuration and 25% in the Small configuration over the VideoMamba baseline. 1 Introduction Egocentric videos face several challenges that fixed cameras typically avoid, such as erratic camera motion, frequent occlusion by the hands, and the possibility of temporarily losing visual contact with the main action. For this reason, a flexible architecture capable of maintaining a global video representation while being continuously updated with noisy or misleading frames is required. These challenges make Transformers 8 particularly well suited for egocentric video understanding, as their self-attention mechanism enables the construction of strong spatio-temporal representations and allows the model to retain important visual information over long sequences, even in the presence of extended periods of misleading input. In addition, the ability of Transformers to incorporate multimodal data into their representations allows skeletal motion information to be used to improve action prediction. Temporal hand skeleton data not only helps mitigate occlusion issues, such as when fingers are hidden behind objects or the hand itself, but also provides clean motion cues related to the performed action, thereby enhancing the overall understanding. This has become increasingly easier with the growing availability of egocentric datasets that provide synchronized hand skeleton annotations alongside the video stream 3 ; 6 ; 10 , most notably the H2O and OakInkV2 dataset 6 ; 10 , which both captures a diverse set of everyday bimanual interactions (pouring, cutting, assembling, etc) from a first-person perspective with ground-truth hand keypoint annotations, making it particularly well suited for evaluating action recognition models in realistic daily-life scenarios. Although these properties make Transformers a natural fit for egocentric video understanding, unfortunately the self-attention mechanism scales quadratically with sequence length, which limits the feasibility of Transformers for online video understanding applications. For this reason, Mamba 5 , a State Space Model (SSM)-based architecture, has recently gained attention due to its linear time complexity and performance comparable to Transformers in NLP tasks. Extensions of Mamba to visual domains, such as Vision Mamba 11 and Video Mamba 7 , have demonstrated promising results in image and video processing. Although skeleton-based action recognition models using Mamba have been explored 9 ; 2 , to the best of our knowledge, no prior work has integrated skeleton-based information with RGB streams within this framework. We address this gap by proposing a cross-modal Mamba architecture and systematically evaluating four strategies for fusing pretrained unimodal Class(CLS) token representations into a unified classification token. 2 Cross-Modal Fusion Design 2.1 Two-Branch Architecture The overall architecture is divided into three main components: (1) a VideoMamba module to extract embeddings from the video stream, (2) a pure Mamba module to encode skeleton data, and (3) a fusion module that combines both information streams into a unified representation, as can be seen in Fig. 1 . Both the video and skeleton embeddings share the same dimensions. However, the number of tokens differs between modalities, as the video stream produces multiple tokens per frame due to spatial patchification. Figure 1 : Overview of the proposed cross-modal architecture. The skeleton stream (top) and video stream (bottom) are encoded independently before being fused into a unified Mixed CLS token for classification. T denotes the number of input frames, H and W the height and width of each frame respectively, and C the embedding dimension. The fusion module is inspired by the Mamba Suite Cross-Modal implementation 1 . They add a learnable modality embedding to each modality-specific representation before concatenating them into a single sequence. Instead of keeping the original CLS tokens from each stream, they remove both and introduce a new one. The concatenated sequence is then processed through a Mamba block, and its final value is used for classification. Our approach differs from the original implementation in how we handle the two modality-specific CLS tokens. In Mamba models, as in Transformers, it acts as an information sink, concentrating the relevant contextual information throughout processing before passing it to the classification head. By using pretrained video and skeleton encoders, their respective CLS tokens already contain dense and structured information. Discarding them entirely would waste this learned behavior. For this reason, we propose four strategies to initialize the new CLS token, which we are referring to as the Mixed CLS token , by leveraging the pretrained representations. A visual representation is shown in Fig. 2 . 2.2 CLS Mixing Strategies Naive. Both modality-specific CLS tokens are discarded, and a new trainable one is used. Average. The Mixed CLS token is initialized as the average by dimension of the video and skeleton CLS tokens. Weighted. A learnable scalar parameter \omega is passed through a sigmoid function to compute a linear combination of the two CLS tokens: \text{CLS}_{mix}=\alpha\,\text{CLS}_{video}+(1-\alpha)\,\text{CLS}_{skeleton}, (1) where \alpha=\sigma(\omega) . Context-based. Instead of learning a static scalar weight, we compute \omega dynamically using a Mamba block that processes the concatenated token representations (excluding both CLS tokens). Specifically: \alpha=\sigma\!\left(\text{Mean}\!\left(\text{Mamba}(\mathbf{T}_{video}\oplus\mathbf{T}_{skel})\right)\right), (2) where \mathbf{T}_{video} and \mathbf{T}_{skel} denote the non-CLS token sequences from each modality, \oplus denotes concatenation, and \text{Mean}(\cdot) denotes taking the mean over the embedding values. The resulting \alpha is then used in Eq. 1 to combine the two CLS tokens, allowing the model to adapt the fusion weight based on the input content. Figure 2 : The four proposed CLS mixing strategies. From top to bottom: Naive (random initialization), Average (equal combination), Weighted (learnable scalar), and Context-based (input-dependent weight computed via a Mamba block). C denotes the embedding dimension. 3 Experiments and Results 3.1 Experimental Setup To evaluate the cross-modal capabilities of Mamba and the proposed fusion strategies, we conducted experiments on the H2O dataset 6 , an egocentric video dataset that provides not only action labels but also ground-truth hand keypoints for each frame. For both the VideoMamba module and the skeleton encoder, we adopted the same configurations defined in the VideoMamba model variants 7 , summarized in Table 1 . Table 1 : Model configurations used for both the video