Efficient Egocentric Action Recognition with Multimodal Data
Marco Calzavara, Ard Kastrati, Matteo Macchini, Dushan Vasilevski, Roger Wattenhofer · 2025
Read Efficient Egocentric Action Recognition with Multimodal Data by Marco Calzavara, Ard Kastrati, Matteo Macchini, Dushan Vasilevski, Roger Wattenhofer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01757.txt.
Introduction
Marco Calzavara and four co-authors study action recognition from wearable devices in a 2025 paper. Head-worn gear can watch what the wearer does, giving systems a first-person view of activity. Running such recognition live is hard because portable devices trade off battery, size, and computing power. The team varied how often two input streams are sampled: video and three-dimensional hand pose.
Cutting the video frame rate while keeping hand pose frequent preserved accuracy and slashed processor use. In some setups, processor demand fell about threefold with little or no loss in recognition. The authors present multi-stream input as a practical route to real-time recognition on extended reality devices.
Catalog record
| CREATOR | Marco Calzavara, Ard Kastrati, Matteo Macchini, Dushan Vasilevski, Roger Wattenhofer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Egocentric Action Recognition with Multimodal Data Marco Calzavara, Ard Kastrati, Matteo Macchini, Dushan Vasilevski, Roger Wattenhofer Abstract The increasing availability of wearable XR devices opens new perspectives for Egocentric Action Recognition (EAR) systems, which can provide deeper human understanding and situation awareness. However, deploying real-time algorithms on these devices can be challenging due to the inherent trade-offs between portability, battery life, and computational resources. In this work, we systematically analyze the impact of sampling frequency across different input modalities—RGB video and 3D hand pose—on egocentric action recognition performance and CPU usage. By exploring a range of configurations, we provide a comprehensive characterization of the trade-offs between accuracy and computational efficiency. Our findings reveal that reducing the sampling rate of RGB frames, when complemented with higher-frequency 3D hand pose input, can preserve high accuracy while significantly lowering CPU demands. Notably, we observe up to a 3× reduction in CPU usage with minimal to no loss in recognition performance. This highlights the potential of multimodal input strategies as a viable approach to achieving efficient, real-time EAR on XR devices. 1 Introduction Since the advent of head-mounted devices, researchers and industry have shown growing interest in leveraging video captured by wearable cameras to extract valuable information. Among these devices, AR glasses such as Magic Leap 2 1 1 1 https://www.magicleap.com/magic-leap-2 stand out by integrating digital content into the physical world. The data they capture can unlock numerous applications, from task assistance to personalized recommendations and contextual awareness. Egocentric Action Recognition (EAR) is a research field focused on identifying human actions from first-person data. It presents unique challenges, including variable viewpoints, frequent occlusions, and the computational constraints of wearable devices. On AR glasses, efficiency is especially critical, not only to preserve battery life but also because these devices typically lack dedicated hardware acceleration and must rely on limited CPU resources. This project tackles these challenges by targeting efficient and accurate real-time action recognition for AR glasses. EAR methods are typically categorized as either uni-modal or multi-modal. Several studies rely solely on RGB data for EAR. For example, Pirsiavash and Ramanan 2012 adopts a Bag-of-Objects approach, while Kapidis et al. 2020 uses frame-level features derived from object detections. Other methods extract features directly from RGB frames. Among these, Sudhakaran and Lanz 2017 combines 2D and 3D CNNs with a ConvLSTM Shi et al. 2015 , also testing frame differences instead of raw frames. Although RGB-based systems perform well, some studies explore whether hand pose alone is sufficient. For instance, Das and Ortega 2021 uses a Spatio-Temporal Graph Convolutional Network on hand skeletons, which is inspired by Yan et al. 2018 . These results highlight a strong link between hand motion and egocentric actions. The main drawback of uni-modal systems is their inability to capture diverse cues, such as object identity or hand-object interactions. Multi-modal models address this limitation by combining multiple inputs. A popular example is the two-stream architecture from Simonyan and Zisserman 2014 , which processes RGB and optical flow separately. In EAR, gaze-based attention is used in Li et al. 2018 , while Sudhakaran and Lanz 2018 shows improved performance by adding an optical flow branch to Sudhakaran and Lanz 2017 . Beyond RGB and flow, other modalities have also been explored; for instance, Shamil et al. 2025 uses RGB with 3D hand poses, avoiding the high computational cost of optical flow. Three-stream models extend this further: Furnari and Farinella 2019 combines RGB, optical flow, and object features, showing better results than any single stream. However, as noted in Wang et al. 2023 , more modalities do not always lead to better performance. Building upon previous work, our approach investigates a targeted two-stream architecture that utilizes RGB frames and 3D hand pose keypoints to predict the current action, aiming to optimize both accuracy and resource efficiency. This paper makes the following contributions: • We introduce a multi-modal EAR system that combines an RGB stream comprising a Vision Transformer (ViT) feature extractor with a hand pose stream. • We present a systematic study of the interplay between sampling frequency, recognition accuracy, and computational cost across modalities. Our analysis reveals how adjusting the relative sampling rates of RGB and hand pose inputs can serve as a design lever to optimize EAR systems for resource-constrained devices. • We demonstrate that downsampling RGB input — when complemented with higher-frequency hand pose signals — preserves accuracy while significantly improving efficiency. Specifically, we show that maintaining hand pose input at 30 Hz while reducing RGB input to 10 Hz achieves competitive accuracy with up to a 3× reduction in CPU usage compared to full-frame-rate RGB baselines. These results highlight the practical value of modality-aware sampling strategies for efficient on-device deployment of EAR systems. 2 Methods 2.1 Model Operating over sequences is crucial for accurate action recognition, as context from multiple consecutive time steps helps detect motion. In this work, we employ a multi-stream architecture consisting of modality-specific feature extractors and sequence models. The model, depicted in Figure 1(a) , processes the two modalities through dedicated modules: a LeViT Graham et al. 2021 feature extractor and Temporal MLP for RGB features and a hand pose feature extractor paired with a Temporal MLP for hand pose features. The outputs from both streams, corresponding to the final time steps, are concatenated and processed through a series of layers for classification. A key component of our approach is the Temporal MLP, which is responsible for modeling temporal dependencies in the feature representations. The architecture of the Temporal MLP, shown in Figure 1(b) , is inspired by Du et al. 2023 and is designed to efficiently capture long-range dependencies without relying on convolutional layers with large kernel sizes. (a) Architecture of the Multimodal Temporal MLP. The model consists of two parallel processing streams: one for RGB features, utilizing a LeViT feature extractor and a Temporal MLP, and another for hand pose features, employing an MLP feature extractor paired with a Temporal MLP. The outputs from both streams at the final time step are concatenated and processed through additional layers for classification. (b) Architecture of the Temporal MLP. This module captures long-range dependencies in temporal sequences without relying on convolutional layers with large kernel sizes. Inspired by Du et al. 2023 , it employs MLP-based operations to model temporal relationships efficiently, making it well-suited for sequence-based tasks such as action recognition. Figure 1 : Architectures of the Multimodal Temporal MLP and Temporal MLP. 2.2 Experimental Conditions Model training was performed using either two or four GPUs, depending on availability. The GPUs used for training included the Titan RTX (24 GB memory), GeForce RTX 3090 (24 GB memory), and Tesla V100 (32 GB memory). For CPU usage, a single thread of an AMD EPYC 7742 CPU (base clock: 2.25 GHz) was utilized. 2.3 Dataset We conducted our experiments on the training and validation sets of the H2O dataset Kwon et al. 2021 . While the H2O dataset is designed for predicting the action performed over a video segment, it can also be used for frame-level prediction by assigning the segment’s action label to each frame within the segment. 2.4 Preprocessing and Augmentation RGB frames are cropped and normalized prior to feature extraction. For 3D hand