EHWGesture - A dataset for multimodal understanding of clinical gestures
Gianluca Amprimo, Alberto Ancilotto, Alessandro Savino, Fabio Quazzolo, Claudia Ferraris, Gabriella Olmo, Elisabetta Farella, Stefano Di Carlo · 2025
Read EHWGesture - A dataset for multimodal understanding of clinical gestures by Gianluca Amprimo, Alberto Ancilotto, Alessandro Savino, Fabio Quazzolo, Claudia Ferraris, Gabriella Olmo, Elisabetta Farella, Stefano Di Carlo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07525.txt.
Introduction
Gianluca Amprimo and 7 co-authors release EHWGesture in 2025, a dataset for reading clinical hand gestures from video. Hand gesture recognition supports human-computer interaction. It also supports checks of hand dexterity in clinics. Dynamic gestures are hard to read, since they vary in space and time. Older datasets often lack varied sensor views, precise tracking, or a score for how well a gesture was done.
EHWGesture holds over 1,100 recordings, about six hours in all. They come from 25 healthy subjects who perform five gestures of clinical use. Two RGB-Depth cameras and an event camera record each session. A motion capture system gives precise ground-truth tracking of the hand. All devices are aligned in space and in time. The recordings are grouped by speed of execution, in classes that mirror clinical dexterity scores. First tests cover gesture naming, trigger detection, and action quality scoring.
Catalog record
| CREATOR | Gianluca Amprimo, Alberto Ancilotto, Alessandro Savino, Fabio Quazzolo, Claudia Ferraris, Gabriella Olmo, Elisabetta Farella, Stefano Di Carlo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07525
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EHWGesture - A dataset for multimodal understanding of clinical gestures Gianluca Amprimo, Alberto Ancilotto, Alessandro Savino, Fabio Quazzolo, Claudia Ferraris, Gabriella Olmo, Elisabetta Farella, Stefano Di Carlo Abstract Hand gesture understanding is essential for several applications in human-computer interaction, including automatic clinical assessment of hand dexterity. While deep learning has advanced static gesture recognition, dynamic gesture understanding remains challenging due to complex spatiotemporal variations. Moreover, existing datasets often lack multimodal and multi-view diversity, precise ground-truth tracking, and an action quality component embedded within gestures. This paper introduces EHWGesture, a multimodal video dataset for gesture understanding featuring five clinically relevant gestures. It includes over 1,100 recordings ( ∼ \sim 6 hours), captured from 25 healthy subjects using two high-resolution RGB-Depth cameras and an event camera. A motion capture system provides precise ground-truth hand landmark tracking, and all devices are spatially calibrated and synchronized to ensure cross-modal alignment. Moreover, to embed an action quality task within gesture understanding, collected recordings are organized in classes of execution speed that mirror clinical evaluations of hand dexterity. Baseline experiments highlight the dataset’s potential for gesture classification, gesture trigger detection, and action quality assessment. Thus, EHWGesture can serve as a comprehensive benchmark for advancing multimodal clinical gesture understanding. 1 Introduction Automatic hand gesture recognition is a complex yet significant challenge. Gestures are fundamental to human interaction and, along with facial expressions, constitute the non-verbal component of inter-subject communication [ 19 ] . Computer systems capable of recognizing hand gestures can support various essential tasks. For instance, conventional applications include sign language recognition [ 25 ] , augmented reality [ 30 ] , robotics, and human-computer interaction [ 15 ] . Early gesture recognition methods primarily relied on wearable devices, such as gloves embedded with inertial measurement units [ 24 ] . Nowadays, computer vision methods, especially those based on deep learning [ 14 ] , have become state-of-the-art, offering a non-invasive and more natural interaction. Gestures can be categorized as static or dynamic [ 26 ] . Static hand poses are analyzed in still images, without any temporal evolution of the hand position. Large datasets, especially those based solely on RGB data, can be easily crowdsourced as they require minimal instrumentation ( e.g . , commercial webcams), as demonstrated by the HaGRID dataset by Kapitanov et al . [ 15 ] . Also researchers in dynamic hand gesture understanding have leveraged crowdsourcing to collect large-scale datasets, as done by Materzynska et al . for the Jester dataset [ 18 ] . However, this approach does not allow to capture multimodal data, despite the potential benefits of incorporating additional modalities such as depth or event-based data to handle the complex spatial and temporal variations typical of dynamic gestures. In particular, event-based gesture datasets that combine neuromorphic data with other modalities remain scarce, making cross-modality comparisons challenging. Furthermore, many dynamic gesture datasets rely solely on automated annotation using deep-learning-based hand tracking methods like MediaPipe [ 21 ] . While efficient, this may result in the absence of a strong ground truth, particularly for precise temporal segmentation of gestures. For example, tracking joint positions with a motion capture system would provide more accurate validation and enhance the reliability of gesture understanding benchmarks. Figure 1 : Dynamic gestures included in the dataset and keyframes acquired by the event (center) and RGB-D (left and right) cameras. From top-left: Finger tapping, hand opening and closing, finger-to-nose, and pronation-supination gestures. Finally, dynamic gesture recognition can be extended to gesture understanding , which involves not only classifying gestures within a given time window but also segmenting their triggering phases ( e.g . , the tapping phase in finger pinching) [ 21 ] or assessing the quality of execution of the gesture ( e.g . , recognizing gesture speed) [ 9 ] . These tasks are central in innovative applications such as automatic hand dexterity assessment of clinical gestures. Indeed, a growing trend at the intersection of medicine and computer vision explores hand tracking and gesture recognition to automatically assess motor impairments caused by conditions such as Parkinson’s disease [ 3 ] . This research typically involves multiple stages, including hand detection and tracking in video, gesture recognition to identify specific movements, and action quality assessment (AQA) to quantify performance based on clinical ratings of disease severity [ 9 ] . However, large-scale, publicly available benchmarks in this domain are lacking [ 3 ] . While pathological data are crucial for studying specific disease-related impairments, even extensive datasets from non-pathological subjects could improve the robustness of early-stage gesture recognition and tracking, which continue to face significant accuracy challenges. This work introduces EHWGesture, a large-scale multimodal video dataset for dynamic gesture understanding. The dataset comprises five gestures commonly used in clinical hand dexterity assessments, performed over 1,100 trials, each lasting 20s, for a total of more than six hours of recordings. Gestures were captured using two high-resolution RGB-Depth cameras positioned at orthogonal viewpoints, recording at 30 frame per seconds (fps), resulting in more than 2.6M RGB and depth frames. Additionally, a neuromorphic event camera sampled events at a high frequency (100 MHz). All recordings took place within the tracking volume of a motion capture system, which tracked key hand landmarks involved in the gestures. This gold standard system provides precise ground truth labeling and enables validation against a highly accurate reference. All recording streams were synchronized to maintain temporal alignment across modalities. Spatial calibration data are also provided to facilitate the fusion of different data modalities from a spatial perspective. Finally, some gestures were performed at varying speeds, with volunteers following a metronome. This enables AQA based on execution speed, adding a new dimension to gesture understanding that mimics clinical differences in hand dexterity. This work also presents a series of experiments using the newly collected dataset to establish baseline models for the main addressed tasks. To summarize, its key contributions are: • Providing a large-scale multimodal dataset, integrating synchronized and spatially calibrated RGB, depth, and event-based data. • Offering high-quality ground truth via a motion capture system for precise hand landmark tracking, enabling accurate validation and benchmarking. • Introducing controlled-speed recordings guided by a metronome, allowing for AQA of the recorded gestures based on execution speed, similarly to clinical assessments. • Establishing baseline models on the EHWGesture dataset for gesture classification, temporal segmentation ( i.e . , triggering detection), and AQA . • Investigating the impact of different modalities, input sequence length and frame rate on gesture understanding. Table 1: Comparison of dynamic hand gesture datasets. Acronyms: RGB – Red, Green, Blue, D - Depth, E - Event, GR – Gesture Recognition, GD – Gesture Detection, GT – Gesture Triggering, AQA – Action Quality Assessment, HT – Hand Tracking, HGT - Hand Ground-Truth, SBJ -Subjects, PoV - Point of View, IR - Infra-Red Dataset Videos SBJ Classes Frames Modality & Resolution (px) HGT Tasks PoV Views PD4T [ 9 ] 1,654 30 2 - RGB: 856 \times 480 - GR, AQA 3rd 1 EgoGesture [ 30 ] 2,081 50 83 2,953,224 RGB, D: 640 \times 480 - GR, GD 1st 1 FHANDS [ 10 ] 1,175 6 27 105,459 RGB: 1920 \times 1080, D: 640 \times 480 ✓ GR, HT 1st 1 IPN Hand [