MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion

Jihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, Dan Guo · 2025

Read MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion by Jihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, Dan Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08344.txt.

Introduction

Jihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, and Dan Guo win a micro-gesture recognition contest by fusing many views of the hand, in a 2025 paper. Micro-gestures are tiny, brief movements that standard cameras easily miss. Their system, MM-Gesture, blends six streams: joint positions, limb shapes, plain video, two motion-enhanced video forms, and depth video. Two proven network designs read the streams, and a weighted vote combines them.

Pretraining the video stream on a larger gesture dataset sharpened it further. On the standard iMiGUE benchmark the system hit 73.2 percent top-one accuracy, taking first place in its challenge track at a major AI conference. Tests removing one stream at a time confirmed each added value. The code is public.

Catalog record

CREATORJihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, Dan Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion Jihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, Dan Guo Abstract In this paper, we present MM-Gesture, the solution developed by our team HFUT-VUT, which ranked 1st in the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025, achieving superior performance compared to previous state-of-the-art methods. MM-Gesture is a multimodal fusion framework designed specifically for recognizing subtle and short-duration micro-gestures (MGs), integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities. Utilizing PoseConv3D and Video Swin Transformer architectures with a novel modality-weighted ensemble strategy, our method further enhances RGB modality performance through transfer learning pre-trained on the larger MA-52 dataset. Extensive experiments on the iMiGUE benchmark, including ablation studies across different modalities, validate the effectiveness of our proposed approach, achieving a top-1 accuracy of 73.213%. [orcid=0009-0009-0141-4807, [email protected], ] [orcid=0009-0004-1142-6434, [email protected], ] [orcid=0000-0001-5083-2145, [email protected], ] *1 [orcid=0000-0001-8818-6740, [email protected], ] [orcid=0000-0002-6597-8048, [email protected], ] [orcid=0000-0003-2594-254X, [email protected], ] keywords Micro-Gesture ,Action Recognition ,Multi-modal ,Ensemble Fusion ,Transfer Learning † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: MiGA@IJCAI25: International IJCAI Workshop on 3rd Human Behavior Analysis for Emotion Understanding, August 29, 2025, Guangzhou, China. † † address: University College London (UCL), Gower Street, London, WC1E 6BT, UK † † address: School of Computer Science and Information Engineering, School of Artificial Intelligence, Hefei University of Technology (HFUT) † † address: ReLER, CCAI, Zhejiang University, China † † address: Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education † † address: Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China † † corresp: Corresponding author. 1 Introduction Micro‑Gestures (MGs) Chen et al. 2019 ; Liu et al. 2021 ; Chen et al. 2023 ; Chen et al. 2024 , defined as spontaneous and fine‑grained movements, such as nose touching, hair scratching, or subtle finger rubs, encode rich affective and cognitive cues that rarely surface in conventional action recognition benchmarks. Compared with conventional actions Li et al. 2021 ; Wang et al. 2025 ; Balazia et al. 2022 ; Li et al. 2025 , MGs are unintentional, short‑duration, and confined to small body regions, which makes them extremely difficult to capture and classify. Due to the subtle changes and short duration of MGs, relying solely on a single modality ( e.g . , RGB Liu et al. 2022 ; Li et al. 2025 ; Guo et al. 2024 , skeleton Li et al. 2023 ; Huang et al. 2023 ) often captures merely partial characteristics of MGs, thus failing to fully and thoroughly exploit the comprehensive information latent in available data. Despite significant advancements in previous studies on micro-gesture and micro-action recognition Guo et al. 2024 ; Li et al. 2025 ; Li et al. 2024 ; Li et al. 2023 ; Gu et al. 2025 ; Sun et al. 2023 ; Dong et al. 2023 , most existing approaches remain confined to utilizing limited modalities, such as RGB combined with skeleton data Chen et al. 2024 ; Huang et al. 2024 ; Duan et al. 2022 . However, these methods have not sufficiently leveraged the abundant and complementary information conveyed by multi-modal. In this work, we propose a novel multi-modal fusion framework MM‑Gesture tailored explicitly for the challenging task of MGs classification. Specifically, we construct baseline models leveraging PoseConv3D Duan et al. 2022 and Video Swin Transformer Liu et al. 2022 ; Wang et al. 2025 , integrating information across six complementary modalities: joint, limb, RGB video, Taylor video, optical flow video, and depth video . In addition, to enhance the performance of the RGB modality, we apply transfer learning by pre-training on the Micro-Action 52 dataset Guo et al. 2024 and fine-tuning on the iMiGUE dataset Liu et al. 2021 . The key contributions of this paper can be summarized as follows: • We present an integrated multi-modal MGs classification network that utilizes complementary information from six diverse modalities: joint, limb, RGB video, Taylor video, optical flow video, and depth video. • We propose an effective ensemble fusion method capable of efficiently integrating six modalities, enabling the joint exploitation of modality-specific strengths for improved MGs classification accuracy. • Extensive experiments on the iMiGUE dataset Liu et al. 2021 demonstrate that the proposed MM‑Gesture achieves state-of-the-art performance, reaching a Top-1 accuracy of 73.213%, which is the highest reported accuracy in previous Micro-gesture Analysis (MiGA) challenges. 2 Related Work Micro-Gestures (MGs) are becoming increasingly important in understanding human emotions, focusing on subtle body movements in daily interactions. Advances in this field have been driven by the development of large benchmark datasets and sophisticated model architectures Chen et al. 2019 ; Liu et al. 2021 ; Chen et al. 2023 ; Guo et al. 2024 . Key datasets include the SMG dataset Chen et al. 2023 , which consists of recordings from 40 participants engaged in storytelling, capturing upper limb micro-gestures and emotional states. The iMiGUE dataset Liu et al. 2021 offers identity-free videos of 72 athletes at press conferences, annotated with 32 micro-gesture categories for analyzing both actions and emotions. The MA-52 dataset Guo et al. 2024 expands the focus to full-body micro-actions, with 22,000 samples covering 52 action-level and 7 body-level categories, sourced from psychological interviews to recognize subtle visual cues. Current models primarily focus on limited modalities. RGB-based methods leverage spatial-temporal modeling strategies, such as a pure Transformer backbone with shifted 3D local attention windows Liu et al. 2022 . MANet Guo et al. 2024 integrates SE and TSM modules with semantic embedding loss for fine-grained micro-action recognition. Skeleton-based approaches include a 3D-CNN model with joint and semantic embedding losses Li et al. 2023 , and an EHCT framework Huang et al. 2023 employs hypergraph-based attention and ensemble Transformers Wang et al. 2024a ; Wang et al. 2024b to capture high-order joint relations and address class imbalance. In contrast, skeleton sequences can be encoded as 3D heatmaps and fused with RGB inputs through a dual-branch multimodal network Duan et al. 2022 . Inspired by this network, Chen et al . Chen et al. 2024 adopt channel-wise cross-attention and prototype refinement to enhance feature fusion and category discrimination, while Huang et al . Huang et al. 2024 design a multi-scale heterogeneous fusion network. Recently, Li et al . Li et al. 2025 propose a hierarchical prototype-based calibration method to resolve ambiguity in fine-grained actions. Overall, current methods only focus on the RGB or skeleton data. To exploit the complementarity between different multimodal data, we propose the MM-Gesture model, adopting a comprehensive multimodal approach that integrates six modalities: joint, limb, RGB video, Taylor video, optical flow video, and depth video. This approach enables a deeper understanding and representation of micro-gestures, capturing their nuances and dynamics. Additionally, we leverage transfer learning from the MA-52 dataset to infuse valuable prior knowledge into the RGB modality, further enhancing its recognition accuracy. Consequently, our model improves performance on existing benchmarks and paves the way for advanced applications in human emotion understanding through micro-gesture analysis. 3 Methodology Figure 1: Pipeline of the proposed multimodal micro-gesture recognition framework ( MM-Gesture ), which

Read the complete text