Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach
Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen Ngo · 2026
Read Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach by Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen Ngo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26849.txt.
Introduction
Nguyen and co-authors recognize micro-expressions from 4D facial mesh data using two camera views. Micro-expressions are brief and faint, so direct mesh processing is hard. The team computes optical flow from each view and splits motion into onset-to-apex and apex-to-offset phases. A three-stream network then fuses the flow channels with attention. On the 4DME dataset, the method scored 0.536 in macro-UF1. That beat the official baseline by over 50 percent and won the challenge.
Catalog record
| CREATOR | Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen Ngo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28355 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen Ngo Abstract Micro-expression recognition is vital for affective computing but remains challenging due to the extremely brief, low-intensity facial motions involved and the high-dimensional nature of 4D mesh data. To address these challenges, we introduce a dual-view optical flow approach that simplifies mesh processing by capturing each micro-expression sequence from two synchronized viewpoints and computing optical flow to represent motion. Our pipeline begins with view separation and sequence-wise face cropping to ensure spatial consistency, followed by automatic apex-frame detection based on peak motion intensity in both views. We decompose each sequence into onset-apex and apex-offset phases, extracting horizontal, vertical, and magnitude flow channels for each phase. These are fed into our Triple-Stream MicroAttNet, which employs a fusion attention module to adaptively weight modality-specific features and a squeeze-and-excitation block to enhance magnitude representations. Training uses focal loss to mitigate class imbalance and the Adam optimizer with early stopping. Evaluated on the multi-label 4DME dataset, comprising 24 subjects and five emotion categories, in the 4DMR IJCAI Workshop Challenge 2025, our method achieves a macro-UF1 score of 0.536, outperforming the official baseline by over 50% and securing first place. Ablation studies confirm that both the fusion attention and SE components each contribute up to 3.6 points of UF1 gain. These results demonstrate that dual-view, phase-aware optical flow combined with multi-stream fusion yields a robust and interpretable solution for 4D micro-expression recognition. keywords Micro-expression, Micro-expression recognition, multi-stream network, optical flow, deep learning * Thi Duyen Ngo, \linkable [email protected] 1 Introduction Micro-Expressions (MEs) are involuntary, brief, and difficult-to-detect facial movements that often occur when people try to hide their true emotions. Unlike regular expressions, which can be faked or controlled, micro-expressions genuinely reflect inner emotions and are therefore of increasing interest in areas such as criminal investigation, negotiation, and behavioral analysis [ 1 ] [ 16 ] [ 10 ] . Micro-expression recognition (MER) is a challenging problem in emotion analysis due to its low intensity and extremely short duration. Traditional methods based on visual data often use 2D features such as LBP-TOP [ 17 ] or HOG [ 2 ] on videos, which have achieved some promising results but still face difficulties in capturing subtle spatio-temporal movements, especially under varying lighting conditions or head poses. Recent deep learning models, including CNNs and RNNsn [ 7 ] [ 12 ] , have shown potential in exploiting spatial and temporal information, but they mainly rely on 2D data or landmark sequences, which are insufficient to fully represent facial dynamics [ 13 , 15 , 9 ] . In recent years, multi-stream or multi-view architectures have shown potential in combining different types of features, such as mesh geometry, landmark displacement vectors, or optical flow (OF) computed from 2D videos. However, these methods still face limitations when dealing with complex micro-movements under varying lighting conditions, viewpoints, and poses, and they have not yet fully exploited the dynamic information that 4D data can provide. To overcome these limitations, 4D (3D + time) data has been explored as a new direction for MER. 3D mesh sequences allow detailed descriptions of facial geometric changes over time, thereby capturing micro-movements that are difficult to observe in 2D space. However, the effective utilization of this data presents significant challenges. First and foremost is the computational trade-off between dimensionality and efficiency. Directly processing raw 4D data (dynamic 3D meshes or point clouds) requires complex geometric deep learning architectures (e.g., PointNet++ [ 11 ] or 3D CNNs [ 14 ] ), which impose a prohibitive computational burden and memory footprint. This complexity hinders the deployment of such systems in real-world scenarios where computational resources may be limited. Therefore, there is a critical need for a method that can leverage the rich depth information of 4D data while maintaining the computational efficiency of 2D CNNs. Our approach addresses this by projecting the 4D spatiotemporal data into synchronized dual-view optical flow maps, effectively reducing the dimensionality while preserving the essential geometric motion cues. Secondly, micro-expressions are characterized by their extremely low intensity and short duration. In traditional RGB-based or raw mesh-based approaches, the subtle pixel intensity changes or vertex shifts caused by micro-expressions are often overshadowed by redundant information such as facial identity or static textures. By converting the sequence into optical flow streams (horizontal, vertical, and magnitude), we explicitly decouple motion from appearance. This ensures that the network focuses solely on the dynamic evolution of facial muscles rather than static facial characteristics. Finally, existing multi-stream architectures typically treat different motion components equally, often employing naive concatenation for feature fusion. However, different micro-expressions manifest differently across motion directions (e.g., eyebrow raising is predominantly vertical, while a smirk may involve significant horizontal stretching). A static fusion strategy fails to capture this nuance. To overcome this, we propose an adaptive multi-stream fusion mechanism. By integrating a Fusion Attention module, our model dynamically learns to recalibrate the importance of Horizontal, Vertical, and Magnitude streams, ensuring that the most discriminative motion features are emphasized for each specific expression instance. Overall, this paper proposes a 4D micro-expression recognition method based on Dual-view optical flow, realized through a triple-stream deep learning architecture with a fusion attention mechanism. The core of the method is to exploit three types of robust motion features (the u, v, and magnitude components of optical flow) computed from 3D mesh sequences, and feed them into parallel streams to model different aspects of micro-expressions. These streams are then combined by an attention module, enabling the system to emphasize important signals and suppress noise. In addition, we investigate data augmentation techniques specifically designed for 4D sequences, such as apex-off augmentation, to improve the generalizability and robustness of the model. Our main contributions include: • Proposing MicroAttNet, a three-branch architecture with a unified attention mechanism, specifically designed for MER on 4D data. • Efficient extraction and analysis of 4D optical flow features (u, v, and magnitude) from 3D mesh sequences. • Introducing a suitable data augmentation strategy for 4D micro-expressions. • Evaluating the proposed method on the 4DMR Challenge dataset [ 6 ] – the first 4D dataset to record spontaneous micro-expressions with high-quality mesh sequences – and achieving outstanding results. Finally, our method is validated in the international 4D Micro-expression recognition for Mind Reading (4DMR 2025) challenge, where it ranked first place on the official leaderboard, confirming the effectiveness and potential for practical applications of the proposed system. 2 The 1st Challenge for 4D Micro-Expression Recognition for Mind Reading The goal of this challenge was to recognize micro-expressions using 4D data, focusing on benchmarking algorithms for detecting subtle, involuntary facial movements in spontaneous high-stakes scenarios. Held as part of the 1st Challenge and Workshop for 4D Micro-Expression Recognition for Mind Reading (4DMR 2025) in conjunction with IJCAI 2025, the event aimed to advance robust techniques leveraging 4D analysis (3D mesh + temporal dynamics) over traditional 2D/3D methods, addressing real-world challenges like illumination variations, head poses, occlusions, and environmental noise. The challenge was hosted on Kaggle (https://www.kaggle.com/competitions/4-dmr-ijcai-workshop-challenge-2025), with top-performing teams ranked by performance metrics. In this challenge, participants are tasked with recognizing five