Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition
Mengzhu Li, Quanxing Zha, Hongjun Wu · 2025
Read Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition by Mengzhu Li, Quanxing Zha, Hongjun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-21004.txt.
Introduction
Mengzhu Li, Quanxing Zha, and Hongjun Wu speed up reading emotions from videos of faces, in a 2025 paper. Faces shift quickly, and most frames carry background noise or repeated expressions. Existing readers waste effort on that clutter.
Their network, AdaTosk, learns in two branches at once. One branch rebuilds hidden frames to learn solid features. The other classifies expressions while softly masking dull moments. Two mask types highlight key expression beats and trim repeated meaning. On standard benchmarks it matched top accuracy while cutting computing cost a lot. Leaner emotion reading suits live apps on modest devices.
Catalog record
| CREATOR | Mengzhu Li, Quanxing Zha, Hongjun Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-21004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition Mengzhu Li, Quanxing Zha, Hongjun Wu Abstract Dynamic Facial Expression Recognition (DFER) facilitates the understanding of psychological intentions through non-verbal communication. Existing methods struggle to manage irrelevant information, such as background noise and redundant semantics, which impacts both efficiency and effectiveness. In this work, we propose a novel supervised temporal soft masked autoencoder network for DFER, namely AdaTosk, which integrates a parallel supervised classification branch with the self-supervised reconstruction branch. The self-supervised reconstruction branch applies random binary hard mask to generate diverse training samples, encouraging meaningful feature representations in visible tokens. Meanwhile the classification branch employs an adaptive temporal soft mask to flexibly mask visible tokens based on their temporal significance. Its two key components, respectively of, class-agnostic and class-semantic soft masks, serve to enhance critical expression moments and reduce semantic redundancy over time. Extensive experiments conducted on widely-used benchmarks demonstrate that our AdaTosk remarkably reduces computational costs compared with current state-of-the-art methods while still maintaining competitive performance. 1 Introduction Figure 1 : Redundancy in visible tokens. Row 1: All original frames. Row 2: Frames after random masked. Row 3: Gray-masked regions indicate redundant temporal information in visible tokens. Yellow lines connect all distinct frames/tokens. (i) Key moments denote the important frames; (ii) Similarity accumulation indicates the redundancy of resemble tokens. Facial expression recognition (FER), which aims to accurately identify facial expressions from static visual contents, has emerged as a vital tool for understanding and conveying psychological intentions, offering profound insights into human behavior and decision-making. Nevertheless, FER struggles to effectively address intractable dynamic expression variations. It is acknowledged that dynamic facial expression recognition (DFER) Li and Deng 2020 ; Guo et al. 2024 ; Xiao et al. 2024 is therefore a promising direction as it well accounts for spatiotemporal signals in videos, potentially enabling dynamical human expressions in real-world scenarios. However, the efficiency of DFER still remains significant challenge due to substantial redundancy inadvertently introduced by temporal continuity in videos. Inspired by the remarkable success of Masked AutoEncoder (MAE) He et al. 2022 , some attempts Liang et al. 2022a ; Li et al. 2023b have improved the efficiency by utilizing the visible patches that have potential to reconstruct the original content. Building on this foundation, recent efforts Sun et al. 2023 ; Sun et al. 2024b further reduce the computational costs during the reconstruction process with informative features learned by efficient encoders, significantly advancing the efficiency and efficacy of DFER. Despite these advancements, a critical limitation remains: the tendency to omit subtle and hard-to-detect expression dynamics in videos. Different from images, video data exhibits not only spatial redundancy but also additional temporal redundancy. Although the reconstruction branch removes many redundant tokens, the remaining visible tokens still contain much irrelevant information, such as background noise. As shown in Fig. 1 Row 3, expression dynamics in the DFER task are sparse and typically triggered by a few salient frames. And these dynamics are generally less pronounced than those in action recognition, occurring mainly in key facial regions, such as the mouth. Hence there is considerable potential for improving efficiency in the DFER. Our main goals: (i) To focus on key moments of expression dynamics while avoiding unnecessary computation on irrelevant frames; (ii) To reduce the temporal accumulation of semantically similar tokens, such as non-essential actions and static information. To alleviate temporal redundancy, the intuitive ideal is amplifying the influence of temporally valuable tokens while reducing the impact of meaningless ones. Inspired by the binary masking mechanism in MAE He et al. 2022 , we design an adaptive masking strategy that applies varying degrees of masking to the visible tokens. For example, we employ lighter masks to informative tokens, preserving their influence. To distinguish these two types of masking, we refer to the binary mask as ‘Hard Mask’ and the adaptive mask as ‘Soft Mask’ . Therefore, a novel adaptive temporal soft mask mechanism is introduced for aforementioned goals. (i) To preserve the representation of expression dynamics without being constrained by predefined categories, we design a class-agnostic dynamic soft mask . By calculating frame-to-frame feature differences, we select and enhance activated frames, with the soft mask defined by their influence on the entire time sequence. (ii) Some tokens may be temporally repetitive but still carry essential information of expression. To address this, we propose a class-semantic similar soft mask to integrate class semantics into similarity accumulation over time. To prevent the soft mask from overemphasizing specific tokens, we exclude some high-score tokens from the last frame during transitions. In light of these, we propose a novel Ada ptive T emporal S o ft Ma sk mechanism, namely AdaTosk , for Efficient Dynamic Expression Recognition. The overview of our AdaTosk is illustrated in Fig. 2 and it mainly contains two parallel branches. Specifically, the self-supervised reconstruction branch applies random hard mask for data augmentation, generating diverse training samples and promoting meaningful representations in visible tokens in each iteration. Corresponding to the hard mask, the parallel supervised classification branch employs a unique soft mask mechanism, assigning adaptive masks to visible tokens based on their temporal redundancy and semantics in the video. The two core strategies, class-agnostic and class-semantic soft masks, respectively capture key moments in expression dynamics and the accumulation of semantic redundancy over time. Our contributions can be summarized as following: (i) Our AdaTosk is the first to integrate supervised VideoMAE into the DFER task, where the optimization of visible tokens significantly enhances the efficiency of expression recognition; (ii) We propose an innovative adaptive temporal soft mask that adaptively adjusts the masking degree. Its two designs: the class-agnostic dynamic and class-semantic similar soft masks, respectively activate key expression dynamics and suppress semantic redundancy over time; (iii) AdaTosk achieves a 3% performance improvement over state-of-the-art methods while reducing the model size by about 10M parameters and lowering computational cost by 6G FLOPs on the FERV39K Wang et al. 2022a , DFEW Jiang et al. 2020 , and MAFW Liu et al. 2022 datasets. Figure 2 : Overall framework of AdaTosk. The left part illustrates the overall model architecture with two parallel branches. After the hard mask, the visible tokens are processed by the encoder. Then the decoder reconstructs the masked tokens using the reconstruction loss \mathcal{L}_{rec} , meanwhile a temporal soft mask is applied for expression recognition with the classification loss \mathcal{L}_{cls} . The right part represents the adaptive temporal soft mask mechanism: (i) Class-agnostic dynamic soft mask identifies key frames; (ii) Class-semantic similar soft mask transfers the accumulative temporal score between adjacent frames; (iii) The final temporal soft mask combines results from (i) and (ii). 2 Related Work Dynamic facial expression recognition. Early deep learning models Jiang et al. 2020 ; Wang et al. 2022a ; Wang et al. 2023 ; Li et al. 2023a ; Zha et al. 2024 introduce end-to-end supervised architectures capable of extracting distinct spatial and temporal features. Later, Transformer-based models Liu et al. 2022 ; Zhao and Liu 2021 , leveraging ViT Dosovitskiy et al. 2020 encoders, have become dominant in this field. Recently, there has been a notable trend toward large-scale models pre-trained on massive datasets in a self-supervised manner. Multi-modal vision-language models