V-NAW: Video-based Noise-aware Adaptive Weighting for Facial Expression Recognition

JunGyu Lee, Kunyoung Lee, Haesol Park, Ig-Jae Kim Gi Pyo Nam · 2025

Read V-NAW: Video-based Noise-aware Adaptive Weighting for Facial Expression Recognition by JunGyu Lee, Kunyoung Lee, Haesol Park, Ig-Jae Kim Gi Pyo Nam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15970.txt.

Introduction

Lee and four co-authors worked to lift video-based face expression reading in twenty twenty-five. Reading feelings on a face helps machines work with people and helps tests of mental health. In video, two snags drag scores down. Labels can be unclear, since a look changes across a clip. Back-to-back frames look near the same, so a model may just memorize them.

Their method, V-NAW, weighs each frame on its own. Frames with more signal count more, and noisy ones count less. A plain trick varies the frames and cuts the sameness between them. The team built it for a feelings contest in the wild, on the Aff-Wild two video set. Broad tests showed clear gains in reading expressions in video. The authors say the wins come from noisy labels and repeat frames, not bigger models.

Catalog record

CREATORJunGyu Lee, Kunyoung Lee, Haesol Park, Ig-Jae Kim Gi Pyo Nam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15970

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

V-NAW: Video-based Noise-aware Adaptive Weighting for Facial Expression Recognition JunGyu Lee, Kunyoung Lee, Haesol Park, Ig-Jae Kim Gi Pyo Nam Abstract Facial Expression Recognition (FER) plays a crucial role in human affective analysis and has been widely applied in computer vision tasks such as human-computer interaction and psychological assessment. The 8th Affective Behavior Analysis in-the-Wild (ABAW) Challenge aims to assess human emotions using the video-based Aff-Wild2 dataset. This challenge includes various tasks, including the video-based EXPR recognition track, which is our primary focus. In this paper, we demonstrate that addressing label ambiguity and class imbalance, which are known to cause performance degradation, can lead to meaningful performance improvements. Specifically, we propose Video-based Noise-aware Adaptive Weighting (V-NAW), which adaptively assigns importance to each frame in a clip to address label ambiguity and effectively capture temporal variations in facial expressions. Furthermore, we introduce a simple and effective augmentation strategy to reduce redundancy between consecutive frames, which is a primary cause of overfitting. Through extensive experiments, we validate the effectiveness of our approach, demonstrating significant improvements in video-based FER performance. 1 Introduction Facial Expression Recognition (FER) has been widely applied in various fields, such as human-robot interaction, non-verbal communication, and mental health monitoring. Previous studies have introduced various methods to address key challenges in facial expression recognition, such as occlusion [ 23 , 38 ] , label ambiguity [ 43 , 46 , 37 , 35 , 45 , 24 ] , and class imbalance [ 42 , 47 , 24 ] . Some studies have focused on constructing datasets, including those collected in controlled laboratory environments [ 30 , 33 , 31 ] , as well as those captured in in-the-wild settings [ 1 , 25 , 32 ] . However, most approaches rely on image datasets, limiting their ability to capture the temporal variations of facial expressions. As human perception inherently relies on both spatial and temporal cues, capturing temporal variations is crucial for accurate emotion recognition. The Affective Behavior Analysis in-the-Wild (ABAW) Challenge has been organized to analyze human affect, utilizing the Aff-Wild2 [ 10 ] dataset, a large-scale video dataset collected in unconstrained environments. In contrast to traditional FER benchmarks that primarily focus on images, Aff-Wild2 provides images in a video format, enabling a more comprehensive analysis of temporal variations in facial expressions in real-world scenarios. In FER, the main challenges include label ambiguity, as facial expressions are inherently subjective and subject to annotator interpretation, and class imbalance, where some expressions are overrepresented in daily life while others are less frequent. These factors contribute to model uncertainty and bias toward majority classes. Previous image-based FER studies [ 35 , 37 , 39 , 46 , 42 , 47 , 24 ] have tried to address these issues, however, these methods perform FER on individual frames, without considering the temporal variations in facial expressions. In addition to these challenges, video datasets often suffer from spatiotemporal redundancy, where consecutive frames contain highly similar information. This redundancy can cause models to overfit on specific regions or redundant facial patterns rather than capturing meaningful temporal variations. To alleviate the problem, several data augmentation approaches have been proposed [ 36 , 2 ] , such as clip skipping and frame erasing. To address these challenges at the same time, we propose Video-based Noise-aware Adaptive Weighting (V-NAW), an extension of Navigating Label Ambiguity (NLA) [ 24 ] for video-based FER. V-NAW adaptively assigns weights to each sample, effectively mitigating label ambiguity and class imbalance in video-based FER datasets. Additionally, we introduce an augmentation strategy using frame skipping and in-frame erasing to mitigate spatiotemporal redundancy of the video dataset. This enables the model to learn meaningful temporal information rather than overfitting to specific regions or repetitive patterns. Through extensive experiments on the Aff-Wild2 dataset, we demonstrate that our method significantly outperforms the baseline, highlighting its effectiveness for video-based FER. The main contributions of our work are as follows: • We propose Video-based Noise-aware Adaptive Weighting (V-NAW), an extension of NAW for video-based FER. V-NAW adaptively assigns loss weights based on the ambiguity of entire video clips, enhancing robustness against label noise and class imbalance. • We introduce an augmentation strategy to mitigate data redundancy in video-based FER. This includes frame skipping to remove redundant frames and in-frame erasing to mask some of the pixels, preventing the model from overfitting to specific regions or repetitive patterns. • We conduct extensive experiments on the Aff-Wild2 dataset, demonstrating the effectiveness of our method with superior performance. 2 Related Work Facial Expression Recognition (FER) can be categorized into image-based and video-based approaches. Recent studies have highlighted the challenges of FER in unconstrained environments, where variations in facial expressions, occlusions, and subjective annotations lead to significant discrepancies [ 22 , 21 , 20 , 7 , 18 , 8 , 19 , 17 , 15 ] . In addition, image-based approaches struggle to capture the temporal variations of facial expressions. To overcome this limitation, recent research trends have increasingly focused on video-based FER, leveraging sequential information to enhance expression recognition in real-world scenarios. 2.1 Image-based FER Facial Expression Recognition (FER) encounters two key challenges: (1) noisy labels, which arise due to the inherent ambiguity of facial expressions and inconsistencies in human annotations, and (2) class imbalance, where certain expressions appear more frequently than others, leading to biased model predictions. To mitigate the impact of noisy labels, various methods have been proposed. SCN [ 37 ] employs re-labeling and ranking regularization to suppress label noise, while DMUE [ 35 ] explores latent label distributions using multiple branches and uncertainty estimation. RUL [ 45 ] leverages feature mixup and a multi-branch framework to learn from relative sample difficulty, and MAN [ 48 ] introduces a co-division module that separates clean and noisy samples to enhance discriminative ability. Lastly, EAC [ 46 ] ensures model robustness by aligning Class Activation Maps (CAMs) [ 49 ] across original and flipped images, preventing models from overfitting to misleading features. Class imbalance has also been a critical challenge in FER, as imbalanced datasets often result in models biased toward majority classes. Face2Exp [ 42 ] integrates large-scale unlabeled face recognition data through a meta-optimization framework. Zhang et al [ 47 ] , in particular, re-balances attention consistency and applies label smoothing to further improve minority class representations. Instead of prior approaches that address either label ambiguity or class imbalance separately, NLA [ 24 ] is the first method to tackle both challenges simultaneously by assigning adaptive weighting based on each sample’s uncertainty. 2.2 Video-based FER Video-based FER methods [ 50 , 34 ] extend image-based FER by leveraging the temporal dynamics of facial expressions. Instead of analyzing a single frame in isolation, these approaches learn from sequences of frames to capture how expressions evolve over time. SAANet [ 27 ] utilizes LSTM [ 4 ] networks and related gated RNN [ 5 ] variants, which have proven effective in modeling subtle changes in facial appearance over consecutive frames. MARLIN [ 2 ] is a self-supervised Masked AutoEncoder (MAE) [ 6 ] that learns universal facial representations from videos by reconstructing spatiotemporal facial details. Lin et al. [ 26 ] propose a lightweight framework for expression classification and

Read the complete text