United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space

R. Gnana Praveen, Jahangir Alam, Eric Charton · 2025

Read United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space by R. Gnana Praveen, Jahangir Alam, Eric Charton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12261.txt.

Introduction

Gnana Praveen, Jahangir Alam, and Eric Charton sort feelings from sound and video even when the two channels do not back each other up. Their paper from twenty twenty-five targets emotion scores on the valence-arousal plane. Audio and visual cues are expected to complement, but at times they clash or one goes weak. A fixed fusion then drags the score down.

Their GRJCA model adds gates to a recursive joint cross-attention design. Each gate weighs attended features against raw ones by how strongly the channels agree. A second, higher gate then weighs the rounds against each other. On the hard AffWild2 set, scores for valence and arousal beat the ungated base model. Test and validation marks both rose. The design suits wild video, where one channel often falters.

Catalog record

CREATORR. Gnana Praveen, Jahangir Alam, Eric Charton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space R. Gnana Praveen, Jahangir Alam, Eric Charton Abstract Audio and visual modalities are two predominant contact-free channels in videos, which are often expected to carry a complementary relationship with each other. However, they may not always complement each other, resulting in poor audio-visual feature representations. In this paper, we introduce Gated Recursive Joint Cross Attention (GRJCA) using a gating mechanism that can adaptively choose the most relevant features to effectively capture the synergic relationships across audio and visual modalities. Specifically, we improve the performance of Recursive Joint Cross-Attention (RJCA) by introducing a gating mechanism to control the flow of information between the input features and the attended features of multiple iterations depending on the strength of their complementary relationship. For instance, if the modalities exhibit strong complementary relationships, the gating mechanism emphasizes cross-attended features, otherwise non-attended features. To further improve the performance of the system, we also explored a hierarchical gating approach by introducing a gating mechanism at every iteration, followed by high-level gating across the gated outputs of each iteration. The proposed approach improves the performance of RJCA model by adding more flexibility to deal with weak complementary relationships across audio and visual modalities. Extensive experiments are conducted on the challenging Affwild2 dataset to demonstrate the robustness of the proposed approach. By effectively handling the weak complementary relationships across the audio and visual modalities, the proposed model achieves a Concordance Correlation Coefficient (CCC) of 0.561 (0.623) and 0.620 (0.660) for valence and arousal respectively on the test set (validation set). This shows a remarkable improvement over the baseline of 0.211 (0.240) and 0.191 (0.200) for valence and arousal, respectively, in the test set (validation set), achieving competitive performance in the valence-arousal challenge of the 8th Affective Behavior Analysis in-the-Wild (ABAW) competition. 1 Introduction With the advancement of technology, empowering machines with affective capabilities is a fundamental requirement for Human-Computer Interaction (HCI). It has a lot of applications in a wide range of environments such as health-care Rajasekhar et al. 2021 , autonomous driving Chen et al. 2022 , e-learning et al. 2020 , etc. Automatic recognition of human emotions is a challenging problem as the expressions pertinent to the emotional states are quite complex and extremely diverse across individuals and cultures Anagnostopoulos et al. 2015 . Emotion Recognition (ER) can be formulated as the problem of classification or regression of emotions. In recent years, regression of expressions is gaining attention as it has the potential to capture a wide range of expressions, which can be useful in many applications such as estimation of pain intensity levels in the healthcare domain Gnana Praveen et al. 2020 , engagement intensity levels in business or educational settings Kaur et al. 2018 , etc. Depending on the granularity of labels, the regression of emotions can be formulated as ordinal regression or continuous regression. Compared to ordinal regression, continuous (or dimensional) regression is even more challenging due to the complex process of obtaining annotations in continuous dimensions. Valence and arousal are widely used dimensions to estimate emotion intensities in a continuous domain. Valence reflects the wide range of emotions in the dimension of pleasantness, from being negative (sad) to positive (happy). In contrast, arousal spans a range of intensities from passive (sleepiness) to active (high excitement). Human emotions can be often conveyed through various modalities such as face, voice, text, physiology, etc. Out of all the modalities, facial and vocal expressions are the widely explored modalities in videos for multimodal ER D’mello and Kory 2015 . Cross Attention (CA) has been widely explored to capture the complementary relationships across audio and visual modalities in several applications such as event localization Duan et al. 2021 , action localization Lee et al. 2021 , and emotion recognition Praveen et al. 2021 . Recently, sophisticated cross attention models have been proposed by introducing joint feature representation in the cross attentional framework to enhance the feature representations by simultaneously capturing intra- and inter-modal relationships Praveen et al. 2022 ; Praveen et al. 2023a ; Praveen et al. 2023c . Even though these models show promising performance, they rely on the assumption that audio and visual modalities always exhibit complementary relationship. However, the audio and visual modalities may not always be complementary to each other, they may also conflict with each other Wang et al. 2022 . Praveen et al. Praveen and Alam 2024a investigated the problem of weak complementary relationships for ER with visual analysis and showed that weak complementary relationships degrade the cross-attended features, resulting in poor audio-visual feature representations. To tackle this problem, Praveen et al. Praveen and Alam 2024b proposed a Dynamic Cross Attention (DCA) model to adaptively choose the most relevant features among the cross attended and unattended features based on gating mechanism. In this work, we further extend their approach on the recently proposed RJCA Praveen et al. 2023c model by introducing the gating mechanism to control the flow of information across the original features and cross attended features of every iteration. By controlling the flow of information across the features of all iterations of RJCA model, we have more flexibility to capture the most relevant information across all the iterations. To further improve the system, we also explored hierarchical gating, where the gating mechanism is employed in every iteration, followed by high-level gating mechanism across the gated outputs of every iteration. The proposed approach demonstrates superior performance over the DCA Praveen and Alam 2024a model as it can leverage the attended features of all iterations instead of only the final attended features. The major contributions of the paper are as follows. • We improve the performance of RJCA model by handling weak complementary relationships across the audio and visual modalities by controlling the flow of information across the features of all iterations and original features. • The proposed model has been further enhanced by introducing a hierarchical gating mechanism to capture more refined semantic features. • A detailed set of experiments are conducted to evaluate the contribution of the proposed model on the Affwild2 dataset, achieving better generalization ability and superior performance over prior state-of-the-art models. 2 Related Work 2.1 A-V fusion for dimensional ER Atmaja et al Atmaja and Akagi 2020 proposed a weighted summation of the loss functions in a multi-task learning framework, where the weights of the individual loss components are adjusted to improve the performance. Enrique et al also Sánchez-Lozano et al. 2013 explored the multi-stage fusion of predictions obtained from the multiple features of each modality in a hierarchical fashion. Haifeng et al Chen et al. 2019 extracted deep spatio-temporal feature vectors of the images and spectrograms by combining pre-trained 2D-CNNs with 1D-CNNs and also exploited spatial-temporal graph convolutional network (STGCN) to obtain the geometric features based on facial landmarks. They have employed hybrid fusion using BLSTMs to obtain final predictions. Mihalis et al Nicolaou et al. 2013 studied the correlation across the emotional dimensions and found that each emotional dimension is more correlated with other emotional dimensions than with the features of individual modalities. Ehab et al AlBadawy and Kim 2018 investigated the optimal trade-off between continuous and discrete

Read the complete text