Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara · 2026

Read Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction by Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21869.txt.

Introduction

Dinithi Dissanayake and coauthors present a two-stage multimodal framework for emotion mimicry intensity prediction, entered in the Hume-ABAW10 challenge. The task predicts six continuous emotion intensities from in-the-wild clips. Modality-specific encoders train independently, then fuse through a regressor with modality dropout and encoder adaptation. The best validation system, fusing text, audio, vision, and motion under a four-to-one split, reaches an average Pearson correlation of zero point four seven two two. Placed third, with test correlation zero point five seven.

Catalog record

CREATORDinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29170
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara Abstract We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration , Amusement , Determination , Empathic Pain , Excitement , and Joy , from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text–audio–vision–motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction. 1 Introduction Emotion mimicry intensity prediction is an important problem at the intersection of affective computing, multimodal machine learning, and human behaviour understanding. Unlike categorical emotion recognition, the EMI task requires predicting continuous intensity values across multiple affective dimensions, making it particularly sensitive to ambiguity, sparsity, and differences in how emotion is expressed across clips. The Hume-ABAW10 EMI Challenge further amplifies these difficulties by presenting in-the-wild data 12 with strong temporal variation, partial modality missingness, and highly imbalanced targets. To address these challenges, we propose a staged multimodal framework that integrates text, audio, and vision, with an optional motion branch. Rather than training a single end-to-end multimodal model from scratch, we first learn modality-specific encoders independently and then combine their representations through a lightweight fusion regressor. This strategy allows each branch to develop a stable unimodal representation before cross-modal interaction, while still permitting limited encoder adaptation during fusion. In addition, modality dropout is used to improve robustness to weak or missing channels. Our exploratory analysis shows that the EMI dataset exhibits long-tailed sequence lengths, substantial label sparsity, and partial incompleteness in the text modality. These properties directly motivate our design choices. The strong imbalance across target dimensions, especially for Empathic Pain , encourages the use of a CCC-oriented regression objective, while the variability in sequence duration motivates temporal pooling and sequence-aware encoders. Missing-text fallback and multimodal fusion are further introduced to reduce reliance on any single modality. Empirically, we find that text and audio provide the strongest unimodal predictive signals, whereas visual and motion features contribute more selectively to dimensions such as Joy , Amusement , and Excitement . Our final fusion systems consistently outperform the unimodal branches, with the best validation result obtained by the text–audio–vision–motion model on the expanded 4:1 split. Overall, this work makes three contributions: (1) a practical staged training strategy for multimodal EMI prediction, (2) an empirical analysis of how different modalities contribute across emotion axes, and (3) a challenge submission that highlights the complementary role of motion cues in multimodal fusion. We have made our code publicly available at https://github.com/Dinithipurna/MimicMetrics to support reproducibility. 2 Related Work Significant advancements in affective and behavioural computing have enhanced the understanding and automated detection of emotional expressions, particularly through facial analysis. Contemporary research increasingly demonstrates that leveraging multimodal data, including text, audio, and visual inputs, leads to improved performance across a wide range of affective computing tasks 11 . In particular, multimodal approaches enable models to capture complementary emotional cues that are not present in a single modality 19 ; 24 . Rather than relying solely on raw video inputs, many approaches adopt pretrained encoders to extract high-level, semantically rich representations from each modality. In such methods, features are often learned separately for each modality, such as audio, visual, and textual streams, allowing the model to capture modality-specific characteristics 6 ; 17 . These representations may then be combined through a fusion mechanism, enabling the integration of complementary information across modalities and supporting a more comprehensive understanding of affective signals 19 ; 24 ; 22 . Prior work has utilised text encoders such as BERT 4 , ELECTRA 3 , and GTE-base 14 to derive informative textual representations. For the audio modality, pretrained models including VGGish 9 and Wav2Vec 2.0 1 are commonly employed, with some approaches further incorporating handcrafted features such as eGeMAPS 13 to enrich acoustic representations. In the visual domain, architectures such as ResNet 8 , EfficientNet 20 , and Masked Autoencoders 7 have been adopted to capture expressive facial and scene-level features. There have been multiple approaches for the fusion of these multimodal features. Ding et al implement a specialised fusion block with cross-attention mechanisms for the integration of multimodal data 5 . Richet et al. 18 proposed a system where each modality is individually processed using TCNs, and subsequently integrated via a co-attention mechanism to perform the final prediction, while Jun Yu et al 23 proposed a system with Dual-Stage Cross-Modal Alignment followed by Quality-Aware Dynamic Fusion. Motivated by these prior efforts, our submission adopts a two-stage multimodal fusion framework that combines audio, visual, and textual representations, together with an optional motion branch. While the motion branch provides only complementary gains in our current setting, we highlight it as a promising alternative direction for further exploration in emotion mimicry intensity prediction. 3 Exploratory Data Analysis 3.1 Dataset Characteristics We first examined dataset composition and input quality across the training and validation splits. The dataset contains 12,660 samples in total, comprising 8,072 training instances and 4,588 validation instances. Sequence durations are highly variable across both splits. In the training set, videos contain 73.33\pm 62.11 frames on average (median = 65, range = 1–2,255), whereas in the validation set, videos contain 75.81\pm 191.39 frames on average (median = 66, range = 1–12,500). Although no missing-frame cases were found after frame extraction, modality completeness analysis revealed missing text embeddings for 5.67% of training samples and 6.34% of validation samples, while face and audio embeddings were available throughout. Following prior challenge papers, we additionally extracted text embeddings as a complementary semantic modality when available 19 ; 23 ; 25 . Overall, these results indicate substantial temporal heterogeneity together with partial modality sparsity. 3.2 Label Distribution The target distributions are clearly imbalanced (Tables 1 ). Among the emotion dimensions, Empathic Pain has the lowest mean intensity (train: 0.0214, validation: 0.0134), whereas Admiration and Excitement exhibit comparatively higher mean values. Label sparsity is also pronounced, with zero-valued targets dominating several dimensions. For example, Empathic Pain contains approximately 89.9% zero-valued targets in the training split and 92.2% in the validation split. This strong skew motivated the use of robust regression objectives based on Concordance Correlation Coefficient (CCC), rather than relying solely on mean squared error. Split Samples Frames (Mean \pm Std) Min Max Adm. Amuse. Determ. Emp. Pain Excite. Joy Train 8072 73.33\pm 62.11 1 2255 0.1785 0.1253 0.0967 0.0214 0.1420 0.1157 Valid 4588 75.81\pm 191.39 1 12500 0.1594 0.1303 0.0668 0.0134 0.1294 0.0994 Table 1: Summary statistics for the training and validation splits, including clip length statistics and mean label values for the six emotion dimensions. 3.3 Input Variability Across Modalities Input variability is

Read the complete text