Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition

Phuong-Anh Nguyen, The-Son Le, Duc-Trong Le, Cam-Van Thi Nguyen* · 2026

Read Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition by Phuong-Anh Nguyen, The-Son Le, Duc-Trong Le, Cam-Van Thi Nguyen* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21565.txt.

Introduction

Phuong-Anh Nguyen and co-authors use self-paced curriculum learning for modality balance in conversational emotion recognition. These approaches mixing words, faces, and voices have grown fast, yet mismatch and uneven learning limit good use of combined information. Their framework introduces a dual-level Difficulty Measurer capturing line-level and whole-chat challenges. The work targets chats where words, faces, and voices must be read as one. It aims to give each mode its fair share of weight.

Catalog record

CREATORPhuong-Anh Nguyen, The-Son Le, Duc-Trong Le, Cam-Van Thi Nguyen*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69993
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21565

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition Phuong-Anh Nguyen, The-Son Le, Duc-Trong Le, Cam-Van Thi Nguyen* Abstract Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for understanding human interactions, where multimodal approaches integrating language, facial expressions, and vocal tone have led to significant advancements. However, a persistent challenge in multimodal architectures, including MERC, is modality misalignment and imbalanced learning. These issues often hinder models from effectively utilizing multimodal information, leading to suboptimal performance despite the availability of multiple modalities. To address this, we design a framework for MERC with a proposed plug-and-play module that builds upon Self-Paced Curriculum Learning ( SPCL ). As in Curriculum Learning (CL), an effective Difficulty Measurer is essential for structuring a meaningful Learning Scheduler . In this work, we propose a dual-level Difficulty Measurer tailored for MERC, addressing both intra- and inter-conversational dynamics. Unlike conventional approaches that assess difficulty only at the utterance level, our dual-level design incorporates a conversation-level difficulty score. The utterance-level score captures fine-grained modality-specific challenges, while the conversation-level score models broader dialogue structures, including emotional dependencies and modality coherence within the conversation. This holistic evaluation enables our Learning Scheduler to dynamically guide training from easier to more challenging instances. By integrating SPCL into existing MERC architectures, our method effectively mitigates modality imbalance and enhances model robustness. Extensive experiments on the IEMOCAP and MELD datasets confirm consistent improvements: on IEMOCAP, SPCL achieves gains ranging from approximately +1.2% to +6.6% in weighted F1-score over baseline models across different architectures and modality settings, while on MELD, it delivers even more pronounced improvements, with gains reaching up to +10.4% over baseline models. These gains underscore the practical value of SPCL for real-world MERC applications, as it substantially improves emotion recognition accuracy while maintaining compatibility as a plug-and-play module across diverse model architectures. keywords Self-paced Curriculum Learning, Modality Balance, Multimodal Emotion Recognition 1 Introduction Multimodal learning has gained significant attention in artificial intelligence due to its ability to integrate information from diverse sources, such as text (T), audio (A), and visual (V) data 62 ; 1 . By leveraging complementary modalities, multimodal models enhance understanding and improve predictive performance 29 ; 30 . Among its diverse applications, Emotion Recognition in Conversations (ERC) has emerged as a critical task spanning artificial intelligence, cognitive science, and social sciences. The primary objective of ERC is to detect the emotional undertones accompanying each utterance in a conversation. While language plays a crucial role, emotions are often conveyed through a combination of verbal and non-verbal cues, such as facial expressions, tonal variations, and bodily gestures 13 . Given the inherently multimodal nature of human communication, integrating multimodal data into ERC has naturally evolved into Multimodal Emotion Recognition in Conversations (MERC). By jointly modeling text, audio, and visual modalities, MERC seeks to enhance emotion recognition beyond the constraints of unimodal approaches 11 ; 21 ; 33 . However, the effectiveness of multimodal integration is often limited by modality imbalance, where certain modalities contribute disproportionately to learning, leading to suboptimal model performance. Prior research has examined this phenomenon from various perspectives, describing it as the dominance of specific modalities 38 , discrepancies in convergence rates 52 ; 46 , or diminishing marginal utility of modalities 55 . 64 categorize these challenges into Property Discrepancy and Quality Discrepancy. Property Discrepancy arises because different modalities exhibit distinct learning behaviors due to their heterogeneous nature. For instance, audio features often require fewer training iterations before overfitting, whereas visual features typically converge more slowly. This inconsistency complicates optimization, making it difficult to balance learning across modalities. Quality Discrepancy refers to the uneven distribution of task-relevant information among modalities. While all modalities aim to represent the same underlying context, some provide stronger discriminative signals than others. Multimodal models, inherently greedy in their learning dynamics, tend to prioritize the most informative modality while underutilizing weaker ones, further exacerbating modality imbalance. Figure 1: Performance gain of training baseline models on full modality (ATV) over training on bi-modality (AT or TV). Evaluation uses W-F1 and Acc metrics across IEMOCAP dataset. In MERC, text is often the dominant modality, providing explicit emotional cues, whereas audio and visual features are more subtle and context-dependent. Consequently, models may become overly reliant on textual data, failing to fully integrate information from other modalities. This imbalance ultimately leads to suboptimal fusion strategies and reduced overall performance. As shown in Figure 1 , we conducted an experiment comparing the performance of tri-modal models (ATV) with their bi-modal versions (TA and AV). While there were instances where the tri-modal approach outperformed its bi-modal counterparts, the results were inconsistent. In half of the cases, incorporating all three modalities led to a performance reduction, with accuracy dropping between 0.05% and 1.18%, and w-F1 dropping between 0.11% and 1.18%. These findings highlight the need to further investigate modality imbalance, a crucial factor affecting MER performance. To address modality imbalance in MERC, previous studies have explored several strategies, including pre-trained unimodal networks 7 ; 58 , auxiliary learning objectives 60 ; 66 , and optimization-based techniques 38 ; 10 ; 36 . Pre-trained unimodal networks leverage separately trained feature extractors for each modality before integrating them into a multimodal framework, improving individual modality representations but often incurring high computational costs and requiring large-scale labeled data 25 ; 23 . Auxiliary learning objectives introduce additional constraints or tasks, such as contrastive or self-supervised learning, to enhance modality-specific representations 66 ; 59 , yet they may struggle to generalize across diverse conversational contexts. Optimization-based methods, which directly adjust gradient updates to prevent modality dominance, have shown promise in balancing multimodal contributions 55 ; 38 ; 28 ; 31 , but their reliance on complex gradient manipulation makes them difficult to implement and tune. Building upon these persistent challenges, we propose a novel approach based on Self-Paced Curriculum Learning to address modality imbalance in MERC. As an extension of Curriculum Learning, Self-Paced Curriculum Learning enhances this process by dynamically selecting training samples based on the model’s learning progress, ensuring a more adaptive and balanced training trajectory. Specifically, we introduce a framework specifically designed for MERC, incorporating our proposed SPCL module to tackle modality imbalance. Our proposed SPCL module comprises two key components: (1) a Difficulty Measurer , which assesses the complexity of each training sample through a dual-level design that captures utterance-level recognition performance and conversation-level modality discrepancy to ensure balanced learning across modalities; and (2) a Learning Scheduler , which dynamically refines sample selection based on the model’s evolving competence, progressively guiding training from easier to more challenging instances. While modality imbalance has been explored in general multimodal learning, its unique challenges in MERC, especially those linked to conversational context and dialogue structures, have not been fully addressed. Our method fills this gap by offering an SPCL strategy that is carefully tailored for MERC. With this design, SPCL improves model stability, reduces problems from modality imbalance, and achieves better performance on MERC tasks. Overall, our key contributions are as follows: • We introduce a novel framework tailored for MERC, integrating our proposed module, SPCL, based on Self-Paced Curriculum Learning (SPCL), to address modality imbalance through adaptive sample selection. • We design a Difficulty Measurer with a

Read the complete text