Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation

Anh-Tuan Mai, Cam-Van Thi Nguyen, Duc-Trong Le · 2026

Read Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation by Anh-Tuan Mai, Cam-Van Thi Nguyen, Duc-Trong Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14274.txt.

Introduction

Anh-Tuan Mai, Cam-Van Thi Nguyen, and Duc-Trong Le work on emotion reading in this 2026 paper. Their task is multimodal emotion recognition in conversation. A system must judge feelings from speech, text, and face cues at once. Each cue type can add its own facts.

Some facts are unique to one cue type. Some are shared across types. Some appear only when types are combined. The authors use those three kinds, named unique, redundant, and synergistic. A good system should use all three in balance, but that balance is hard to reach. Their divide-and-refine method aims to improve both the mix and how well the system can explain its calls.

Catalog record

CREATORAnh-Tuan Mai, Cam-Van Thi Nguyen, Duc-Trong Le
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation Anh-Tuan Mai, Cam-Van Thi Nguyen, Duc-Trong Le Abstract Multimodal emotion recognition in conversation (MERC) requires representations that effectively integrate signals from multiple modalities. These signals include modality-specific cues, information shared across modalities, and interactions that emerge only when modalities are combined. In information-theoretic terms, these correspond to unique , redundant , and synergistic contributions. An ideal representation should leverage all three, yet achieving such balance remains challenging. Recent advances in contrastive learning and augmentation-based methods have made progress, but they often overlook the role of data preparation in preserving these components. In particular, applying augmentations directly to raw inputs or fused embeddings can blur the boundaries between modality-unique and cross-modal signals. To address this challenge, we propose a two-phase framework D ivide and R efine ( DnR ). In the Divide phase, each modality is explicitly decomposed into uniqueness, pairwise redundancy, and synergy. In the Refine phase, tailored objectives enhance the informativeness of these components while maintaining their distinct roles. The refined representations are plug-and-play compatible with diverse multimodal pipelines. Extensive experiments on IEMOCAP and MELD demonstrate consistent improvements across multiple MERC backbones. These results highlight the effectiveness of explicitly dividing, refining, and recombining multimodal representations as a principled strategy for advancing emotion recognition. Our implementation is available at https://github.com/mattam301/DnR-WACV2026 Figure 1 : Illustrative examples of modality interactions in MER. Left : verbal, acoustic, and visual cues consistently convey anger, reflecting redundancy across modalities. Right : text and visual cues redundantly suggest positivity, yet the unique acoustic cue ( fast, low tone ) provides critical evidence which, when integrated synergistically, reveals the true emotion of nervousness. These cases demonstrate that unique information, though often overshadowed by redundancy, plays a vital role in synergy and accurate emotion recognition. 1 Introduction Multimodal machine learning seeks to integrate information from heterogeneous data sources [ 2 , 21 , 25 ] , with the goal of constructing representations that preserve complementary and overlapping signals for downstream modeling. In multimodal emotion recognition in conversation (MERC), this involves leveraging audio, visual, and textual cues, each offering distinct perspectives on affective states [ 1 , 26 ] . Prior work has explored multimodal representations through graph- or attention-based fusion [ 14 , 16 , 24 ] and contrastive alignment in shared spaces [ 31 ] , achieving strong results but oversimplifying modality interactions. In reality, these interactions are nuanced: some cues are modality-specific, others are redundantly available, and still others emerge only through joint integration [ 33 , 35 , 23 ] . In information-theoretic terms, these correspond to unique , redundant , and synergistic contributions [ 19 ] . Fusion-based models risk overemphasizing redundancy, while graph-based models oversimplify dependencies. Recent advances in Partial Information Decomposition (PID) [ 33 , 4 , 19 , 6 ] provide a principled framework to distinguish these contributions, offering a finer-grained view of multimodal interactions beyond simple feature fusion or connectivity. In MERC, these distinctions are intuitive. Unique information ( U ) arises when only one modality provides a discriminative cue, such as acoustic prosody indicating sarcasm despite neutral text. Redundant information ( R ) corresponds to cues independently available from multiple modalities, for example, both a smiling face and a cheerful voice signaling happiness. Synergistic information ( S ) emerges only through joint integration, as in irony, which requires reconciling positive lexical content with a negative vocal tone. Figure 1 illustrates these cases: the left example reflects redundancy across modalities, whereas the right shows how a unique acoustic cue, when combined synergistically, reveals the correct emotion. These examples underscore that U , R , and S are not theoretical abstractions but interpretable roles in multimodal emotion understanding. Modeling these factors remains challenging. Fusion strategies often entangle U , R , and S , leading to degenerate solutions where redundant signals dominate while unique and synergistic cues are suppressed. Contrastive methods such as CoMM [ 6 ] attempt to emphasize underutilized signals through augmentation, but they rely on crafted manipulations of inputs that may distort high-level emotional cues. For example, changing the color of a flower image is a valid augmentation for object recognition, whereas applying a similar transformation to an image of a laughing person may corrupt the abstract and semantically fragile cue of a smile. Factorization-based methods such as SMURF [ 34 ] highlight the benefits of separating unique and redundant subspaces, but they neither enhance the informativeness of shared representations nor explicitly capture synergy. To address these limitations, we propose a two-phase framework based on the principle of Divide and Refine - DnR . In the first phase ( Divide ), modality representations are disentangled into uniqueness, redundancy, and synergy, yielding structured features that explicitly separate informational roles. In the second phase ( Refine ), targeted objectives are applied: redundancy is augmented in representation space to improve robustness, while uniqueness and synergy are preserved to maintain distinct contributions. This design enhances the informativeness of multimodal features and remains compatible with diverse MER backbones. Our main contributions are threefold: • We introduce a unified formulation of multimodal representations for MERC grounded in Partial Information Decomposition, where the Divide phase disentangles modality features into uniqueness, redundancy, and synergy. • We propose a redundancy-focused augmentation and refinement strategy, forming the Refine phase, which strengthens shared information while preserving the distinct contributions of unique and synergistic signals. • We integrate the proposed Divide and Refine - DnR framework as a model-agnostic module and demonstrate consistent improvements across multiple MER backbones on IEMOCAP and MELD, validating its generality and effectiveness. 2 Related Work 2.1 Multimodal Emotion Recognition Multimodal emotion recognition in conversation (MERC) aims to classify utterance-level emotions by jointly leveraging speech, text, and visual modalities. Early studies primarily focused on textual context, with models such as DialogueGCN [ 9 ] , COSMIC [ 8 ] , and DialogueCRN [ 13 ] employing sequential or graph-based architectures to capture interaction flow. More recently, transformer-based approaches like DialogXL [ 30 ] incorporated pretrained language models to enhance context modeling. To integrate multimodal signals, fusion-based methods combine features across modalities. Approaches such as bc-LSTM [ 32 ] and CMN [ 10 ] adopt temporal modeling or simple feature concatenation, while more advanced designs such as SMIN [ 18 ] and SDT [ 22 ] refine fusion through semi-supervised learning and transformer-based hierarchical integration. However, these approaches often allow dominant modalities to overshadow subtle yet informative cues, making them sensitive to modality imbalance. Graph-based methods attempt to address this limitation by explicitly encoding conversational and multimodal structures. MMGCN [ 14 ] showed the benefits of undirected graphs for multimodal integration, while COGMEN [ 16 ] and CORECT [ 24 ] introduced structured conversation graphs with enhanced interaction modeling. More recently, GraphSmile [ 17 ] unified emotion recognition and sentiment analysis through multi-task optimization. Despite these advances, most existing approaches emphasize architectural design while overlooking the informational roles carried by each modality. Multimodal signals may contribute uniquely, redundantly, or synergistically, and failing to disentangle these factors limits the expressiveness of learned representations. This motivates our representation-centric perspective, where we employ Partial Information Decomposition (PID) as a principled tool to divide and refine the distinct

Read the complete text