Multimodal Video Emotion Recognition with Reliable Reasoning Priors
Zhepeng Wang, Yingjian Zhu, Guanghao Dong, Hongzhu Yi, Feng Chen, Xinming Wang *, Jun Xie * · 2025
Read Multimodal Video Emotion Recognition with Reliable Reasoning Priors by Zhepeng Wang, Yingjian Zhu, Guanghao Dong, Hongzhu Yi, Feng Chen, Xinming Wang *, Jun Xie * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03722.txt.
Introduction
Zhepeng Wang and co-authors feed machine-written notes into a video emotion reader in a 2025 paper. Judging feelings in clips is hard when some feelings show up far more than others. The team had a large vision-word model write fine notes on each clip, split by channel, and fed those notes in as prior guides.
A new loss scheme evens out both the gaps between feeling classes and the spread inside each class. On a 2024 test set, the prior-fed design gained a lot over the same net without the notes. Careful notes from a big model, the team says, pair well with a light fusion net.
Catalog record
| CREATOR | Zhepeng Wang, Yingjian Zhu, Guanghao Dong, Hongzhu Yi, Feng Chen, Xinming Wang *, Jun Xie * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Video Emotion Recognition with Reliable Reasoning Priors Zhepeng Wang, Yingjian Zhu, Guanghao Dong, Hongzhu Yi, Feng Chen, Xinming Wang *, Jun Xie * Abstract This study investigates the integration of trustworthy prior reasoning knowledge from MLLMs into multimodal emotion recognition. We employ Gemini to generate fine-grained, modality-separable reasoning traces, which are injected as priors during the fusion stage to enrich cross-modal interactions. To mitigate the pronounced class-imbalance in multimodal emotion recognition, we introduce Balanced Dual-Contrastive Learning, a loss formulation that jointly balances inter-class and intra-class distributions. Applied to the MER2024 benchmark, our prior-enhanced framework yields substantial performance gains, demonstrating that the reliability of MLLM-derived reasoning can be synergistically combined with the domain adaptability of lightweight fusion networks for robust, scalable emotion recognition. 1 1 footnotetext: Corresponding Author. 1 Introduction Recent advances in reinforcement learning [ 1 ] with verified reward (RLVR) [ 2 ] have accelerated the adoption of reinforcement-learning-based post-training and chain-of-thought [ 3 ] procedures that follow a Reflection–Answer paradigm across multimodal large language models (MLLMs), gaining great traction. These substantially expand the generalization frontier beyond what standard in-context learning or supervised fine-tuning (SFT) [ 4 ] can achieve. Empirical studies consistently show that embedding a CoT paradigm within MLLMs markedly improves generalization. Nevertheless, this test-time scaling [ 5 ] strategy also introduces considerable computational overhead during both training and inference, exceeding that of conventional multimodal fusion pipelines. Moreover, identifying the appropriate level of reasoning granularity for different task scenarios remains an open question. Focusing on multimodal video emotion recognition, this work advances two central inquiries: (i) For a well-trained reasoning model, can its strong, generalized reasoning priors be integrated into an LLM-agnostic, multimodal recognition framework? (ii) In multimodal video analysis, how can we conduct reliable and coherent reasoning over the emotional cues conveyed by both the human subjects and their surrounding context? To address the aforementioned challenges, we propose a modified multimodal fusion framework. This approach elicits reliable reasoning priors from MLLMs and then fuses these priors into a task-specific multimodal architecture to guide subsequent training. Conceptually, this can be viewed as targeted distillation, where the MLLMs acts as a teacher model at the instructional level and the traditional lightweight networks serve as a student model at the implementation level. Such distillation elevates the performance ceiling of the student model. Our contributions are as follows: • We distill high-level reasoning priors from MLLMs into a lightweight multimodal recognition model, enhancing generalization with minimal computational cost. • We introduce a balanced contrastive strategy to address label imbalance and improve emotion class separability in multimodal feature space. 2 Related Work Video Emotion Recognition Development of multimodal fusion and LLMs brings new prospects for video emotion recognition. transformer audio-visual fusion method like AVT-CA [ 6 ] and TACFN [ 7 ] which synchronises and filters cross-modal cues for new highs on CMU-MOSEI [ 8 ] , RAVDESS [ 9 ] . Vision-language prompting method like EmoVCLIP [ 10 ] and SoVTP [ 11 ] utilise prompt tuning and modality dropout in the multimodal fusion process. Also, Video-Emotion-OVR [ 12 ] and AffectGPT [ 13 ] shift from closed-set labels to free-form textual emotion descriptions, enriching emotion expression. Visual Reasoning Model As the CoT progressively propagates the notion of “System-2” reasoning [ 14 ] , the research community has begun to focus more intently on inference mechanisms for large multimodal models. Visual-RFT [ 15 ] has empirically validated the efficacy of RLVR in purely visual settings, while VCTP leverages visual chain-of-thought prompts to enhance zero-shot visual reasoning. [ 16 , 17 , 18 ] have further sought to improve the interpretability and trustworthiness of model outputs [ 19 ] . Nevertheless, whether such reasoning traces can be distilled into reliable prior knowledge that meaningfully guides conventional models remains an open and intriguing research question. 3 Method 3.1 Emotion Recognition with Reliable Reasoning Although MLLMs exhibit impressive zero-shot capabilities and strong generalisability on general-purpose benchmarks, they often struggle to achieve high discriminative accuracy on domain-specific video-emotion corpora—underscoring a sizeable domain gap between broad and specialised tasks. Reliable identification and categorization of human emotions have long been a focal point of sociological research. Traditionally, scholars have modelled affect by mapping facial expressions onto either the Arousal–Valence [ 20 , 21 ] continuum or the Facial Action Coding System (FACS) [ 22 ] . FACS localises 44 anatomically defined Action Units (AUs), and canonical combinations of these AUs constitute six basic-emotion prototypes, providing a robust quantitative framework for emotion classification. Figure 1 : Mapping between Action Units and emotions Let \mathcal{C}=\{c_{1},c_{2},\dots,c_{|\mathcal{C}|}\} denote the set of discrete emotion classes and let \mathcal{A}=\{\text{AU}_{1},\text{AU}_{2},\dots,\text{AU}_{44}\} denote the complete repertoire of facial Action Units. For every emotion label c\in\mathcal{C} we have its AU-support set as \mathcal{S}_{c}\;:=\;\bigl\{\,\text{AU}_{j}\in\mathcal{A}\;\bigm|\;\text{AU}_{j}\;\text{is associated with emotion }c\bigr\}\; (1) When performing emotion recognition, describing facial expressions through AU patterns as I_{V}:=\{\text{AU}_{i}\in\mathcal{A}\} offers empirical support for labelling decisions, where I generated from MLLMs with Instructions. As Gemini model family demonstrates strong multimodal reasoning and generalisation, we employ the Gemini-2.0-exp [ 23 ] model to analyse representative video samples along three complementary dimensions: • AU combinations extracted from facial frames I_{V} • Prosodic cues derived from the audio track I_{A} ; • Semantic content from subtitle transcripts I_{T} . The model further quantifies the relative contribution of video, audio, and textual channels, enabling an integrated affective judgement. Figure 2 : Reliable Reasoning Process The whole reasoning priors can be expressed as P:=\{c^{*}~|~I_{V},I_{A},I_{T},R\} (2) where R denotes the contribution of each modality. For each modality prior, it may imply different emotion labels c with a latent mapping S(I_{m}):\rightarrow{c} , MLLMs will synthesize all priors to recognize a certain emotion. c^{*}=\arg\max S(I_{m}) (3) In subsequent training stages, these trustworthy reasoning traces can be incorporated as priors guided dataset, guiding the multimodal fusion network and enhancing its integrative performance. 3.2 Trustworthy MultiModal Fusion Paradigm Annotating multimodal video–based emotion-recognition corpora is intrinsically difficult, as automated labeling pipelines typically yield sub-par accuracy, whereas human verification is prohibitively expensive. As a result, these datasets characteristically contain a substantial proportion of unlabeled samples. Each video segment can be decomposed into three complementary information streams—visual frame sequences, acoustic waveforms, and transcript-level dialogue text. After passing each stream through its dedicated modality-specific encoder, we obtain three heterogeneous feature vectors, which are subsequently projected and harmonised within a common latent space by a unified feature projector. Figure 3 : The whole framework of the multimodal fusion paradigm. Our multimodal fusion framework is optimised through a two-stage training regimen as Fig 3 : The first stage is Large-scale Semi-supervised Pre-training , which leverages both labeled and unlabeled data to learn robust cross-modal representation. And the second stage is Reliable Prior Guided Tuning , wherein domain-specific priors are incorporated to refine the model for downstream emotion-recognition tasks with enhanced reliability. During the first stage of large-scale pre-training, our objective is to enforce intra-class compactness while simultaneously maximizing inter-class separability across all emotion categories. Therefore, we need to pull the feature vectors of the same category as close as possible, and pull the feature vectors of different categories as far as possible. The contrastive learning strategy of infoNCE can effectively achieve this separability. However, since the labels of the original expression