A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis
Dongning Rao, Yunbiao Zeng, Zhihua Jiang, Jujian Lv · 2025
Read A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis by Dongning Rao, Yunbiao Zeng, Zhihua Jiang, Jujian Lv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22741.txt.
Introduction
Dongning Rao, Yunbiao Zeng, Zhihua Jiang, and Jujian Lv present TEXT, a model for reading sentiment across text, audio, and video together. Multimodal sentiment analysis matters wherever machines interact with people, but the value of generated explanations and of aligning the timing of speech and image streams has been underexplored. TEXT augments the data with explanations produced by multimodal large language models, then aligns audio and video representations through a temporal alignment block.
That block merges a Mamba architecture with temporal cross-attention. A new text-routed sparse mixture-of-experts, fused through its gates, then combines the modalities with the explanations. Across four datasets, TEXT beat every tested rival, including recent specialist models and multimodal large models, winning at least four of six metrics everywhere.
Catalog record
| CREATOR | Dongning Rao, Yunbiao Zeng, Zhihua Jiang, Jujian Lv |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis Dongning Rao, Yunbiao Zeng, Zhihua Jiang, Jujian Lv Abstract Human-interaction-involved applications underscore the need for M ulti-modal S entiment A nalysis (MSA). Although many approaches have been proposed to address the subtle emotions in different modalities, the power of explanations and temporal alignments is still underexplored. Thus, this paper proposes the T ext-routed sparse mixture-of- E xperts model with e X planation and T emporal alignment for MSA (TEXT). TEXT first augments explanations for MSA via M ulti-modal L arge L anguage M odels (MLLM), and then novelly aligns the representations of audio and video through a temporality-oriented neural network block. TEXT aligns different modalities with explanations and facilitates a new text-routed sparse mixture-of-experts with gate fusion. Our temporal alignment block merges the benefits of Mamba and temporal cross-attention. As a result, TEXT achieves the best performance across four datasets among all tested models, including three recently proposed approaches and three MLLMs. TEXT wins on at least four metrics out of all six metrics. For example, TEXT decreases the mean absolute error to 0.353 on the CH-SIMS dataset, which signifies a 13.5% decrement compared with recently proposed approaches. *Corresponding authors who contributed equally. Copyright © 2026, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Code — https://github.com/fip-lab/TEXT 1 Introduction Applications in healthcare and human-computer interaction rely heavily on multi-modal sentiment analysis. Thus, popular datasets for MSA like MOSI ( 24 ) , are proposed. However, more comprehensive approaches are needed to understand the subtle emotional nuances conveyed in audio and video ( 22 ) . Figure 1: An example of MSA from MOSI. On the left side, from top to bottom, are the textual, visual, and audio modalities of the short video. The corresponding modalities with MLLM-generated explanations are on the right. On the left side, only text can correctly predict the polarity, with a predicted score of 2.500. As a comparison, the label of this example is 1.400, and the TEXT’s prediction is 1.390. With explanations, both video and audio modalities can correctly predict the polarity. The left part of Fig. 1 is an example from MOSI, where MSA demands us to predict not only the polarity but also a score for short videos. For this example, only the text modality can correctly predict the polarity, and the estimated score using all modalities from previous studies ( 27 ) is 1.080. Considering that the label of this sample is 1.400 (deviation: 0.320), there is still room for improvement. While fusion is the key to a comprehensive understanding ( 27 ) , recent studies notice that different modalities contribute disparately to MSA ( 22 ) . For example, there is always a dominant modality ( 8 ) (e.g., the text in Fig. 1 ) and text-guide fusion is promising ( 22 ) . However, as two-thirds of the modalities might cause misjudgment in Fig. 1 , we posit that alignment is the crucial link between representation learning and multimodal fusion. Moreover, despite the aforementioned advances, the power of text in this large language model (LLM) ( 2 ) era has not been fully explored. Therefore, we propose the T ext-routed sparse mixture-of- E xperts model with e X planation and T emporal alignment for multi-modal sentiment analysis (TEXT) in this paper. • To explore the power of text, TEXT first facilitates multi-modal LLMs (MLLM) to generate explanations that will be encoded by BERT ( 7 ) . The MLLM is VideoLLaMA 3 ( 26 ) , which is fine-tuned with the EMER-fine dataset ( 10 ) . • Then, considering how video or audio can mislead (such as in Fig. 1 ), this study aligns audio (Librosa ( 12 ) ) and video (OpenFace ( 3 ) ) encoding with explanations for the first time. This module uses a C ross- A ttention(CA)-based alignment. • To improve temporal fusion, we propose a novel temporal alignment between the aligned audio/video representations. This block combines, simplifies, and outperforms Mamba ( 6 ) and temporal CA ( 28 ) . • Then, for better decisions, we implement a text-routed sparse mixture-of-experts (SMoE) ( 18 ) because of the dominance of text. That is, experts are activated based on the text. • At last, a multi-layer perceptron (MLP) with gate fusion (GF) ( 14 ) is applied as a classifier. The right part of Fig. 1 is the result of TEXT. With explanations, both visual and audio modalities can correctly predict the polarity. Furthermore, the final score given by TEXT is 1.390, which is close to 1.400 (deviation: 0.010). By contrast, the representative MLLM, QWen 2.5-vl ( 2 ) predicts a score of 2.500 (deviation: 1.100). The deviation is significantly reduced from 1.100 to 0.010, indicating a substantial improvement in prediction accuracy. To test the effectiveness of TEXT, we compare TEXT with three recently proposed models and three MLLMs on four datasets. The three compared models are ALMT ( 27 ) , KuDA ( 8 ) ,, and DEVA ( 22 ) . The three MLLMs are Qwen2.5-vl ( 2 ) , GPT-4o ( 1 ) and VideoLlama3-7B ( 26 ) ; the four datasets are MOSI ( 24 ) , MOSEI ( 25 ) , CH-SIMS ( 23 ) , and CH-SIMSv2 ( 11 ) . Experiment results show that TEXT outperforms all compared models. For example, on CH-SIMS TEXT improve the mean absolute error (MAE) from 0.449 (ALMT), 0.408 (KuDA), and 0.424 (DEVA) to 0.353 (i.e., a decrease of 13.5%). Further, our ablation study suggests that temporal alignment is the most crucial component for MAE. Our contribution can be summarized as follows: 1. TEXT aligns encoded audio and video through a novel temporality-oriented neural network block; 2. TEXT first augments data for MSA via MLLM with explanations that can be aligned with audio and video; 3. TEXT uses a new text-routed SMoE; 4. TEXT is the winner among all tested models on four datasets across six metrics. 2 Related Work 2.1 Multi-modal Sentiment Analysis MSA has been investigated for its attractive applications like fraud detection ( 13 ) trading systems ( 4 ) , human-machine interaction ( 16 ) , and health care applications ( 17 ) . Existing methods for MSA can be classified into two categories: representation learning-centered methods (e.g., ALMT and KuDA) and multi-modal fusion-centered methods ( 27 ) (e.g., DEVA). Both KuDA and DEVA are claimed to be state-of-the-art ( 8 ; 22 ) . Formally, inputs of MSA models include text ( t ), audio ( a ), and visual ( v ). Our goal is to fuse the data from different modalities and predict the sentimental polarity \hat{y} along with a sentiment score between [-1, 1] or [-3, 3]. A score greater than, equal to, or less than zero represents positive, neutral, and negative, respectively. As a regression problem, the basic optimization objective function of MSA is the MSE loss. 2.2 Cross-Modal Temporal Fusion Figure 2: Two temporal alignment block designs: (a) TCA, and (b) Mamba. Legends are also applicable for Fig. 5 \sim 6 . In (a), F_{v} , F_{a} , and F_{t} represent the input vectors to