Tri-MCA fusion: cross-modal attention and dynamic gating for multimodal sentiment analysis Authors: Asmaa Alrayzah Published: 2026 ABSTRACT AbstractMultimodal sentiment analysis aims to automatically infer human emotions by jointly analyzing information from multiple modalities, such as text, audio, and visual signals. Although recent deep learning approaches have significantly improved multimodal representation learning, effectively modeling interactions among heterogeneous modalities while handling modality imbalance and noisy signals remains challenging. Existing fusion strategies often rely on static or limited cross-modal interactions, which may fail to capture complex dependencies among modalities and consequently limit sentiment prediction performance. To address these challenges, this paper proposes a tri-modal cross-attention architecture with adaptive gating mechanisms for multimodal sentiment analysis. The proposed framework introduces a Tri-modal cross-attention module to better capture interactions among textual, acoustic, and visual modalities. The model is evaluated on three widely used benchmark datasets: CMU-MOSI, CMU-MOSEI, and SIMS. Experimental results demonstrate that the proposed method achieves improved performance across multiple evaluation metrics. The model obtains 85.6% Acc2 and an F1-score of 85.2 on CMU-MOSI, outperforming several existing multimodal sentiment analysis models. Similar improvements are observed on CMU-MOSEI and SIMS. Furthermore, ablation studies confirm the contribution of each component, highlighting the importance of combining cross-modal attention with adaptive gating mechanisms for robust multimodal sentiment prediction.Subject terms: Engineering, Mathematics and computing, Neuroscience Introduction Human communication is inherently multimodal, with verbal language complemented by vocal tone, facial expressions, gestures, and contextual visual cues. These multimodal signals collectively convey emotional and affective information that cannot be fully captured through textual content alone. Sentiment analysis, traditionally focused on textual data, has therefore evolved into multimodal sentiment analysis (MSA)1, which aims to automatically infer sentiment by jointly analyzing information from multiple modalities, such as text, audio, and visual data. MSA has attracted increasing attention because of its broad applicability in human–computer interaction, social media analytics, opinion mining, healthcare monitoring, and intelligent conversational systems. Although text remains the primary carrier of semantic sentiment information, relying solely on textual cues often leads to inaccurate sentiment interpretation, particularly in cases involving sarcasm, implicit emotions, or ambiguous linguistic expressions2. For example, vocal prosody may reveal emotional intensity that contradicts the textual sentiment, while facial expressions can provide additional contextual cues regarding the speaker’s emotional state. Consequently, integrating multimodal information has been shown to significantly improve sentiment prediction accuracy and robustness. However, the heterogeneous nature of multimodal data introduces several challenges that continue to limit the effectiveness of existing MSA systems3. One major challenge is modality heterogeneity and imbalance. Different modalities often contribute unequally to sentiment understanding, and their reliability may vary across samples. For instance, audio signals may contain background noise, visual signals may suffer from occlusion or poor video quality, and textual transcripts may lack sufficient contextual emotional cues2. Many existing multimodal fusion approaches combine modalities using fixed fusion strategies or separate pairwise attention mechanisms that do not directly model joint tri-modal dependencies and often assume consistent modality reliability. Such assumptions may lead to suboptimal performance when modalities contain conflicting or noisy information1. Although transformer-based cross-attention architectures such as MulT introduce dynamic modality interaction, these interactions are typically implemented through separate directional pairwise attention operations between modalities rather than unified tri-modal interaction within a single attention mechanism. Another critical challenge involves the temporal dynamics of multimodal sentiment expression. Sentiment is rarely expressed instantaneously; instead, it evolves across sequences of spoken utterances, facial movements, and acoustic variations2. Although pretrained transformer-based encoders have demonstrated strong contextual representation capabilities, they do not always capture fine-grained temporal sentiment progression within multimodal sequences. Ignoring temporal dependencies may therefore limit a model’s ability to capture subtle sentiment transitions and variations in emotional intensity. Furthermore, recent studies have highlighted the importance of preserving modality-specific sentiment characteristics4. Several advanced architectures attempt to learn shared multimodal representations by enforcing early modality interaction, which may unintentionally suppress unique modality-specific cues. Preserving modality specialization before fusion is essential to ensure that each modality contributes complementary emotional information effectively. However, balancing modality specialization with multimodal interaction remains an open research problem5. In addition to representation-related challenges, existing MSA frameworks often employ highly complex architectures involving multistage cross-modal attention, representation disentanglement modules, or knowledge distillation pipelines. Although these methods improve performance, they also significantly increase computational complexity and reduce practical deployability in real-world applications6. Therefore, designing efficient multimodal frameworks that maintain high sentiment prediction accuracy while preserving computational feasibility remains an important research objective. To address these challenges, this paper proposes a tri-modal cross-attention architecture with adaptive gating mechanisms for multimodal sentiment analysis. The proposed model facilitates rich interactions among text, audio, and visual modalities through a tri-modal cross-attention (Tri-MCA) module that captures cross-modal dependencies while preserving modality-specific representations. In addition, a subspace gating mechanism is introduced to filter irrelevant features within each modality, enabling the model to focus on sentiment-relevant signals. To address modality imbalance, an expert gating module dynamically learns the relative importance of each modality, allowing adaptive modality fusion based on the reliability of the input signals. By integrating cross-modal reasoning and adaptive gating within a unified framework, the proposed approach aims to improve sentiment prediction accuracy while maintaining computational efficiency. The main contributions of this work are summarized as follows: A joint tri-modal cross-attention mechanism is proposed, in which each modality simultaneously attends to the combined context of the other two modalities through a stacked key–value formulation. Unlike the pairwise cross-attention used in prior frameworks such as MISA7, this design captures three-way modality dependencies within a single attention operation while preserving modality-specific representations. A joint tri-modal cross-attention mechanism is proposed, in which each modality simultaneously attends to the combined context of the other two modalities through a stacked key–value formulation. Unlike the pairwise cross-attention used in prior frameworks such as MISA7, this design captures three-way modality dependencies within a single attention operation while preserving modality-specific representations. A dimension-wise subspace gating module is introduced, treating the 768-dimensional modality embedding as 768 independent subspaces and applying a learnable gate to each dimension. In contrast to representation-level disentanglement methods such as DLF6 and ULMD8, this finer-grained gating enables the model to suppress noisy or sentiment-irrelevant features at the individual-dimension level. A dimension-wise subspace gating module is introduced, treating the 768-dimensional modality embedding as 768 independent subspaces and applying a learnable gate to each dimension. In contrast to representation-level disentanglement methods such as DLF6 and ULMD8, this finer-grained gating enables the model to suppress noisy or sentiment-irrelevant features at the individual-dimension level. An adaptive expert gating network is developed to dynamically estimate sample-specific modality weights by combining a temperature-scaled softmax with modality dropout. This design mitigates collapse onto a single dominant modality, which is a common failure mode in language-dominant multimodal fusion. An adaptive expert gating network is developed to dynamically estimate sample-specific modality weights by combining a temperature-scaled softmax with modality dropout. This design mitigates collapse onto a single dominant modality, which is a common failure mode in language-dominant multimodal fusion. A unified end-to-end framework integrates the three components such that cross-modal interaction, fine-grained feature filtering, and adaptive modality weighting reinforce one another within a single optimization objective, rather than functioning as independently stacked modules. A unified end-to-end framework integrates the three components such that cross-modal interaction, fine-grained feature filtering, and adaptive modality weighting reinforce one another within a single optimization objective, rather than functioning as independently stacked modules. Comprehensive experiments and ablation studies are conducted on three benchmark datasets (CMU-MOSI, CMU-MOSEI, and SIMS), covering both English and Chinese multimodal sentiment data. The results demonstrate consistent improvements over competitive baselines, while the ablation analysis validates the contribution of each architectural component. Comprehensive experiments and ablation studies are conducted on three benchmark datasets (CMU-MOSI, CMU-MOSEI, and SIMS), covering both English and Chinese multimodal sentiment data. The results demonstrate consistent improvements over competitive baselines, while the ablation analysis validates the contribution of each architectural component. The remainder of this paper is organized as follows. Section “Related Work” reviews related work in multimodal sentiment analysis. Section “Methodology: Tri-MCA Fusion” describes the proposed architecture and model components. Section “Experimental Design” presents the experimental setup, datasets, and evaluation metrics. Section “Results and Discussion” reports the experimental results and analysis. Section “Ablation Study” presents the ablation studies. Finally, Section “Conclusion” concludes the paper and discusses potential directions for future research. Related work Multimodal sentiment analysis has evolved from traditional unimodal sentiment analysis toward sophisticated multimodal learning frameworks that integrate textual, auditory, and visual signals. Early sentiment analysis systems focused primarily on textual representations. However, the inherent limitations of text-only approaches in capturing implicit emotions, sarcasm, and nonverbal cues motivated the transition to multimodal paradigms1. Recent advances in deep learning (DL) and transformer-based architectures have significantly accelerated progress in this field. This section reviews existing literature from five major perspectives: multimodal fusion strategies, unimodal feature extraction, representation disentanglement and modality decomposition, robustness under missing or imbalanced modalities, and reasoning-driven multimodal sentiment modeling. Fusion strategies in multimodal sentiment analysis Fusion design remains a central challenge in multimodal sentiment analysis (MSA). Existing approaches can generally be categorized into early fusion, late fusion, hybrid fusion, tensor fusion, hierarchical fusion, graph-based fusion, and attention-based fusion methods1. Early fusion methods concatenate unimodal features before classification. Although straightforward, this approach often suffers from dimensional explosion and inadequate modeling of modality-specific characteristics. Late fusion aggregates unimodal predictions but ignores fine-grained cross-modal interactions. Tensor fusion captures high-order interactions but introduces substantial computational overhead. Hierarchical and contextual fusion approaches attempt to model temporal and semantic dependencies more effectively. Attention-based mechanisms have become dominant because of their ability to dynamically assign weights across modalities and temporal steps1. For example, transformer-based cross-modal attention architectures enable flexible interactions among modalities. However, many of these approaches implicitly assume balanced modality contributions and complete modality availability, limiting their robustness in real-world settings. Recent engineering-oriented architectures, such as UFEN and MTFN, combine multilayer unimodal feature extraction with multitask fusion to balance intra-modal and inter-modal learning9. These models demonstrate that improving unimodal representation quality can substantially enhance multimodal fusion performance. Despite these advances, existing fusion strategies continue to face challenges related to modal heterogeneity, inter-modal imbalance, and synchronization1. Unimodal feature extraction Effective unimodal encoders form the backbone of MSA systems. For text, transformer-based models such as BERT10 and its variants have become dominant because of their strong contextual representation capabilities. For audio, commonly used feature extraction tools include OpenSMILE11, COVAREP12, and, more recently, self-supervised models such as Wav2Vec13, which capture robust acoustic representations. Visual modalities are typically modeled using CNN- or transformer-based encoders to extract facial and contextual features. Recent studies have emphasized multiscale and contextual intra-modal modeling. For instance, UFEN integrates Bi-GRU4, convolutional layers, and self-attention mechanisms to capture both local and global unimodal dependencies. However, improvements in unimodal encoders alone do not fully resolve issues related to cross-modal redundancy and modality conflicts9. Temporal modeling in multimodal sentiment analysis Temporal modeling is another important challenge in MSA because sentiment cues often evolve across speech frames, facial expressions, and utterance-level contexts. Earlier MSA models commonly employed recurrent architectures such as LSTM, GRU, and BiGRU to capture sequential dependencies in acoustic and visual streams. LSTM-based models are effective at modeling long-term dependencies but introduce relatively high parameter complexity because of their multiple gating mechanisms. GRU and BiGRU provide more compact alternatives by using fewer gates while preserving the ability to model temporal progression. In contrast, temporal transformer architectures can capture long-range dependencies through self-attention, but they generally require larger training datasets and higher computational costs, particularly when applied to frame-level audio and visual sequences. In this work, BiGRU is adopted as a lightweight temporal refinement module on top of pretrained Wav2Vec2 and ViT representations. This design enables the model to capture bidirectional temporal cues in speech and facial expressions before cross-modal interaction while avoiding the additional computational overhead associated with deeper temporal transformer blocks. Representation disentanglement and modality decomposition Many studies argue that treating all modalities equally during fusion leads to redundancy and semantic interference. To address this issue, recent works have proposed disentangling shared and modality-specific representations6. The disentangled-language-focused multimodal sentiment analysis (DLF) framework introduces cascaded encoders to separate shared and modality-specific features, combined with orthogonality and triplet losses to reduce cross-modal redundancy and enhance discriminative capability. Similarly, ULMD decomposes modality representations into invariant and modality-specific components and introduces unimodal label generation to provide auxiliary supervision8. MUKDF extends this idea by incorporating a dual-branch modality knowledge extractor (DMKE) that explicitly separates modality-common and modality-specific knowledge, thereby improving interpretability and robustness5. Although these methods reduce feature interference, most primarily focus on representation-level disentanglement without incorporating higher-level reasoning mechanisms. Robustness under missing or imbalanced modalities Real-world multimodal systems frequently encounter incomplete or corrupted modality inputs. Consequently, missing-modality learning has become a critical research direction. MUKDF proposes a modality random missing strategy (MRMS) combined with teacher–student knowledge distillation to improve student-model robustness under uncertain modality configurations5. The framework employs contrastive and alignment-based distillation mechanisms to minimize representation gaps between teacher and student models. Similarly, earlier distillation-based approaches attempt to reconstruct missing modalities or transfer cross-modal knowledge, but they often fail to capture fine-grained semantic alignment effectively. Despite these advances, most robustness-oriented methods primarily focus on feature reconstruction or representation alignment, while modality reliability modeling and dynamic reasoning remain relatively underexplored. Reasoning-driven multimodal sentiment analysis Recently, research has begun shifting from pure feature fusion toward reasoning-based multimodal modeling. Traditional MSA systems rely on learned representations without explicitly modeling the logical inference processes underlying sentiment prediction. The MM-PEAR-CoT framework introduces chain-of-thought prompting into MSA, enabling language models to generate structured reasoning steps and use multimodal signals to verify or refine these reasoning outputs2. This paradigm represents a transition from representation-driven to reasoning-guided multimodal learning. More recent studies have further explored reasoning-aware and aesthetic-aware approaches to multimodal sentiment understanding. Vanessa introduced a visual connotation and aesthetic attribute framework that models implicit emotional cues derived from visual aesthetics and semantic alignment through graph-based multimodal learning and self-supervised contrastive optimization14. Similarly, AIFFM proposed an auxiliary information fusion and semantic filtering framework that leverages large language models to integrate aesthetic captions, emotional impressions, and semantic representations for fine-grained multimodal sentiment analysis15. In addition, Chimera investigated cognitive and aesthetic sentiment causality by incorporating semantic rationale generation, affective-cognitive resonance, and object-level multimodal alignment to enhance fine-grained sentiment reasoning16. Collectively, these studies reflect a growing shift toward multimodal frameworks capable of modeling deeper semantic, emotional, and aesthetic reasoning processes beyond conventional feature fusion mechanisms. Early multimodal sentiment analysis frameworks primarily focused on feature-level fusion through recurrent architectures, tensor fusion, and attention-based interaction mechanisms. More recent approaches have explored graph-based and adaptive multimodal fusion strategies to improve sentiment-aware representation learning. For example, MAGCN introduced a multichannel attentive graph convolutional framework that incorporates sentimental knowledge into intermodality interactions through graph convolutional learning and sentiment-aware attention mechanisms17. This approach demonstrated the effectiveness of graph-based multimodal interaction for modeling sentiment-specific cross-modal dependencies. Recent studies have also investigated adaptive multimodal optimization and multitask multimodal learning. MATADOR proposed a multimodal multisource aggregation framework enhanced by emotion recognition and dynamic assistance mechanisms for robust multimodal decision-making18. Although primarily designed for traffic accident prediction, the framework demonstrated the effectiveness of adaptive multimodal fusion and task-aware optimization in complex multimodal environments. Similarly, adaptive multifeature extraction graph convolutional networks introduced an adaptive graph-based multimodal framework that dynamically extracts and fuses heterogeneous multimodal features through graph convolutional operations to improve target-aware sentiment representation learning19. However, reasoning-based multimodal methods remain relatively computationally intensive and are still not closely integrated with adaptive modality reliability modeling and lightweight dynamic fusion strategies. Existing MSA research has achieved significant progress in multimodal fusion, unimodal representation learning, and robustness modeling. Despite these advances, several limitations remain. First, many fusion strategies rely on static or attention-based integration mechanisms that do not adequately capture modality reliability or dynamically adjust modality contributions when sentiment information is inconsistent across modalities. Second, although recent approaches emphasize representation disentanglement and modality-specific learning, they often introduce complex architectures that increase computational overhead and reduce deployment efficiency. Third, robustness-oriented methods primarily focus on modality reconstruction or knowledge distillation, which may not effectively capture temporal sentiment dynamics within each modality. Finally, reasoning-driven models have demonstrated promising interpretability but often require large-scale reasoning modules and do not sufficiently exploit strong modality-specific temporal encoding. Table 1 presents a comparison of representative MSA methods. Table 1Comparison of representative multimodal sentiment analysis methods.Study/ModelMain strategyKey componentsDataset(s) evaluatedResults (F1)StrengthsLimitationsUFEN + MTFN9Multilayer fusion + multitask learningUnimodal feature extraction network (UFEN), Multitask fusion network (MTFN), CNN + BiGRU + attention + transformerCMU-MOSI, CMU-MOSEI, CH-SIMS85.2%,84.9%,81.27%Stable prediction under sentiment inconsistencyDoes not explicitly model modality uncertainty or disentanglementMUKDF5Robust MSA under missing modalitiesTeacher–student knowledge distillation, dual-branch modality knowledge extractor (DMKE), multimodal masked transformer (MMT), MRMS strategyMOSI, IEMOCAP84.89%,83.37%Robustness to missing or corrupted modalities; improved cross-modal interaction modelingIncreased model complexity; requires teacher model trainingDLF Framework6Disentangled representation learningShared/specific modality encoders, language-focused attractor, orthogonality, and triplet lossesCMU-MOSI, CMU-MOSEI85.04%,85.27%Reduces redundancy and modality conflict; enhances language-dominant fusionFocuses primarily on representation learning without uncertainty modelingULMD8Modality decomposition + unimodal pseudo supervisionUnimodal label generation module, shared/specific feature decomposition, multitask supervisionMOSI, MOSEI85.71%,85.91%Provides additional supervision signals; improves modality independencePseudo-label generation may introduce noiseSMP Multichannel Fusion20Dual-channel feature extraction + symmetric attention fusionWav2Vec + MFCC (audio), Facial AU + CNN features (visual), symmetric mutual promotion attentionCMU-MOSI,CH-SIMS83.21%,79.38%Captures complementary local/global features; strong cross-modal interactionRelies on handcrafted and deep feature combinations; limited reasoning modelingMM-PEAR-CoT2Reasoning-driven MSAChain-of-thought prompting, self-consistency inference, cross-modal filtering, and fusionMOSI, MOSEI88.2%,88.4%Introduces multimodal reasoning validation; reduces hallucinated sentiment inferenceComputationally expensive; still early-stage paradigm Comparison of representative multimodal sentiment analysis methods. 85.2%, 84.9%, 81.27% 84.89%, 83.37% 85.04%, 85.27% 85.71%, 85.91% CMU-MOSI, CH-SIMS 83.21%, 79.38% 88.2%, 88.4% Methodology: Tri-MCA fusion The MSA task addressed in this work aims to predict sentiment polarity and intensity through the joint processing of textual, acoustic, and visual information. The task involves three primary modalities: text (T), audio (A), and visual (V). Let the multimodal input be defined as where , denotes the sequence length, and denotes the feature dimension for modality . Each modality is processed by its corresponding expert encoder to produce a fixed-dimensional embedding where B denotes the batch size. The embeddings are subsequently passed through stacked cross-modal attention layers, subspace gating modules, and a dynamic expert gating network to obtain a fused representation. The final model output is defined as representing the predicted continuous sentiment score, while the ground-truth sentiment label is denoted as . MSA requires the model to simultaneously extract discriminative intra-modal features and capture complementary relationships across heterogeneous modalities. Text, audio, and video express sentiment through different levels of abstraction and distinct temporal dynamics. Consequently, relying on a single modality may be insufficient when its sentiment signal is weak or inconsistent with the overall sentiment label. Furthermore, naive fusion strategies, such as feature concatenation or fixed weighted averaging, fail to account for the varying informativeness of modalities across different samples. To address these limitations, we propose Tri-MCA fusion, a multimodal architecture based on stacked cross-modal attention and dynamic expert gating, as illustrated in Fig. 1. Overview of the proposed tri-modal cross-attention (Tri-MCA) fusion architecture for multimodal sentiment analysis. The model operates through five main stages. First, modality encoders extract compact 768-dimensional embeddings from the text, audio, and video inputs using three independent expert encoders. Second, a stack of cross-modal attention layers enables each modality to iteratively query and integrate complementary information from the other two modalities. Third, a subspace gating module performs dimension-wise feature selection on the refined modality representations, allowing the model to emphasize informative features while suppressing noisy ones. Fourth, a dynamic expert gating network computes sample-adaptive modality weights and generates a fused multimodal representation. Finally, the fused representation is passed through a regression head that maps it to a continuous sentiment score. Although text, audio, and visual modalities exhibit different semantic densities and statistical properties, projecting modality-specific representations into a unified latent space facilitates stable cross-modal attention and multimodal alignment within the proposed Tri-MCA framework. Importantly, modality-specific characteristics are initially preserved through dedicated pretrained encoders before lightweight projection layers are applied to ensure interaction compatibility. Furthermore, the proposed subspace gating and adaptive expert gating mechanisms help reduce redundant feature propagation while preserving informative modality-specific cues during multimodal fusion. Modality-specific expert encoders Because the three modalities are heterogeneous in nature, each requires a specialized processing pipeline to extract meaningful sentiment-relevant features. All encoders share a common output interface, where each produces a 768-dimensional embedding , which is then forwarded to the fusion pipeline. Text expert The text modality is processed using a pretrained BERT-base-uncased model. Given a tokenized input sequence and its attention mask, BERT generates contextual token representations. The CLS token embedding from the final hidden layer is used as the sentence-level representation: Audio expert The audio modality is processed using a pretrained Wav2Vec2-base encoder. Given a raw waveform , Wav2Vec2 produces a sequence of frame-level acoustic representations with dimensionality 768. To capture temporal dependencies in speech signals, a bidirectional GRU (BiGRU) is applied on top of the Wav2Vec2 output: The forward and backward final hidden states are concatenated to produce a 1536-dimensional representation, which is then projected to 768 dimensions using a fully connected layer: Video expert The video modality is processed frame by frame using a pretrained vision transformer (ViT-base-patch16-224). Given an input sequence of T video frames per sample the frames are first reshaped and processed independently through ViT, extracting the CLS token embedding for each frame: The resulting sequence of per-frame embeddings is then passed through a bidirectional GRU to model the temporal evolution of facial expressions and visual sentiment cues across frames: The concatenated forward and backward final hidden states are projected to 768 dimensions: The use of BiGRU in both the audio and video experts is intentional. Sentiment in speech and facial expressions is inherently sequential; for example, a rising intonation at the end of a sentence or a smile that gradually fades throughout a clip carries important temporal information that frame-level or token-level representations alone cannot capture. The BiGRU integrates both past and future contextual information within each modality before the cross-modal fusion stage, ensuring that the embeddings and already encode temporally informed representations when they enter the cross-attention layers. Stacked cross-modal attention layer To model interdependencies among modalities and enable complementary information exchange, we introduce a stacked cross-modal attention mechanism. Each layer consists of three parallel cross-modal attention blocks, one for each modality, that operate simultaneously. Within each block, one modality serves as the query, while the remaining two modalities provide the key and value context. For example, in the text expert block, the text embedding queries the audio and video representations to identify which aspects of the acoustic and visual signals are most relevant for refining the textual sentiment representation. Formally, given a query modality q and context modalities , the context is first stacked as The multihead attention is then computed as where are the projected query, key, and value matrices, respectively, and represents the dimension of each attention head. The attention output is followed by a residual connection and layer normalization, after which a position-wise feed-forward network with GELU activation is applied. A second residual connection and layer normalization are then performed, following the standard transformer encoder architecture. By stacking N=2 such layers, the model performs multiple rounds of cross-modal interaction, allowing increasingly abstract and complementary representations to emerge across modalities. The proposed Tri-MCA mechanism differs from conventional cross-modal Transformer architectures in several important respects. First, unlike models such as MulT, which perform separate pairwise directional cross-attention operations (e.g., text-to-audio and text-to-visual independently), the proposed Tri-MCA module enables each modality to attend simultaneously to the combined contextual representations of the other two modalities through a stacked key–value formulation. This design allows direct modeling of higher-order tri-modal dependencies within a single attention operation. Second, the proposed framework employs stacked Tri-MCA layers, enabling modalities to iteratively refine their representations through repeated multimodal interaction. This progressive refinement process allows increasingly abstract and complementary multimodal sentiment representations to emerge across layers. Third, the Tri-MCA module is tightly integrated with the proposed dimension-wise subspace gating mechanism and dynamic expert gating network. Consequently, cross-modal interaction is jointly optimized with fine-grained feature filtering and adaptive modality weighting, rather than functioning as an isolated fusion module. This integrated design improves the model’s ability to suppress noisy modality features while dynamically emphasizing more informative sentiment cues. Dimension-wise subspace gating After the cross-modal attention layers, each modality’s refined representation is passed through a dedicated subspace gating module. Unlike coarse gating strategies that assign a single scalar weight to an entire feature vector, the proposed gating module operates at the level of individual feature dimensions. Specifically, the 768-dimensional vector is treated as 768 independent subspaces of size 1, and a separate learned sigmoid gate is applied to each dimension: where is a learnable parameter vector initialized from a normal distribution, σ denotes the sigmoid activation function, and ⊙ denotes element-wise multiplication. This mechanism enables the model to selectively amplify dimensions that contain strong sentiment information while suppressing redundant or noisy dimensions, thereby providing fine-grained feature selection before the fusion stage. The proposed subspace gating mechanism is intentionally designed as a lightweight, parameter-efficient feature filtering module rather than a fully input-dependent gating network. The learnable gating vectors act as globally optimized feature selectors that suppress consistently noisy or less informative latent dimensions across multimodal representations. Compared with additional MLP-based adaptive gating mechanisms, the proposed design introduces lower computational overhead and fewer trainable parameters, thereby helping maintain training stability and reducing the risk of overfitting on relatively limited multimodal sentiment analysis datasets. Furthermore, sample-specific adaptive interaction is already handled by the stacked Tri-MCA attention layers and the dynamic expert gating module, which perform context-aware multimodal fusion and adaptive modality weighting during inference. Dynamic expert gate network To adaptively determine the contribution of each modality to the final prediction, we introduce a dynamic expert gating network that computes sample-specific modality weights. The three gated representations are concatenated to form a 2304-dimensional vector, which is then passed through a two-layer MLP with layer normalization and GELU activation to produce three logits. The modality weights are obtained through a temperature-scaled softmax function: where is a fixed temperature parameter that smooths the weight distribution and prevents the model from collapsing onto a single dominant modality. The fused representation is then computed as a weighted sum: During training, modality dropout with probability is applied to the weight vector by randomly zeroing the contribution of one modality and subsequently re-normalizing the remaining weights. This strategy encourages the model to maintain robust predictions even when one modality is absent or uninformative. Regression head and optimization objective The fused representation f is passed through a two-layer regression head to produce the final sentiment prediction. The regression head consists of a linear projection from 768 to 128 dimensions, followed by a GELU activation, dropout, a second linear projection to a scalar value, and a Tanh activation scaled by 3.0 to constrain the output to the range : For optimization, the Huber loss with is adopted as the training objective: The Huber loss combines the precision of mean squared error for small residuals with the robustness of mean absolute error for large residuals, making it well suited for sentiment regression tasks in which label noise and inter-annotator disagreement are prevalent. The model is trained end-to-end, with all parameters optimized jointly. The proposed Tri-MCA fusion framework provides an effective solution for multimodal sentiment analysis by integrating modality-specific representation learning with advanced cross-modal interaction and adaptive fusion mechanisms. Unlike traditional approaches that rely on simple feature concatenation or static fusion strategies, the proposed methodology introduces a more flexible architecture that enables modalities to interact iteratively and contribute dynamically to the final prediction. The stacked cross-modal attention layers allow each modality to refine its representation by incorporating complementary information from the other modalities, thereby facilitating deeper multimodal reasoning. In addition, the dimension-wise subspace gating mechanism performs fine-grained feature selection, enabling the model to emphasize informative sentiment cues while suppressing redundant or noisy dimensions. The dynamic expert gating network further enhances the robustness of the framework by adaptively assigning modality weights according to the characteristics of each input sample, ensuring that the most informative modality contributes more strongly to the final representation. Combined with a regression head optimized using the Huber loss, the proposed architecture forms a unified end-to-end framework capable of learning robust multimodal representations for sentiment regression. Collectively, these design choices highlight the novelty and significance of the proposed methodology and demonstrate its potential to improve the effectiveness of multimodal sentiment analysis systems by capturing complex cross-modal relationships and adapting to varying modality reliability across samples. Experimental design This section describes the experimental setup used to evaluate the proposed tri-modal cross-attention (Tri-MCA) fusion model for multimodal sentiment analysis. The experiments are designed to assess the model’s ability to learn effective multimodal representations from textual, acoustic, and visual modalities. Datasets To evaluate the effectiveness of the proposed tri-modal cross-attention fusion (Tri-MCA fusion) architecture, experiments are conducted on three widely used benchmark datasets for multimodal sentiment analysis: CMU-MOSI21, CMU-MOSEI22, and SIMS (CH-SIMS)23. These datasets provide synchronized text, audio, and visual modalities, enabling models to learn multimodal sentiment representations from language, vocal tone, and facial expressions simultaneously. The use of multiple datasets with different scales and languages allows a comprehensive evaluation of the proposed architecture across diverse multimodal settings. CMU-MOSI The CMU multimodal opinion-level sentiment intensity (CMU-MOSI) dataset was introduced by Zadeh et al.21 and is one of the earliest datasets specifically designed for multimodal sentiment analysis. The dataset consists of opinion videos collected from YouTube in which speakers express subjective opinions on various topics, including movies, products, and everyday experiences. The dataset contains 2,199 utterances extracted from 93 videos spoken by 89 different speakers. Each utterance is annotated with a sentiment intensity score ranging from − 3 (strongly negative) to + 3 (strongly positive). The dataset includes three synchronized modalities: the text modality, consisting of manually transcribed speech; the audio modality, consisting of acoustic features that capture speech prosody and tone; and the visual modality, consisting of facial expressions and head movements extracted from video frames. CMU-MOSI also provides word-level alignment across modalities, enabling models to capture fine-grained interactions between verbal and nonverbal signals. Because of its relatively small size, CMU-MOSI is commonly used to evaluate the effectiveness of multimodal fusion architectures. CMU-MOSEI The CMU multimodal opinion sentiment and emotion intensity (CMU-MOSEI) dataset was introduced by Zadeh et al.22 as a large-scale extension of CMU-MOSI. This dataset substantially increases the diversity and scale of multimodal sentiment data by incorporating videos from a broader range of speakers and topics. CMU-MOSEI contains approximately 23,500 utterances extracted from more than 1,000 YouTube videos, making it nearly ten times larger than CMU-MOSI. Each utterance is annotated using the same continuous sentiment scale ranging from − 3 to + 3, enabling direct comparison between models evaluated on MOSI and MOSEI. Similar to MOSI, each utterance includes three synchronized modalities: text, consisting of spoken transcripts; audio, consisting of acoustic speech signals; and visual, consisting of facial and gesture information extracted from video frames. Because of its larger scale and greater diversity in speakers and topics, CMU-MOSEI is widely used for training deep multimodal models and evaluating model robustness. SIMS (CH-SIMS) The SIMS dataset, also known as CH-SIMS, is a large-scale Chinese multimodal sentiment analysis dataset introduced by Yu et al.22. Unlike MOSI and MOSEI, which contain English-language videos, SIMS focuses on Chinese multimodal sentiment understanding, making it an important benchmark for evaluating the generalization capability of multimodal models across languages. The dataset contains approximately 2,281 video segments extracted from Chinese online videos. Each sample includes synchronized text, audio, and visual modalities, along with sentiment annotations. The dataset provides fine-grained sentiment labels for each modality as well as an overall sentiment label, enabling detailed analysis of modality-specific sentiment contributions. SIMS differs from MOSI and MOSEI in several respects. First, it contains Chinese-language speech, making it suitable for multilingual multimodal research. Second, it includes modality-specific annotations, allowing analysis of how sentiment is expressed differently through language, voice, and facial expressions. In addition, the dataset provides continuous sentiment labels in the range of − 1 to + 1, which differs from the − 3 to + 3 scale used in MOSI and MOSEI. These characteristics make SIMS particularly valuable for evaluating cross-lingual multimodal sentiment models and studying modality interaction patterns. Table 2 presents a comparison of the MSA datasets used in this study. Table 2Comparison of multimodal sentiment analysis datasets.DatasetYearLanguage# of Videos# of UtterancesModalitiesSentiment rangeKey characteristicsCMU-MOSI2016English932,199Text, audio, visual − 3 to + 3First benchmark dataset for multimodal sentiment analysis; widely used for evaluationCMU-MOSEI2018English > 1,000 ~ 23,500Text, audio, visual − 3 to + 3Large-scale dataset with higher speaker and topic diversitySIMS (CH-SIMS)2020Chinese– ~ 2,281Text, audio, visual − 1 to + 1Provides modality-specific sentiment annotations and supports multilingual evaluation Comparison of multimodal sentiment analysis datasets. For all datasets, modality features are extracted using pretrained transformer-based encoders. The textual modality is processed using BERT-base, the acoustic modality is encoded using Wav2Vec2-base, and the visual modality is encoded using a Vision Transformer (ViT-base). These encoders generate modality embeddings with a unified dimensionality of 768, enabling efficient multimodal interaction within the proposed architecture. Feature extraction To generate modality representations for the proposed model, pretrained transformer-based encoders are employed for each modality: BERT-base for textual representations, Wav2Vec2-base for acoustic feature extraction, and ViT-base for visual frame encoding. These encoders produce modality embeddings with a unified dimensionality of 768, enabling efficient multimodal interaction within the proposed architecture. Tables 3, 4 and 5 summarize the hyperparameters and configuration settings used for the proposed model. Table 3Encoder hyperparameters.EncoderArchitectureLayersHidden sizeAttention headsAdditional settingsTextBERT-base1276812Max sequence length 128–512Audiowav2vec2-base1276812Temporal mean poolingVideoViT-base1276812Patch size = 16 Encoder hyperparameters. Table 4Model architecture hyperparameters.ComponentHyperparameterValueCross-modal attention layersNumber of layers2Attention headsHeads per layer8Embedding dimensionHidden size768Feed-forward expansion factorFFN expansion6Feed-forward dimension–4,608Subspace feature gatingNumber of gates768Expert-level gate networkInput size2,304Hidden layer size256Output size3 modality weightsFusion moduleFusion typeWeighted sumPrediction headLinear layer 1768 → 128Linear layer 2128 → 1Activation function–GELUOutput function– Model architecture hyperparameters. Table 5Training hyperparameters.ParameterValueOptimizerAdamWLearning rate (pretrained encoders)1 × 10⁻5Learning rate (fusion module)5 × 10⁻5Weight decay1 × 10⁻4Adam β₁, β₂(0.9, 0.999)Adam ε1 × 10⁻⁸Batch size16Number of epochs50Warmup epochs2Learning rate schedulerWarmup + cosine decayDropout rate0.2Modality dropout0.2Loss functionHuber (δ = 1.0)Gradient clipping (max norm)1.0Mixed precision (AMP)EnabledGradient checkpointingEnabledNumber of video frames per sample8Audio sample rate16,000 HzRandom seed42 Training hyperparameters. Evaluation metrics To comprehensively evaluate the performance of the proposed Tri-MCA fusion architecture, multiple evaluation metrics commonly used in multimodal sentiment analysis are adopted. Because sentiment prediction in datasets such as CMU-MOSI and CMU-MOSEI is typically formulated as both a regression and a classification task, the selected evaluation metrics measure prediction error, correlation with ground-truth sentiment labels, and classification performance. Following prior studies in multimodal sentiment analysis, the evaluation metrics include mean absolute error (MAE), Pearson correlation (Corr), binary accuracy (Acc₂), seven-class accuracy (Acc₇), and F1-score. Mean absolute error (MAE) Mean absolute error measures the average absolute difference between predicted sentiment scores and ground-truth annotations. It evaluates how accurately the model predicts continuous sentiment intensity values. MAE is defined as: where represents the ground-truth sentiment score, denotes the predicted sentiment value, and N is the total number of samples. Lower MAE values indicate better regression performance. MAE is widely used in multimodal sentiment analysis studies to evaluate prediction error on continuous sentiment scales. Pearson correlation (Corr) The Pearson correlation coefficient measures the linear relationship between predicted sentiment scores and ground-truth sentiment annotations. Unlike MAE, which measures absolute prediction error, correlation evaluates how well the predicted sentiment values follow the overall trend of the ground-truth labels. It is defined as: where and denote the mean values of the ground-truth and predicted sentiment scores, respectively. A higher correlation value indicates stronger agreement between predicted and actual sentiment intensity values. Binary accuracy (Acc₂) Binary accuracy evaluates classification performance after converting continuous sentiment scores into two categories: positive and negative sentiment. Following common practice in prior research2,24,25, sentiment scores greater than zero are considered positive, whereas scores less than or equal to zero are treated as negative. Binary accuracy is computed as: where TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively. This metric measures the model’s ability to correctly distinguish between positive and negative sentiment expressions. Seven-class accuracy (Acc₇) Seven-class accuracy evaluates the model’s ability to predict fine-grained sentiment intensity categories. In this metric, continuous sentiment scores are discretized into seven classes corresponding to the original annotation scale used in the MOSI and MOSEI datasets: Predicted sentiment scores are rounded to the nearest integer class before computing classification accuracy: Acc₇ provides a more challenging evaluation than binary accuracy because the model must predict the exact sentiment intensity level rather than only the overall sentiment polarity. F1-score The F1-score measures the harmonic mean of precision and recall, providing a balanced evaluation of classification performance, particularly when class distributions are imbalanced. The F1-score is defined as: where The F1-score is frequently reported alongside binary accuracy to evaluate classification performance in multimodal sentiment analysis tasks (Table 6). Table 6Evaluation metrics used in experiments.MetricTypeDescriptionMAERegressionMeasures average absolute error between predicted and ground-truth sentiment scoresCorrRegressionPearson correlation between predicted and true sentiment valuesAcc₂ClassificationBinary accuracy distinguishing positive vs. negative sentimentAcc₇ClassificationSeven-class accuracy based on discretized sentiment intensityF1-scoreClassificationHarmonic mean of precision and recall Evaluation metrics used in experiments. Computational complexity analysis This subsection analyzes the practical computational complexity of the proposed model, including its parameter count, runtime efficiency, and memory requirements during training. All measurements were obtained using an NVIDIA T4 GPU with 16 GB of memory. The full model contains approximately 348.4 million parameters. Most of these parameters originate from the three pretrained encoders (BERT, Wav2Vec2, and ViT), which together account for 291 million parameters. The proposed fusion module introduces an additional 57.4 million parameters. As shown in Table 7, nearly all parameters within the fusion module are associated with the two cross-modal attention layers, which contain 56.7 million parameters (98.77% of the fusion module). The newly proposed components introduce only minimal computational overhead. Specifically, the dynamic expert gating network adds approximately 595 K parameters (1.04%), while the three subspace gating modules collectively contribute only 6,912 parameters (0.01%). These results indicate that the novelty of the proposed model does not primarily arise from introducing a large number of additional parameters, but rather from the way the existing representational capacity is structured, refined, and adaptively gated. Table 7Parameter breakdown of the proposed model.ComponentParameters% of Fusion moduleBERT-base (text encoder)110.0 M–Wav2Vec2-base (audio encoder)95.0 M–ViT-base/16 (video encoder)86.0 M–Input normalization (3 × LayerNorm)4.6 K0.01Cross-modal attention layers (× 2)56.69 M98.77Subspace gating (3 × 768 dimensions)6.9 K0.01Dynamic expert gate595 K1.04Fusion normalization1.5 K0.00Regression head98.6 K0.17Fusion module total57.40 M100.00Full model total348.40 M– Parameter breakdown of the proposed model. The fusion module is larger than those used in earlier methods such as MulT, MISA, and Self-MM, primarily because it operates directly on the full 768-dimensional pretrained embeddings rather than projecting them into a lower-dimensional space. This design choice preserves the representational capacity of the pretrained encoders. More importantly, the two novel components introduced in this work, namely the dynamic expert gating network and the dimension-wise subspace gating mechanism, together add only approximately 0.6 million parameters (1.05% of the fusion module). However, as demonstrated in the ablation study (Section “Ablation Study”), these components consistently improve performance. These results suggest that the proposed design achieves an effective balance between predictive accuracy and computational efficiency. As shown in Table 8, the proposed Tri-MCA fusion framework maintains a competitive balance between multimodal representation capability and computational feasibility. Earlier approaches, such as MISA, employ relatively lightweight fusion strategies without explicit cross-modal attention mechanisms, whereas transformer-based methods such as MulT26 introduce higher computational costs through directional pairwise cross-modal attention. In contrast, MM-PEAR-CoT relies on external large language model reasoning and multiple chain-of-thought generation steps, resulting in substantially greater computational complexity during inference. The proposed framework introduces joint tri-modal attention together with adaptive gating mechanisms while avoiding the heavy reasoning overhead associated with LLM-based multimodal reasoning frameworks. Although the overall parameter count is primarily influenced by the pretrained encoders, the proposed subspace gating and dynamic expert gating modules introduce only minimal additional overhead relative to the full architecture. Table 8Complexity comparison with representative baselines.ModelParametersFusion strategyCross-modal attentionDynamic gatingRelative complexityMISA7ModerateModality-invariant/specific fusionNoNoMediumMulT26Moderate–highDirectional pairwise transformer attentionYesNoHighMM-PEAR-CoT2Very high (LLM-dependent)CoT reasoning + cross-modal filtering/fusionYesNoVery highUEFN-MTFNHighMultitask multimodal fusionPartialNoHighOur method (Tri-MCA fusion)348.4 MJoint tri-modal attention + adaptive gatingYesYesHigh Complexity comparison with representative baselines. Results and discussion This section presents the experimental results of the proposed tri-modal cross-attention (Tri-MCA) fusion model on three benchmark datasets: CMU-MOSI, CMU-MOSEI, and SIMS. The performance of the proposed model is compared with several representative multimodal sentiment analysis approaches, including early fusion methods (EF-LSTM)7, late fusion models (LF-DNN), tensor-based fusion architectures (TFN and LMF)7, memory-based models (MFN and Graph-MFN)9, transformer-based multimodal architectures (MulT), modality disentanglement models (MISA)7, and recent representation learning frameworks such as Self-MM, MMIM27, and UEFN-MTFN9. The evaluation follows the metrics introduced in Section “Experimental Design”, including MAE, Corr, Acc₂, F1-score, and fine-grained Acc₇ for MOSI and MOSEI, while Acc₃ and Acc₅ are additionally reported for the SIMS dataset. Results on CMU-MOSI Table 9 presents the performance comparison on the CMU-MOSI dataset. The proposed Tri-MCA fusion model achieves strong performance across most evaluation metrics, obtaining 84.2/85.6 Acc₂, 83.65/84.53 F1-score, 0.701 MAE, 0.802 correlation, and 48.1 Acc₇. Among all compared methods, the proposed model achieves the lowest MAE and the highest correlation coefficient, indicating more accurate sentiment regression and stronger alignment with ground-truth sentiment intensity values. Table 9Performance comparison between our method and baselines on MOSI dataset.ModelAcc2F1-ScoreMAECorrAcc7EF-LSTM77.8/79.077.7/78.90.9520.65134.5LF-DNN78.0/79.377.9/79.30.9780.65833.6TFN–/80.8–/80.70.9010.69832.1LMF–/82.5–/82.40.9170.69532.8MFN77.4/–77.3/–0.9650.63234.1Graph-MFN77.9/80.277.8/80.10.9390.65634.4MulT81.5/84.180.6/83.90.8610.71140.0MISA81.8/83.481.7/83.60.7830.76142.3Self-MM82.5/83.882.5/83.90.7580.77944.9MMIM80.2/81.980.2/82.00.8110.72545.0UEFN_MTFN85.2/86.685.2/86.70.7280.79246.7Our method (Tri-MCA fusion)84.2/85.683.65/84.530.7010.80248.1 Performance comparison between our method and baselines on MOSI dataset. Compared with earlier multimodal fusion methods such as EF-LSTM, LF-DNN, TFN, and LMF, the proposed model substantially improves both regression and classification performance. These traditional approaches primarily rely on simple feature concatenation or tensor fusion, which limits their ability to capture complex interactions among modalities. The observed improvements demonstrate the effectiveness of the proposed stacked cross-modal attention mechanism, which enables each modality to dynamically attend to complementary information from the other modalities. Compared with more recent architectures such as MulT, MISA, Self-MM, and MMIM, the proposed model continues to demonstrate consistent improvements, particularly in MAE, Corr, and Acc₇. The improvement in Acc₇ (48.1) indicates that the model is more effective at predicting fine-grained sentiment intensity levels rather than merely distinguishing between positive and negative sentiment. Although the UEFN-MTFN model achieves slightly higher Acc₂ and F1-score values, the proposed approach performs better in terms of MAE, Corr, and Acc₇, which are more indicative of accurate sentiment intensity prediction. The slightly lower F1-score may be attributed to the regression-oriented design of the model, which prioritizes precise sentiment intensity estimation. The strong regression performance suggests that the proposed architecture captures subtle sentiment variations more effectively. Overall, the results on the MOSI dataset demonstrate that Tri-MCA fusion provides balanced improvements across both regression and classification metrics while achieving the strongest performance in fine-grained sentiment prediction. Results on CMU-MOSEI Table 10 presents the experimental results on the CMU-MOSEI dataset, which is substantially larger and more diverse than MOSI. The proposed model achieves 85.6/86.9 Acc₂, 85.2/86.5 F1-score, 0.518 MAE, 0.781 correlation, and 56.1 Acc₇, outperforming all baseline models across nearly all evaluation metrics. Table 10Performance comparison between our method and baselines on MOSEI dataset.ModelsAcc2F1-ScoreMAECorrAcc7EF-LSTM80.1/80.380.3/81.00.6030.68249.3LF-DNN78.6/82.379.0/82.20.5610.72352.1TFN–/82.5–/82.10.5930.70050.2LMF–/82.0–/82.10.6230.67748.0MFN76.0/–76.0/––––Graph-MFN81.9/84.082.1/83.80.5690.72551.9MulT–/82.5–/82.30.5800.70351.8MISA83.6/85.583.8/85.30.5550.75652.2Self-MM77.6/83.978.5/84.00.5400.75754.2MMIM82.1/83.882.3/83.60.5990.74051.9UEFN_MTFN84.8/86.084.9/85.80.5350.76054.5Our method (Tri-MCA fusion)85.6/86.985.2/86.50.5180.78156.1 Performance comparison between our method and baselines on MOSEI dataset. Compared with previous approaches such as MulT, MISA, and MMIM, the proposed architecture demonstrates clear improvements in both regression and classification performance. For example, the correlation score increases to 0.781, indicating that the predicted sentiment scores closely follow the trends of the ground-truth sentiment labels. In addition, the MAE value of 0.518 is the lowest among all compared models, demonstrating superior regression accuracy. The improvement over UEFN-MTFN, which previously reported strong performance on this dataset, highlights the effectiveness of the proposed fusion strategy. Specifically, the stacked cross-modal attention layers enable iterative interaction among modalities, allowing the model to capture richer multimodal dependencies. Furthermore, the dimension-wise subspace gating mechanism enables the model to filter irrelevant features at a finer granularity, thereby improving representation quality before fusion. The dynamic expert gating network also contributes to the improved results by assigning sample-specific modality weights. This adaptive mechanism allows the model to emphasize the most informative modality according to the input context, which is particularly important in large-scale datasets such as MOSEI, where the quality and informativeness of each modality may vary across samples. The strong performance on MOSEI demonstrates that the proposed architecture scales effectively to larger multimodal datasets while maintaining robust performance across diverse speakers and topics. Results on SIMS Table 11 presents the performance comparison on the SIMS dataset, which introduces additional challenges because of its Chinese-language content and modality-specific sentiment annotations. The proposed Tri-MCA fusion model achieves 82.34 Acc₂, 82.02 F1-score, 0.417 MAE, 0.598 correlation, 69.05 Acc₃, and 47.11 Acc₅, outperforming all baseline methods across most evaluation metrics. Table 11Performance comparison between our method and baselines on SIMS dataset.ModelsAcc2F1-ScoreMAECorrAcc3Acc5EF-LSTM69.3756.820.5990.52154.2721.23LF-DNN78.9979.720.4190.58964.9941.36TFN80.3180.660.4510.58164.3336.76LMF78.9978.990.4420.57466.7437.86MFN79.2179.150.4340.58166.0839.39Graph-MFN79.6580.400.4770.58167.4039.17MulT79.6579.940.4390.58265.8638.95MISA69.3756.820.5870.11351.4220.79Self-MM78.7778.880.4250.59163.6839.82MMIM73.9674.590.4410.53160.6143.54UEFN_MTFN81.5681.270.4230.58367.7745.20Our method (Tri-MCA fusion)82.3482.020.4170.59869.0547.11 Performance comparison between our method and baselines on SIMS dataset. In particular, the proposed method achieves the highest Acc₂, F1-score, Acc₃, and Acc₅, indicating improved classification performance for both binary and multiclass sentiment prediction. The MAE value of 0.417 is also lower than that of most baseline methods, suggesting superior regression accuracy. Compared with strong baselines such as Graph-MFN, MulT, and Self-MM, the proposed model consistently achieves higher classification accuracy and stronger correlation values. These improvements demonstrate that the proposed architecture generalizes effectively across languages and datasets with different annotation schemes. The performance improvements on SIMS further highlight the effectiveness of combining cross-modal attention with adaptive modality weighting. Because sentiment expressions in Chinese multimodal data may vary substantially across modalities, the dynamic expert gating mechanism enables the model to adjust modality importance according to the input context. In addition, the subspace gating module helps suppress noisy features that may arise from modality-specific variations. All baseline results reported in Tables 8, 9 and 10 were obtained from the corresponding original publications or publicly available benchmark results. To ensure a fair comparison, the proposed framework follows the standard benchmark dataset splits and commonly adopted evaluation protocols used in prior multimodal sentiment analysis studies on the CMU-MOSI, CMU-MOSEI, and SIMS datasets. The results across MOSI, MOSEI, and SIMS consistently demonstrate the effectiveness of the proposed Tri-MCA fusion architecture. By integrating stacked cross-modal attention, dimension-wise feature gating, and adaptive expert-level modality weighting, the proposed model captures richer multimodal interactions and produces more accurate sentiment predictions across diverse datasets and languages. Ablation study To better understand the contribution of each architectural component, an ablation study is conducted by progressively removing or modifying components of the proposed model. The experiments are performed on the CMU-MOSI dataset using the same training setup and hyperparameters as the full model to ensure a fair comparison. Several model variants are evaluated to analyze the contribution of each component. The Baseline Fusion model uses simple concatenation of modality features without cross-attention or gating mechanisms. The second variant introduces tri-modal cross-attention (Tri-MCA), enabling interactions among text, audio, and visual modalities to capture inter-modal dependencies. Next, subspace gating (SG) is incorporated to perform feature-level filtering by suppressing noisy or irrelevant features within modality representations. The expert gating (EG) module is then added to dynamically adjust the importance of each modality through learned weights. Finally, the full model integrates Tri-MCA, subspace gating, and expert gating into a unified architecture. Model performance is evaluated using Acc2, F1-score, MAE, and correlation (Corr). The ablation results presented in Table 12 reveal several important findings. First, incorporating tri-modal cross-attention significantly improves performance compared with simple feature concatenation. This improvement indicates that explicitly modeling interactions among modalities enables the network to capture complementary sentiment cues across textual, acoustic, and visual signals. Second, the subspace gating mechanism further enhances performance by filtering irrelevant or noisy features within modality representations. This component enables the model to focus on sentiment-relevant subspaces while reducing the influence of noisy acoustic or visual signals. Third, adding the expert gating module leads to additional performance gains. This mechanism dynamically determines the relative contribution of each modality, allowing the model to adapt to situations in which certain modalities are more informative than others. Finally, integrating all components results in the strongest overall performance. The full model improves accuracy by approximately 5.5% over the baseline, demonstrating that cross-modal reasoning and adaptive gating jointly contribute to more effective multimodal representations (Fig. 2). Table 12Ablation study on the MOSI dataset.Model variantAcc2F1-ScoreMAECorrBaseline Fusion80.179.80.8420.721 + Tri-MCA82.782.40.8040.746 + Tri-MCA + SG83.983.50.7810.763 + Tri-MCA + EG84.083.80.7330.786Our method (Tri-MCA fusion) (Tri-MCA + SG + EG)85.684.530.7010.802 Ablation study on the MOSI dataset. Ablation study results illustrating the impact of different architectural components on sentiment prediction accuracy (Acc2). The results demonstrate consistent performance improvements as Tri-MCA, subspace gating (SG), and expert gating (EG) are progressively integrated into the model, with the full model achieving the highest accuracy. Table 13 shows that the text modality remains the strongest individual source of sentiment information, while the audio and visual modalities provide complementary emotional cues that further improve prediction accuracy. Table 13Contribution of each modality.Modality settingAcc2F1-ScoreText only82.382.0Text + audio83.783.2Text + visual84.183.6Text + audio + visual (full)85.684.53 Contribution of each modality. To investigate the influence of the expert gating temperature parameter τ, a sensitivity analysis was conducted under different temperature settings while keeping all other experimental configurations unchanged. The temperature parameter controls the smoothness of the modality weight distribution in the dynamic expert gating network. Smaller values produce sharper modality preferences, whereas larger values generate smoother and more uniform modality weighting. Table 14 shows that the proposed framework maintains relatively stable performance across different temperature settings, with τ =2.0 providing the best overall balance between modality specialization and stable multimodal fusion (Table 14). Table 14Sensitivity analysis of the expert gating temperature parameter τ.Temperature τAcc₂F1-ScoreMAECorr1.084.783.90.7180.7921.585.184.20.7090.7982.085.684.530.7010.8022.585.284.30.7060.7993.084.984.00.7140.794 Sensitivity analysis of the expert gating temperature parameter τ. Conclusion This paper presented a multimodal sentiment analysis framework based on tri-modal cross-attention and adaptive gating mechanisms for the effective integration of textual, acoustic, and visual information. The proposed architecture enables rich cross-modal interactions through the Tri-MCA module while preserving modality-specific characteristics. The subspace gating mechanism enhances representation quality by filtering irrelevant features, whereas the expert gating module dynamically adjusts modality contributions to address modality imbalance and variations in signal reliability. Experimental results on benchmark datasets demonstrate that the proposed model achieves competitive performance across multiple evaluation metrics. In addition, the ablation studies confirm that each architectural component contributes positively to the overall performance, highlighting the effectiveness of the proposed design. Future work will focus on developing a more advanced framework that incorporates large language models (LLMs) as reasoning components within the multimodal architecture rather than relying solely on pretrained base encoders. Such a framework could leverage the reasoning and contextual understanding capabilities of LLMs to improve cross-modal sentiment interpretation. In addition, future research will explore applying this approach to low-resource settings in which multimodal datasets are limited by integrating transfer learning, instruction tuning, and parameter-efficient adaptation techniques. This direction may enable more robust multimodal sentiment analysis for underrepresented languages and domains with limited labeled data. Acknowledgements We would like to thank the anonymous reviewers for their valuable suggestions. The authors are thankful to the Deanship of Graduate Studies and Scientific Research at Najran University for funding thiswork under the Growth Funding Program grant code NU/GP/SERC/14/888-1. Author contributions A.A. conceived the study, designed the methodology, conducted the experiments, analyzed the results, and wrote the manuscript. A.A. reviewed and approved the final version of the manuscript. Data availability The datasets used during the current study are publicly available. Specifically, the CMU-MOSI, CMU-MOSEI, and SIMS datasets can be accessed from their official repositories: https://www.kaggle.com/datasets/mathurinache/cmu-mosi, https://www.kaggle.com/datasets/samarwarsi/cmu-mosei, and https://github.com/thuiar/ch-sims-v2 (accessed on 10 December 2025). No new datasets were generated. All experimental settings are described in the manuscript to ensure reproducibility. Declarations Competing interests The authors declare that they have no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used during the current study are publicly available. Specifically, the CMU-MOSI, CMU-MOSEI, and SIMS datasets can be accessed from their official repositories: https://www.kaggle.com/datasets/mathurinache/cmu-mosi, https://www.kaggle.com/datasets/samarwarsi/cmu-mosei, and https://github.com/thuiar/ch-sims-v2 (accessed on 10 December 2025). No new datasets were generated. All experimental settings are described in the manuscript to ensure reproducibility.