Sequential translation-based multimodal sentiment analysis under uncertain missing modalities Authors: Yan Hai, Shanqi Lu, Zhizhong Liu, Ling Shang, Hongxiang Sun, Jing Wang Published: 2026 ABSTRACT Multimodal Sentiment Analysis (MSA) aims to fuse information from multiple modalities to achieve precise sentiment classification. Recently, the issue of uncertain missing modalities has become one of the new challenges in MSA. Previous studies have attempted to solve this issue by building information interactions on modality pairs consisting of two modalities. However, existing methods typically rely on interactions between paired modalities to compensate for missing information. Such representations struggle to accurately reconstruct true cross-modal semantics due to the absence of guidance from a third modality. Additionally, existing approaches have neglected the effective utilization of text modality and the complexity of the models is relatively high. To tackle the above issues, we propose a sequential translation-based MSA model (STMSA). This model incorporates two key designs. First, the text-centric bidirectional translation mechanism leverages the dominant role of the text modality in affective tasks to sequentially establish bidirectional mappings with the audio and video modalities. This mechanism fully explores the deep connections among the three modalities through semantic guidance from text, enabling cross-modal representations that more closely align with real affective distributions. Second, the low-complexity non-modal completion architecture performs distributed fitting on joint representations in a shared space using only an encoder-decoder, thereby avoiding complex missing-modality generation processes. Extensive experiments were conducted on two public datasets, CMU-MOSI and IEMOCAP, demonstrating that the proposed model outperforms 10 state-of-the-art baseline models. Introduction With the rapid advancement of the Internet, numerous prominent social media platforms have emerged, such as Weibo, Twitter (now X), and YouTube. Users utilize these platforms to share images, videos, and textual comments, expressing their personal viewpoints. These processes produce large amounts of multimodal information, encompassing visual (such as images), auditory (such as speech and audio), and textual information1. Compared to single-modality data, multimodal data conveys richer emotional information by integrating complementary features across different modalities, offering significant advantages for sentiment analysis2. To effectively leverage these advantages for more accurate identification of complex emotional states, Multimodal Sentiment Analysis (MSA) has been proposed and has garnered widespread attention from researchers. MSA refers to the comprehensive analysis of emotional states expressed in videos by integrating information from multiple modalities3. Currently, MSA has broad application in areas such as opinion mining, human-computer interactions, product monitoring, and marketing strategy enhancement. With the advancement of deep learning, extensive research has employed neural networks for MSA. For example, text-enhanced transformer4 fusion networks have been utilized to learn paired cross-modal mappings5, and Wang et al.6 proposed a multi-attention method for extracting common multimodal representations. Zhu et al.7 introduced a Sentiment Knowledge Enhanced Attention Fusion Network (SKEAFN), which strengthens multimodal fusion by incorporating supplementary representations from an external knowledge module. These studies demonstrate that neural networks provide effective technical support and play a significant role in the advancement of MSA. However, many studies are typically conducted under the assumption that all multimodal data is always available8–10. This assumption differs significantly from real-world application scenarios. In practical environments, the data collection process is highly susceptible to interference from various uncontrollable factors, making the uncertainty or loss of modal information the norm rather than the exception, as shown in Fig. 1. When a camera is obstructed, visual information cannot be obtained, resulting in the loss of the visual modality in the sample; when ambient noise is too loud, actual auditory information is drowned out, or when audio equipment malfunctions, the audio modality is lost; in clinical psychological assessments, patients may be unable to provide complete verbal feedback due to physical discomfort; in social media analysis, user-uploaded content may inherently lack a certain modality. This modality loss is not merely random noise; it fundamentally alters the structure and semantic integrity of the data. When critical modalities are missing, the performance of traditional multimodal models–which rely on complete modal inputs–will plummet or even fail entirely. Therefore, ensuring that multimodal sentiment analysis models possess robustness in the face of uncertain modality loss has become a core hurdle that must be overcome to advance this technology from the laboratory to real-world applications.Fig. 1The uncertain missing modalities problem in MSA. The uncertain missing modalities problem in MSA. In recent research, several studies have addressed the issue of missing modalities. For example, Liu et al.11 proposed a modality-translation-based MSA model that uses the transformer4 to translate visual and auditory modalities into text, generating missing joint features based on the translation output to enhance the quality of feature fusion. The success of this bimodal mapping relies heavily on the simultaneous presence of both the source and target modalities. However, in scenarios with uncertain missing modalities, if the visual modality is missing, the visual-to-text mapping pathway is disrupted, resulting in the model’s lack of adaptability when faced with various missing modality patterns. Furthermore, this pairwise mapping cannot leverage complementary information from a third available modality (such as audio) to compensate for the missing modality. Wang et al.12 developed an end-to-end translation model (TransM) that utilizes transformer-based translation information between the source and target modalities as inter-modal representations to facilitate inter-modal interaction. However, this method failed to design a text-centric interaction mechanism, resulting in visual and audio features lacking guidance and calibration from high-quality semantic information. Sun et al.13 constructed a pool of similar modal samples to fill in missing modalities through a completion strategy. However, this completion strategy limits the model’s complexity and scalability to the number of predefined missing modalities. As the number of modalities increases or the combinations of missing modalities diversify, the model structure expands dramatically, making it difficult to deploy and maintain in practical applications. The shortcomings of the above methods can be summarized as follows:The robustness of existing models that based on modality translation is weak. Existing approaches primarily capture cross-modal mappings by translating between two modalities. However, when one modality in a pair is missing, the mapping between these two modalities becomes invalid. Additionally, methods based on paired modal translation cannot leverage complementary information from a third available modality (such as the audio modality) to compensate for the missing modality, resulting in a lack of adaptability when the model encounters various missing modality scenarios.Existing models failed to make full use of the textual modality in multimodal fusion. Existing research has failed to adequately account for the disparity in information across modalities. Specifically, textual modalities exhibit far greater semantic richness and stability compared to visual and audio modalities, whose emotional features remain relatively ambiguous. This oversight results in models lacking a text-centric cross-modal interaction mechanism, hindering their ability to leverage the dominance of text to guide semantic alignment of visual and audio features. Consequently, achieving deep complementarity among the three modalities remains unattainable.The complexity of existing MSA models under the uncertain missing mode is relatively high. For the problem of MSA under uncertain missing modalities, existing works attempt to develop strategies for completing missing modalities, which significantly increases models’ complexity. Furthermore, completion strategies typically require the complete modality features to participate in the decision-making process. When modalities are missing, the number of high-quality complete modalities is uncertain. However, existing works can not handle missing modalities without requiring specific completion strategies.To address these issues, the motivation of this study is to design a text-centric sequential translation model that uses a low-complexity unified architecture to adaptively handle uncertain modality gaps, thereby enhancing the robustness and generalization capabilities of multimodal sentiment analysis. For this purpose, we propose a sequential translation-based multimodal sentiment analysis model (STMSA). First, the initial feature module extracts self-attention-encoded features from each modality to form the basis of cross-modal interactions. Next, the sequential translation module uses an end-to-end translation architecture to establish mapping relationships between the three modalities. Text is the dominant modality in these relationships, and the sequential translation process is implemented according to a bidirectional translation strategy. Next, the common space module fuses the translated modality features and the text features encoded by the transformer into missing joint features (MJFs), while also fusing the self-attention-encoded features of the three modalities encoded by the transformer into initial joint features (IJFs). Supervised by the pre-trained task, the transformer encoder-decoder architecture learns the joint representation of the MJFs and IJFs. This enables it to fit the joint distribution of all modalities. Finally, the output of the joint representation is used for the final sentiment analysis task. The main contributions of our work are as follows:To improve the robustness of MSA model considering uncertain missing modalities, we propose a sequential translation method. First, one modality is translated into a second modality. Then, the resulting modality is translated into a third modality. This approach can alleviate dependency problems that arise when only two modalities are involved in the translation process. Furthermore, this chained translation process incorporates the complementary features of all three modalities. Consequently, regardless of which modality is missing, the final translation output retains the informational characteristics of the available modalities.To fully utilize the features of the textual modality, we propose a bidirectional translation strategy. On the one hand, the forward translation process prioritizes translating the visual modality to the textual modality and then to the audio modality. On the other hand, the backward translation process prioritizes translating the audio modality to the textual modality and then to the visual modality. This strategy allow for direct interactions between the textual modality and the other two modalities, while also embedding textual modality information into the features of the other two modalities to enhance the accuracy of sentiment analysis.To develop a low-complexity MSA model that does not require specific completion strategies, we design an end-to-end translation architecture. This architecture does not rely on complete modality features for decision-making. Instead, it allows multiple modality data sources as input at one end, enabling the model to directly extract and fuse information from the input features. This approach achieves unified handling of different missing modalities scenarios. Under the supervision of a pre-trained task, we conduct extensive experiments on two public datasets: CMU-MOSI and IEMOCAP. In addition, we thoroughly analyzed the existing experimental results. These results have demonstrated that our model outperforms ten baseline models. The robustness of existing models that based on modality translation is weak. Existing approaches primarily capture cross-modal mappings by translating between two modalities. However, when one modality in a pair is missing, the mapping between these two modalities becomes invalid. Additionally, methods based on paired modal translation cannot leverage complementary information from a third available modality (such as the audio modality) to compensate for the missing modality, resulting in a lack of adaptability when the model encounters various missing modality scenarios. The robustness of existing models that based on modality translation is weak. Existing approaches primarily capture cross-modal mappings by translating between two modalities. However, when one modality in a pair is missing, the mapping between these two modalities becomes invalid. Additionally, methods based on paired modal translation cannot leverage complementary information from a third available modality (such as the audio modality) to compensate for the missing modality, resulting in a lack of adaptability when the model encounters various missing modality scenarios. Existing models failed to make full use of the textual modality in multimodal fusion. Existing research has failed to adequately account for the disparity in information across modalities. Specifically, textual modalities exhibit far greater semantic richness and stability compared to visual and audio modalities, whose emotional features remain relatively ambiguous. This oversight results in models lacking a text-centric cross-modal interaction mechanism, hindering their ability to leverage the dominance of text to guide semantic alignment of visual and audio features. Consequently, achieving deep complementarity among the three modalities remains unattainable. Existing models failed to make full use of the textual modality in multimodal fusion. Existing research has failed to adequately account for the disparity in information across modalities. Specifically, textual modalities exhibit far greater semantic richness and stability compared to visual and audio modalities, whose emotional features remain relatively ambiguous. This oversight results in models lacking a text-centric cross-modal interaction mechanism, hindering their ability to leverage the dominance of text to guide semantic alignment of visual and audio features. Consequently, achieving deep complementarity among the three modalities remains unattainable. The complexity of existing MSA models under the uncertain missing mode is relatively high. For the problem of MSA under uncertain missing modalities, existing works attempt to develop strategies for completing missing modalities, which significantly increases models’ complexity. Furthermore, completion strategies typically require the complete modality features to participate in the decision-making process. When modalities are missing, the number of high-quality complete modalities is uncertain. However, existing works can not handle missing modalities without requiring specific completion strategies. The complexity of existing MSA models under the uncertain missing mode is relatively high. For the problem of MSA under uncertain missing modalities, existing works attempt to develop strategies for completing missing modalities, which significantly increases models’ complexity. Furthermore, completion strategies typically require the complete modality features to participate in the decision-making process. When modalities are missing, the number of high-quality complete modalities is uncertain. However, existing works can not handle missing modalities without requiring specific completion strategies. To improve the robustness of MSA model considering uncertain missing modalities, we propose a sequential translation method. First, one modality is translated into a second modality. Then, the resulting modality is translated into a third modality. This approach can alleviate dependency problems that arise when only two modalities are involved in the translation process. Furthermore, this chained translation process incorporates the complementary features of all three modalities. Consequently, regardless of which modality is missing, the final translation output retains the informational characteristics of the available modalities. To improve the robustness of MSA model considering uncertain missing modalities, we propose a sequential translation method. First, one modality is translated into a second modality. Then, the resulting modality is translated into a third modality. This approach can alleviate dependency problems that arise when only two modalities are involved in the translation process. Furthermore, this chained translation process incorporates the complementary features of all three modalities. Consequently, regardless of which modality is missing, the final translation output retains the informational characteristics of the available modalities. To fully utilize the features of the textual modality, we propose a bidirectional translation strategy. On the one hand, the forward translation process prioritizes translating the visual modality to the textual modality and then to the audio modality. On the other hand, the backward translation process prioritizes translating the audio modality to the textual modality and then to the visual modality. This strategy allow for direct interactions between the textual modality and the other two modalities, while also embedding textual modality information into the features of the other two modalities to enhance the accuracy of sentiment analysis. To fully utilize the features of the textual modality, we propose a bidirectional translation strategy. On the one hand, the forward translation process prioritizes translating the visual modality to the textual modality and then to the audio modality. On the other hand, the backward translation process prioritizes translating the audio modality to the textual modality and then to the visual modality. This strategy allow for direct interactions between the textual modality and the other two modalities, while also embedding textual modality information into the features of the other two modalities to enhance the accuracy of sentiment analysis. To develop a low-complexity MSA model that does not require specific completion strategies, we design an end-to-end translation architecture. This architecture does not rely on complete modality features for decision-making. Instead, it allows multiple modality data sources as input at one end, enabling the model to directly extract and fuse information from the input features. This approach achieves unified handling of different missing modalities scenarios. Under the supervision of a pre-trained task, we conduct extensive experiments on two public datasets: CMU-MOSI and IEMOCAP. In addition, we thoroughly analyzed the existing experimental results. These results have demonstrated that our model outperforms ten baseline models. To develop a low-complexity MSA model that does not require specific completion strategies, we design an end-to-end translation architecture. This architecture does not rely on complete modality features for decision-making. Instead, it allows multiple modality data sources as input at one end, enabling the model to directly extract and fuse information from the input features. This approach achieves unified handling of different missing modalities scenarios. Under the supervision of a pre-trained task, we conduct extensive experiments on two public datasets: CMU-MOSI and IEMOCAP. In addition, we thoroughly analyzed the existing experimental results. These results have demonstrated that our model outperforms ten baseline models. Related work Research on multimodal sentiment analysis With data growing continuously in scale and diversifying in form, multimodal sentiment analysis has emerged as a new trend in the field. Current technologies can mine rich emotional information from different modality data to perform sentiment analysis tasks14–16. For example, Yang et al.17 utilized a multi-task learning framework to capture intra-modal representations and balance the contribution of specific modality information through cross-modal interactions. Zhu et al.18 proposed an interactive image-text network (ITIN) that can explore the relationship between emotional image regions and text. Han et al.19 learned modality representations from two modality pairs and fused them for emotion classification tasks. Fan et al.20 proposed multimodal feature enhancement networks that perform intermodal learning on features extracted from visual, audio, and remote photoplethysmographic (rPPG) signals using a transformer architecture. Then, they designed graph fusion networks to obtain multimodal fusion representations. In addition, the incremental cross-modal interaction module designed by Zhang et al.21 proposed a new approach to progressively aligning and fusing text, audio, and video features, as well as mitigating semantic misalignment across modalities. Multimodal data fusion is the process of combining the representations of each modality into a coherent multimodal representation using fusion strategies22. These strategies can be divided into three categories: early, late, and hybrid fusion. Early fusion involves merging the latent features of each modality in the sample into a common representation. This representation is then input into a classification module for processing. For example, Sun et al.23 utilized a gated cross-modal attention mechanism to learn emotional information from modality pairs. Late fusion integrates the confidence scores of each modality in the classification task to make a comprehensive decision. Zheng et al.24 designed a three-modal feature extraction scheme and employed decision fusion to perform sentiment classification. Hybrid fusion effectively combines the processes of early and late fusion. Mai et al.25 proposed a three-modal representation model based on hybrid contrastive learning to mine cross-modal interactions, preserve interclass relationships, and narrow modal gaps. Additionally, some researchers have explored multi-stage fusion for MSA. For instance, Peng et al.26 employed multi-granularity cross-modal information for multi-stage training, applying multi-task learning to complex modality combinations in order to integrate information. Similarly, Liang et al.27 broke the fusion problem down into stages that were specialized and effective. They combined this approach with recurrent neural networks to model interactions across temporal and modality domains. Zedah et al.28 proposed a tensor fusion network to minimize information redundancy and enhance model performance. Chen et al.29 proposed a multi-level, multi-modal, dynamic fusion network that explores correlations among unimodal, bimodal, and trimodal data and leverages joint cross-modal correlation representations. These studies have significantly enhanced MSA performance by leveraging multimodal data. However, these models suffer from severe performance degradation in the presence of missing modalities due to the lack of robust handling mechanisms. Therefore, MSA models that can handle missing modalities are more effective in complex, dynamic, and often unpredictable real-world multimodal scenarios. MSA with missing modalities There are currently three main categories of approaches to the problem of missing modality: generative methods, joint representation methods, and modality translation methods. Generative methods: The aim is to generate new data that matches the existing distribution in order to compensate for missing modalities. Tran et al.30 proposed a cascaded residual auto-encoder that uses correlations between different modalities to estimate missing data and simulate the gap between predicted and original data. Zhao et al.31 proposed a Missing Modality Imagination Network (MMIN) to learn robust joint multimodal representations. The MMIN aims to predict missing modalities based on available modalities. Zuo et al.32 proposed an Inertial Feature-based MMIN, which uses a CRA and a recurrent consistency learning-based imagination module to generate representations of missing modalities given corresponding available modality representations. Shang et al.33 used generative adversarial networks (GANs) to reconstruct missing views based on paired data across views. Zhang et al.34 proposed a semantic compensation strategy and employed dynamic parameterization to achieve adaptive fusion; this method mitigates the modal-semantic gap and enhances the robustness of emotion recognition. Joint learning methods: The aim is to learn the latent features of existing modalities for feature fusion. Zeng et al.35 proposed a tag-assisted multimodal sentiment analysis model that uses tags to determine the missing modality and obtain joint representations in a common space for sentiment classification. Li et al.36 used cross-modal attention mechanisms to extract features within and between modalities for missing modality reconstruction. They also designed a fine-grained similarity constraint based on distribution-level, modality-invariant representations to learn effective, modality-invariant representations. Pham et al.37 proposed a Seq2Seq model that aims to obtain multimodal representations of data through unsupervised learning. Le et al.38 performed contrastive learning at the modality-specific representation level, introducing cross-modal alignment to reduce zero-filling noise in missing modalities and enhance intermodal feature consistency. Chi et al.39 introduced a novel sampling strategy specifically designed for missing modalities, which serves as an effective auxiliary training component for existing models. Modality translation methods: The aim is to convert one modality into another by learning the features of the new modality. Pham et al.40 proposed the Modality Cycle Translation Network (MCTN), which translates one modality into another through a recurrent propagation process (forward and backward propagation), learning joint features across modalities to guide sentiment classification. Zhang et al.41 utilized a multi-stage graph fusion network (MSGFN) to explore effective fusion representations between different modality pairs. Lu et al.42 utilized emotion-interactive convolutions to model syntactic dependencies and emotional associations within their framework. The above works attempt to solve the missing modality problem using various methods. Among these, generative methods offer the core advantage of completing missing information from a data distribution perspective. However, their output quality heavily relies on the completeness of training data, and the generation process may introduce noise, leading to degraded performance in downstream tasks. In contrast, joint learning approaches circumvent explicit generation by learning modality-invariant features, yet they face information loss when modalities are entirely absent. Modal translation methods attempt to transfer information through cross-modal mapping, yet existing translation approaches are largely confined to bimodal interactions, struggling to fully leverage the complementary features of trimodal data. Additionally, existing models’ completion strategies need to consider complex missing environments, resulting in an absence of a unified MSA model for dealing with missing modalities. Methodology Problem description Assume that the multimodal data consist of three modalities: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P{=}[X_v,X_t,X_a]$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{v}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{t}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{a}$$\end{document} represent the visual, text, and auditory modalities, respectively. Without loss of generality, we use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{x}^{m}$$\end{document} to denote a missing modality, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x\in \{v,t,a\}$$\end{document}. For example, when the visual modality is missing, the multimodal data can be represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{X_{v}^{m},X_{t},X_{a}\}$$\end{document}. In addition, to enable the public space module to dynamically integrate multimodal data, six weight matrices were established: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{va}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vt}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{ta}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vta}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{atv}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{m}$$\end{document}. The research problem addressed in this paper is sentiment analysis based on multimodal data with uncertain missing modalities. For convenience, we use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{X_{v}^{m},X_{t},X_{a}\}$$\end{document} to represent such multimodal data in the following sections. Overview To address the issue of missing modalities in MSA, we propose a sequential translation-based MSA model (STMSA), the overall structure of which is depicted in Fig. 2. STMSA runs as follows: First, the initial feature module (The green area in Fig. 2) receives multimodal data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{X_{v}^{m},X_{t},X_{a}\}$$\end{document} with missing modalities and produces self-attention-encoded features for each modality. Next, the sequential translation module (STM, the blue area in Fig. 2) performs the following bidirectional translation tasks: (1) forward translation process: translate the visual modality to the text modality, then to the audio modality; (2) backward translation process: translate the audio modality to the text modality, then to the visual modality. Next, the common space module (The orange area in Fig. 2) fuses the STM outputs and self-attention-encoded features of each modality to generate missing joint features (MJFs) and initial joint features (IJFs) separately. Then, the transformer encoder-decoder architecture learns the joint representation of MJFs and IJFs under the supervision of the pre-trained task, enabling it to fit the joint distribution of the complete modalities. Finally, the output of the joint representation is passed through a softmax function to perform the final sentiment analysis task. Initial feature module As shown in the green area in Fig. 2, first, samples with uncertain missing modalities are fed into a fully connected network to obtain multimodal data, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{X_{v}^{m},X_{t},X_{a}\}$$\end{document}, with the same dimensions. Then, each modality is input into a transformer encoder for self-attention-encoded to extract its contextual features, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{E_v', E_t', E_a'\}$$\end{document}. The calculation process is shown in Eqs. (1–3).1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_v' = \mathrm{MultiHead}(X_v^m, X_v^m, X_v^m)\end{aligned}$$\end{document}2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_t' = \mathrm{MultiHead}(X_t, X_t, X_t)\end{aligned}$$\end{document}3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_a' = \mathrm{MultiHead}(X_a, X_a, X_a)\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_v' \in \mathbb{R}^{l_v \times d}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_t' \in \mathbb{R}^{l_t \times d}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_a' \in \mathbb{R}^{l_a \times d}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_v^m\in \mathbb {R}^{l_v\times d}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{a}\in \mathbb {R}^{l_{a}\times d}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{t}\in \mathbb {R}^{l_{t}\times d}$$\end{document}. Because we need to extract information from each modality and the transformer applies the multi-head attention mechanism, the values of Q, K, and V are identical. Next, the features extracted from each modality are connected via a residual connection and fed into a layernorm layer for normalization. This process is is shown in Eqs. (4–6).4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E^{\prime \prime }{}_v=\textrm{LayerNorm}(X_v^m+E^{\prime }{}_v)\end{aligned}$$\end{document}5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_t^{\prime \prime }=\textrm{LayerNorm}(X_t+E_t^{\prime })\end{aligned}$$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_a^{\prime \prime }=\textrm{LayerNorm}(X_a+E_a^{\prime })\end{aligned}$$\end{document}Then, the normalized single-modality features are input into a fully connected feedforward layer for linear transformation. In this way, the encoding of the three single-modality data is completed. The calculation process is shown in Eqs. (7–9).7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_v=\textrm{Relu}(E^{\prime \prime }{}_vW_{vl}^1+b_{vl}^1)W_{vl}^2+b_{vl}^2\end{aligned}$$\end{document}8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_t=\textrm{Relu}(E^{\prime \prime }{}_tW_{tl}^1+b_{tl}^1)W_{tl}^2+b_{tl}^2\end{aligned}$$\end{document}9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_a=\textrm{Relu}(E^{\prime \prime }{}_aW_{al}^1+b_{al}^1)W_{al}^2+b_{al}^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vl}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{tl}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{al}$$\end{document} are weight matrices; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{vl}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{tl}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{al}$$\end{document} are bias values; and ReLU is the activation function.Fig. 2The structure of STMSA. The structure of STMSA. Sequential translation module To address the issue of uncertain modality missing, we propose a sequential translation approach. When a modality is missing, its encoded features lose genuine affective information, making it difficult for translations based solely on paired modalities to effectively capture complete cross-modal complementary features. To mitigate this, we extend the translation results to a third modality and design a bidirectional translation architecture incorporating both forward and reverse translation to reduce the impact of modality missing on affect classification tasks. The following sections will first discuss the rationale for determining the translation sequence among the three modalities, then elaborate on the design objectives of the bidirectional translation structure, and finally detail the specific translation process. The issue of sequential translation order The determination of translation order did not employ a random selection strategy or a priority-based approach based on modality quality. Instead, we established it based on the emotional expression characteristics of the three modalities. According to research by Liu et al.11, text modalities typically exhibit superior emotional features, followed by audio modalities and then visual modalities. Therefore, we designated text modalities as the stable carriers of emotional semantics and designed the translation order around them. In our translation sequence, we avoided direct translation between non-text modalities (e.g., from visual to audio). This is because non-text modalities exhibit weaker emotional characteristics, and their feature distributions deviate from those of text modalities. Direct translation could exacerbate this deviation, leading to semantic distortion between visual and audio modalities and consequently limiting the effectiveness of translations between non-text modalities. This was confirmed in subsequent experimental sections. Therefore, we selected the text modality as the intermediate node in the chained translation process. This ensures that high-quality features from the text modality are reliably propagated throughout the translation chain, while both the visual and audio modalities can establish cross-modal mapping relationships with the text modality. Bidirectional translation structure The core objective of designing our forward and reverse translation structures is to address mapping failures caused by missing modalities in traditional bimodal translation. By utilizing the text modality as an intermediate nodal point, the forward translation process enables directed transmission from the visual modality to the audio modality, while the reverse translation process facilitates back-calibration from the audio modality to the visual modality. Together, they achieve comprehensive coverage of tri-modal information. If the bidirectional complementary design is removed (e.g., retaining only a single forward translation process), the absence of the initial modality (such as the visual modality) would break the translation chain, making it impossible to reconstruct the missing information through other modalities. Therefore, this bidirectionally complementary translation structure is a critical design that balances information transmission integrity, semantic accuracy, and robustness to missing modalities. Translation process As shown in the blue area of Fig. 2, we utilize the encoded features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_v, E_t, E_a}$$\end{document} from three modalities to perform two sequential translation tasks. The following discussion will focus on the forward translation process (from visual modality to text modality to auditory modality). The translation process is illustrated in Fig. 3. The first step in sequential translation is to map visual modalities to textual modalities to obtain cross-modal features. We utilize the decoder to model the correlations between the two modalities, ensuring that the generated cross-modal features incorporate the most relevant information from the target modality. Specifically, for the encoded features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_v, E_t, E_a}$$\end{document} from the three modalities, we use the encoded visual feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_v$$\end{document} as both the Key and Value in the multi-head attention mechanism, while the encoded textual feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_t$$\end{document} serves as the Query. Next, apply a residual connection to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}^{\prime }$$\end{document} and input it into the layernorm layer for normalization. Then, input the result into the feedforward fully connected layer for linear transformation to obtain the transformer decoder output, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document}. The calculation process is shown in Eqs.(10–12).10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vt}^{\prime }=\textrm{MultiHead}(E_t,E_v,E_v)\end{aligned}$$\end{document}11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vt}^{\prime \prime }=\textrm{LayerNorm}(E_t+D_{vt}^{\prime })\end{aligned}$$\end{document}12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vt}=\textrm{Relu}(D_{vt}^{\prime \prime }W_{vtl}^1+b_{vtl}^1)W_{vtl}^2+b_{vtl}^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vtl}$$\end{document} is the weight matrix, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{vtl}$$\end{document} is the bias and ReLu is the activation function.Fig. 3The sequential translation process from visual modality to text modality to audio modality. The sequential translation process from visual modality to text modality to audio modality. Next, the second step in sequential translation involves further refining high-level semantic information within cross-modal features using an encoder. We feed the translation feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document} from the visual modality to the text modality into the encoder. Here, the Query, Key, and Value of the multi-head attention mechanism all correspond to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document}. Finally, the encoded output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{vt}$$\end{document} of the translation feature is obtained. The calculation process is shown in Eqs. (13–15).13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{vt}^{\prime }=\textrm{MultiHead}(D_{vt},D_{vt},D_{vt})\end{aligned}$$\end{document}14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{vt}^{\prime \prime }=\textrm{LayerNorm}(D_{vt}+E_{vt}^{\prime })\end{aligned}$$\end{document}15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{vt}=\textrm{Relu}(E_{vt}^{\prime \prime }W_{evtl}^1+b_{evtl}^1)W_{evtl}^2+b_{evtl}^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{evtl}$$\end{document} is the weight matrix, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{evtl}$$\end{document} is the bias and ReLu is the activation function. Finally, the third step in sequential translation involves using the decoder to model dependencies between cross-modal features and the third modality, ensuring that the final representation maintains semantic consistency and feature complementarity across all three modalities. We use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{a}$$\end{document} as the query for the multi-head attention mechanism, and we use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{vt}$$\end{document} as both the key and the value. This allows us to integrate translation features from the text and visual modalities with the audio modality features, ensuring that the extracted contextual information contains features from all three modalities. Next, we normalize and fully connect the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vta}^{\prime }$$\end{document} of the multi-head attention mechanism to obtain the final output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vta}$$\end{document} of the sequential translation. The calculation process is shown in Eqs. (16–18).16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vta}^{\prime }=\textrm{MultiHead}(E_a,E_{vt},E_{vt})\end{aligned}$$\end{document}17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vta}^{\prime \prime }=\textrm{LayerNorm}(E_a+D_{vta}^{\prime })\end{aligned}$$\end{document}18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{vta}=\textrm{Relu}(D_{vta}^{\prime \prime }W_{vtal}^1+b_{vtal}^1)W_{vtal}^2+b_{vtal}^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vtal}$$\end{document} is the weight matrix, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{vtal}$$\end{document} is the bias and ReLu is the activation function. The same method can be used to obtain the results \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{at}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{atv}$$\end{document} of the backward translation process. We use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vta}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{at}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{atv}$$\end{document} as translation features to improve the accuracy of sentiment classification. Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{at}$$\end{document} interact directly with the text mode, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vta}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{atv}$$\end{document} integrate the three modality relationships. The next subsection explains how to use these output features. Common space module The common space module of this paper is shown as the orange area in Fig. 2 and as depicted in Fig. 4. Inspired by the work of Zeng et al.35, we created six weight matrices: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{va}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vt}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{ta}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{vta}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{atv}$$\end{document}and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{m}$$\end{document}. This module generates initial joint features (IJFs) and missing joint features (MJFs) by linearly projecting \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{E_v,E_t,E_a\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{D_{vt},D_{vta},D_{at},D_{atv}\}$$\end{document}. The IJFs contain the self-attention-encoded features of the three modalities, and the MJFs contain the decoder output features of the sequential translation process. We assigned weight matrices to the IJFs and MJFs to ensure that the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{E_v,E_t,E_a\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{D_{vt},D_{vta},D_{at},D_{atv}\}$$\end{document} could learn intermodal information dynamically and in a balanced manner during training. The calculation process is shown in Eqs. (19–24).19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_v=[W_{va}E_v||W_{vt}E_v]\end{aligned}$$\end{document}20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_t=[W_{vt}E_t||W_{ta}E_t]\end{aligned}$$\end{document}21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_a=[W_{va}E_a||W_{ta}E_a]\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||$$\end{document} represents the concatenation operation, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{v}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{a}$$\end{document} represent the self-correlated common spaces of the translated visual and audio modalities, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{t}$$\end{document} represents the self-correlated common space of the text modality output by the encoder.22\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{vta}=[W_{vta}D_{vta}||W_{atv}D_{vta}]\end{aligned}$$\end{document}23\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{atv}=[W_{vta}D_{atv}||W_mD_{atv}]\end{aligned}$$\end{document}24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{vtat}=[W_{atv}D_{vt}||W_mD_{at}]\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||$$\end{document} represents the concatenation operation, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{vta}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{atv}$$\end{document} represent the self-correlation spaces of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vta}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{atv}$$\end{document}, respectively, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{vtat}$$\end{document} represents the cross-correlation space of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{at}$$\end{document}.Fig. 4Illustration of the common space module. Illustration of the common space module. This spatial structure facilitates the exchange of information between multimodal features and different weights during training, ensuring that joint features can approximate complete modalities, regardless of which is missing. Next, concatenate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{v}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{t}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{a}$$\end{document} to form the initial joint features (IJFs). Then, concatenate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{vtat}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{vta}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{atv}$$\end{document} to form the missing joint features (MJFs). Finally, input the IJFs and MJFs into the transformer encoder-decoder architecture for final sentiment classification. The calculation process is shown in Eqs. (25–26).25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} IJFs=[C_v||C_a||C_t]\end{aligned}$$\end{document}26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} MJFs=[C_{vtat}||C_{vta}||C_{atv}]\end{aligned}$$\end{document} Sentiment classification stage We use the transformer encoder-decoder architecture to capture the conditional relationship between initial joint features (IJFs) and missing joint features (MJFs). In the encoder, the IJFs serve as the query for the multi-head attention mechanism, while the MJFs serve as the key and value. This brings the MJFs closer to the IJFs. The calculation process is shown in Eqs. (27–29).27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{all}^{\prime }=\textrm{MultiHead}(IJFs,MJFs,MJFs)\end{aligned}$$\end{document}28\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{all}^{\prime \prime }=\textrm{LayerNorm}(EJFs+E_{all}^{\prime })\end{aligned}$$\end{document}29\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{all}=\textrm{Relu}(E_{all}^{\prime \prime }W_e^1+b_e^1)W_e^2+b_e^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_e^{1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_e^{2}$$\end{document} are weight matrices, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_e^{1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_e^{2}$$\end{document} are learnable biases, and ReLU is the activation function. We feed the output of the encoder into the decoder and set the query, key, and value to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document}. This enables the decoder to concentrate on internal dependencies within the multimodal space while preserving contextual consistency. The calculation process is shown in Eqs. (30–32).30\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{all}^{\prime }=\textrm{MultiHead}(E_{all},E_{all},E_{all})\end{aligned}$$\end{document}31\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{all}^{\prime \prime }=\textrm{LayerNorm}(D_{all}^{\prime }+E_{all})\end{aligned}$$\end{document}32\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{all}=\textrm{Relu}(D_{all}^{\prime \prime }W_d^1+b_d^1)W_d^2+b_d^2\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_d^{1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_d^{2}$$\end{document} are weight matrices, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_d^{1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_d^{2}$$\end{document} are learnable biases, and ReLU is the activation function. Finally, we achieve deep modal fusion through decoder loss and use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} to perform emotion classification. Modal contribution In this subsection, we delve into the roles of the three modalities within the model and elucidate the connections among them. Text modality: The text modality first achieves emotion-related semantic enhancement through a self-attention mechanism. Subsequently, within the sequential translation module, the text modality serves as the core of bidirectional translation, embedding semantic features into the chained translation process of the visual and audio modalities. Throughout this process, translation loss ensures cross-modal features maintain semantic consistency. Within the public space module, the text modality’s self-attention encoded feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_t}$$\end{document} participates in constructing the IJFs via weight matrices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{vt}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{ta}}$$\end{document}. Simultaneously, other two modalities’ translated features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{vt}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{at}}$$\end{document} contribute to building the MJFs. During classification, text modality-related features dominate the sentiment semantic orientation of the joint representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_{all}}$$\end{document} through a Transformer encoder-decoder architecture. The role of textual modality in final sentiment prediction involves deep engagement in the translation process and leveraging its high-quality features during the classification stage to enhance accuracy. Visual modality: After enhancing emotion-related semantics through the self-attention mechanism, the encoded features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_v}$$\end{document} from the visual modality are transformed into cross-modal features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{vt}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{vta}}$$\end{document} during the forward translation process in the sequential translation module. Additionally, these features supplement visual information as a third modality during the reverse translation process. Within the public space module, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_v}$$\end{document} participates in IJFs construction via \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{va}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{vt}}$$\end{document}, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{vt}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{vta}}$$\end{document} contribute to MJFs fusion. Its role in final sentiment prediction is to clarify emotional inclinations through visual features when textual semantics are ambiguous or audio characteristics are unclear, thereby enhancing prediction accuracy. For instance, when the neutral word “ok” is paired with a smiling emoji, the predicted sentiment intensity leans more toward “positive”. Audio modality: After enhancing emotion-related semantics through the self-attention mechanism, the encoded features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_a}$$\end{document} from the audio modality undergo a reverse translation process within the sequential translation module. This converts audio features into cross-modal features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{at}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{atv}}$$\end{document} that are compatible with both the text modality and visual modality. Additionally, during the forward translation process, these features serve as a third modality to supplement audio information. Within the public space module, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E_a}$$\end{document}participates in IJFs construction via \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{va}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_{ta}}$$\end{document}, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{at}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D_{atv}}$$\end{document} contribute to MJFs fusion. Its role in final sentiment prediction is to supplement textual semantics with an emotional intensity dimension. For instance, when paired with high-energy audio, the text “good” yields sentiment intensity predictions leaning toward “strongly positive”. Training objective For the STMSA model, we use four losses to guide training toward the target: pre-trained loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{pre}$$\end{document}), translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{trans}$$\end{document}), decoder loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{de}$$\end{document}), and classification loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{cls}$$\end{document}). As shown in Eq. (33).33\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L=L_{cls}+\alpha L_{pre}+\beta L_{de}+\lambda L_{trans}\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document} are the loss weights. Each loss will be described in detail below. First, the Kullback–Leibler (KL) divergence method is commonly used to measure the difference between two probability distributions. However, its inherent asymmetry can cause the optimization process to skew in one direction. Therefore, we use symmetric Jensen–Shannon (JS) divergence to calculate loss. The calculation process is shown in Eqs. (34–35).34\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{KL}(p||q)=\sum _{i=1}^Np(x_i)\cdot \frac{p(x_i)}{q(x_i)}\end{aligned}$$\end{document}35\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(p||q)=\frac{1}{2}D_{KL}(p||q)+\frac{1}{2}D_{KL}(q||p)\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q$$\end{document} are two probability distributions. Pre-trained loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{pre}$$\end{document}): The objective is to make the model-generated joint features with missing modalities converge toward complete joint features. It is determined by the JS divergence between the transformer encoder’s output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} and the pre-trained output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{pre}$$\end{document}, where the pre-trained model is trained using complete modalities. The calculation process is shown in Eqs. (36–37). Pre-trained loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{pre}$$\end{document}): The objective is to make the model-generated joint features with missing modalities converge toward complete joint features. It is determined by the JS divergence between the transformer encoder’s output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} and the pre-trained output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{pre}$$\end{document}, where the pre-trained model is trained using complete modalities. The calculation process is shown in Eqs. (36–37). Pre-trained loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{pre}$$\end{document}): The objective is to make the model-generated joint features with missing modalities converge toward complete joint features. It is determined by the JS divergence between the transformer encoder’s output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} and the pre-trained output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{pre}$$\end{document}, where the pre-trained model is trained using complete modalities. The calculation process is shown in Eqs. (36–37). 36\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{pre}=JS(E_{all}||E_{pre})\end{aligned}$$\end{document} 37\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(E_{all}||E_{pre})=\frac{1}{2}D_{KL}(E_{all}||E_{pre})+\frac{1}{2}D_{KL}(E_{pre}||E_{all})\end{aligned}$$\end{document} (2)Sequential translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{trans}$$\end{document}): The objective is to guide the model to learn mapping rules during the translation process of one modality to another and to supervise the process to preserve the target modality’s semantic information and emotional features. In the sequential translation module, translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoT}$$\end{document}) from the visual modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the visual to the text modality. Translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoT}$$\end{document}) from the audio modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the audio to the text modality. The calculation process is shown in Eqs. (38–41). (2)Sequential translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{trans}$$\end{document}): The objective is to guide the model to learn mapping rules during the translation process of one modality to another and to supervise the process to preserve the target modality’s semantic information and emotional features. In the sequential translation module, translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoT}$$\end{document}) from the visual modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the visual to the text modality. Translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoT}$$\end{document}) from the audio modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the audio to the text modality. The calculation process is shown in Eqs. (38–41). Sequential translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{trans}$$\end{document}): The objective is to guide the model to learn mapping rules during the translation process of one modality to another and to supervise the process to preserve the target modality’s semantic information and emotional features. In the sequential translation module, translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoT}$$\end{document}) from the visual modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the visual to the text modality. Translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoT}$$\end{document}) from the audio modality to the text modality is determined by the JS divergence between the text modality’s encoder output and the decoder output from the audio to the text modality. The calculation process is shown in Eqs. (38–41). 38\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{VtoT}=JS(D_{vt}||E_t)\end{aligned}$$\end{document}39\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(D_{vt}||E_t)=\frac{1}{2}D_{KL}(D_{vt}||E_t)+\frac{1}{2}D_{KL}(E_t||D_{vt})\end{aligned}$$\end{document}40\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{AtoT}=JS(D_{at}||E_t)\end{aligned}$$\end{document}41\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(D_{at}||E_t)=\frac{1}{2}D_{KL}(D_{at}||E_t)+\frac{1}{2}D_{KL}(E_t||D_{at})\end{aligned}$$\end{document}In order to enhance the ability of modality translation methods to deal with missing modalities, we continue to translate it to the remaining third modality based on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{vt}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{at}$$\end{document}. We calculate the JS divergence between the translation features of visual modality to text modality and then to audio modality and the encoding features of audio modality as the translation loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoTtoA}$$\end{document}). Similarly, the translation loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoTtoV}$$\end{document} can be obtained. The calculation process is shown in Eqs. (42–45).42\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{VtoTtoA}=JS(D_{vta}||E_a)\end{aligned}$$\end{document}43\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(D_{vta}||E_a)=\frac{1}{2}D_{KL}(D_{vta}||E_a)+\frac{1}{2}D_{KL}(E_a||D_{vta})\end{aligned}$$\end{document}44\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{AtoTtoV}=JS(D_{atv}||E_v)\end{aligned}$$\end{document}45\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(D_{atv}||E_v)=\frac{1}{2}D_{KL}(D_{atv}||E_v)+\frac{1}{2}D_{KL}(E_v||D_{atv})\end{aligned}$$\end{document}For convenience, let the sum of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoT}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoT}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{VtoTtoA}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{AtoTtoV}$$\end{document} be denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{trans}$$\end{document}. As shown in Eq. (46).46\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{trans}=L_{VtoT}+L_{AtoT}+L_{VtoTtoA}+L_{AtoTtoV}\end{aligned}$$\end{document} (3)Decoder loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{de}$$\end{document}): To achieve deep multimodal fusion, we need the final joint representation to align with the initial joint features derived from complete modalities. The decoder loss supervises this alignment by minimizing the divergence between the output of the transformer encoder–decoder architecture and the initial joint features, ensuring that the missing joint features are effectively reconstructed under the guidance of complete ones. Therefore, We calculate the JS divergence between the output of the transformer encoder-decoder architecture and the IJFs, which allows \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} to become a joint representation of the MJFs guided by the IJFs. The calculation process is shown in Eqs. (47–48). (3)Decoder loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{de}$$\end{document}): To achieve deep multimodal fusion, we need the final joint representation to align with the initial joint features derived from complete modalities. The decoder loss supervises this alignment by minimizing the divergence between the output of the transformer encoder–decoder architecture and the initial joint features, ensuring that the missing joint features are effectively reconstructed under the guidance of complete ones. Therefore, We calculate the JS divergence between the output of the transformer encoder-decoder architecture and the IJFs, which allows \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} to become a joint representation of the MJFs guided by the IJFs. The calculation process is shown in Eqs. (47–48). Decoder loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{de}$$\end{document}): To achieve deep multimodal fusion, we need the final joint representation to align with the initial joint features derived from complete modalities. The decoder loss supervises this alignment by minimizing the divergence between the output of the transformer encoder–decoder architecture and the initial joint features, ensuring that the missing joint features are effectively reconstructed under the guidance of complete ones. Therefore, We calculate the JS divergence between the output of the transformer encoder-decoder architecture and the IJFs, which allows \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} to become a joint representation of the MJFs guided by the IJFs. The calculation process is shown in Eqs. (47–48). 47\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{de}=JS(D_{all}||IJFs)\end{aligned}$$\end{document} 48\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} JS(D_{all}||IJFs)=\frac{1}{2}D_{KL}(D_{all}||IJFs)+\frac{1}{2}D_{KL}(IJFs||D_{all})\end{aligned}$$\end{document} (4)Classification loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{cls}$$\end{document}): The ultimate goal is accurate sentiment prediction. We adopt the cross-entropy loss to measure the discrepancy between predicted labels and ground-truth labels, driving the model to optimize its classification performance. We feed the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} into a fully connected layer network, using Softmax as the activation function to compute the predicted value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}$$\end{document}. As shown in Eq. (49). (4)Classification loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{cls}$$\end{document}): The ultimate goal is accurate sentiment prediction. We adopt the cross-entropy loss to measure the discrepancy between predicted labels and ground-truth labels, driving the model to optimize its classification performance. We feed the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} into a fully connected layer network, using Softmax as the activation function to compute the predicted value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}$$\end{document}. As shown in Eq. (49). Classification loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{cls}$$\end{document}): The ultimate goal is accurate sentiment prediction. We adopt the cross-entropy loss to measure the discrepancy between predicted labels and ground-truth labels, driving the model to optimize its classification performance. We feed the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{all}$$\end{document} into a fully connected layer network, using Softmax as the activation function to compute the predicted value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}$$\end{document}. As shown in Eq. (49). 49\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{y}=\textrm{Softmax}(W_cE_{all}+b_c)\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{c}$$\end{document} is the weight matrix and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{c}$$\end{document} is the bias. Cross-entropy loss reduces the likelihood that the model will overfit or underfit, resulting in more consistent predictions with the sample features. Therefore, we use Eq. (50) as the final classification loss.50\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{cls}=-\frac{1}{N}\sum _{n=1}^Ny_n\log \hat{y_n}\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N$$\end{document} is the number of samples, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{n}$$\end{document} is the true label of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}th sample, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{n}$$\end{document} is the predicted label. Experiments Benchmark datasets We conducted experiments on two well-established datasets (IEMOCAP43 and CMU-MOSI44) to validate our proposed model. First, the IEMOCAP dataset captures conversational scenes using professional equipment. While it exhibits high modal completeness, it allows simulating real-world controlled missing-modal scenarios by randomly partitioning missing modalities. This aligns with the logic of modal loss occurrence in practical applications. Additionally, the CMU-MOSI dataset comprises video samples sourced directly from online platforms like YouTube. Consequently, single-modality or multi-modality gaps inevitably occur due to limitations in recording equipment or format processing during user uploads. Therefore, we conduct experiments under missing modalities alongside full-modality experiments on the two datasets mentioned above. Next, we present detailed information about these two datasets. IEMOCAP: An emotional dataset capturing fine-grained actions in conversations, with sentiment labels categorized into 9 classes: neutral, frustrated, angry, sad, happy, excited, surprised, fearful, and disappointed. It comprises 5 recorded videos of dyadic conversations involving 10 speakers in total. Each session features spontaneous dialogue between one male and one female speaker, with an average conversation duration of 5 minutes. Session types encompass 3 emotional scripts and 8 emotional scenarios. In our experiments, we conducted a binary classification task on IEMOCAP, mapping emotion labels to positive and negative categories. Specifically, negative emotion labels include frustration, anger, sadness, fear, and disappointment, while positive emotion labels include happiness and excitement. The distribution within the dataset is shown in Table 4. CMU-MOSI: The Carnegie Mellon University Multimodal Viewpoint Sentiment and Intensity dataset captures the fine-grained spectrum of emotional expression. This dataset segments videos into utterance-level clips and labels each utterance with a sentiment orientation. It comprises 93 videos sourced from online sharing platforms like YouTube, with subjective sentiment annotations applied to 2,199 video segments. Additionally, it includes 1503 objectively annotated video segments. Emotion annotations span a linear scale from -3 (strongly negative) to +3 (strongly positive), corresponding to eight emotional categories. We conducted a three-class classification experiment on the CMU-MOSI dataset. We used sentiment tagging to create emotion recognition settings and evaluated the model using ACC and M-F1 scores. As shown in Eqs. (51–52).51\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Acc}=\frac{N_{true}}{N}\end{aligned}$$\end{document}52\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathrm {M-F}1=\frac{2PR}{P+R}.\end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{true}$$\end{document} is the number of correctly predicted samples, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N$$\end{document} is the total number of samples, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P$$\end{document} is the positive predictive value, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R$$\end{document} is the recall rate. Data preprocessing The samples in the above datasets contain three modalities, but our model’s encoders only require independent features from each modality as input. Therefore, we used the method proposed by Zeng et al.35 to extract each modality’s features individually. Visual modality feature extraction: We used the OpenFace 2.045 toolkit to extract facial features. The extraction process provided feedback on three key pieces of information: facial details, head movements, and eye movements. We used the 709-dimensional visual representation of these three pieces of information to train the model. Additionally, we used frame numbers, face IDs, timestamps, confidence scores, and other information to supervise the extraction process. Text modality feature extraction: We used a pre-trained BERT46 model to extract text features. This model has 768 hidden units, 12 attention heads, and 12 encoder layers with a total of 110 million parameters, through which we obtained a 768-dimensional text feature representation. Audio modality feature extraction: We used Librosa47 for multidimensional feature extraction. This tool can obtain spectral, time-domain, and structural features from the original audio signal. We mixed the audio information from CMU-MOSI and IEMOCAP into a monaural signal and resampled it at 16,000 Hz. We then extracted three features: zero-crossing rate, Mel-frequency cepstral coefficients (MFCCs), and constant Q transform (CQT). We divided each frame into 512 samples and used the 33-dimensional audio features composed of these three features as the audio modality. Our experimental setup is shown in Table 1.Table 1Parameter settings of STMSA.DescriptionSymbolValueBatch size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b$$\end{document}32Epoch number\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e$$\end{document}20Dropout rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p$$\end{document}0.8Hidden size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d$$\end{document}300Learning rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{r}$$\end{document}0.001Missing rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m$$\end{document}[0,0.5]Maximum text length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{t}$$\end{document}25Maximum audio length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{a}$$\end{document}150Maximum video length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{v}$$\end{document}100Loss weights\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha ,\beta ,\lambda$$\end{document}0.1 Table 1Parameter settings of STMSA.DescriptionSymbolValueBatch size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b$$\end{document}32Epoch number\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e$$\end{document}20Dropout rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p$$\end{document}0.8Hidden size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d$$\end{document}300Learning rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{r}$$\end{document}0.001Missing rate\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m$$\end{document}[0,0.5]Maximum text length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{t}$$\end{document}25Maximum audio length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{a}$$\end{document}150Maximum video length\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{v}$$\end{document}100Loss weights\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha ,\beta ,\lambda$$\end{document}0.1 Parameter settings of STMSA. Baseline models We selected the following ten state-of-the-art benchmark models for comparison experiments to verify the performance of STMSA. The models are described below:AE48: This is a method for encoding and decoding input vectors through nonlinear mapping to learn their latent representations.CRA30: A network that simulates the difference between predicted and original data values by superimposing residual encoders and ultimately estimating missing modalities.MCTN40: A model that translates one modality into another through an iterative process of translation. During this process, the model learns the joint features between the two modalities.TransM12: An end-to-end model that uses transformer to fuse source modality and target modality information and proposes parallel translation.ICDN49: A model that maps other modalities to the target modality through a special cross-modal transformer structure, and then uses self-supervision to obtain single-peak emotion labels to guide emotion analysis.MMIN31: A network that models sentiment-relatedness from sentence-level representations and predicts missing modalities based on available modalities.TATE_C35: A model that assists joint representation learning by covering missing cases using tag encoding technology.MTMSA11: A model that translates visual and audio modalities into text modalities through modality translation, aiming to improve the quality of feature fusion and thus enhance the robustness of the model.TATE_J50: This model adds different weights to different modalities based on TATE_C to fully utilize each modality.SMCMSA13: A model that uses a pre-constructed pool of similar samples and cosine similarity to identify similar modalities and complete missing modalities. AE48: This is a method for encoding and decoding input vectors through nonlinear mapping to learn their latent representations. AE48: This is a method for encoding and decoding input vectors through nonlinear mapping to learn their latent representations. CRA30: A network that simulates the difference between predicted and original data values by superimposing residual encoders and ultimately estimating missing modalities. CRA30: A network that simulates the difference between predicted and original data values by superimposing residual encoders and ultimately estimating missing modalities. MCTN40: A model that translates one modality into another through an iterative process of translation. During this process, the model learns the joint features between the two modalities. MCTN40: A model that translates one modality into another through an iterative process of translation. During this process, the model learns the joint features between the two modalities. TransM12: An end-to-end model that uses transformer to fuse source modality and target modality information and proposes parallel translation. TransM12: An end-to-end model that uses transformer to fuse source modality and target modality information and proposes parallel translation. ICDN49: A model that maps other modalities to the target modality through a special cross-modal transformer structure, and then uses self-supervision to obtain single-peak emotion labels to guide emotion analysis. ICDN49: A model that maps other modalities to the target modality through a special cross-modal transformer structure, and then uses self-supervision to obtain single-peak emotion labels to guide emotion analysis. MMIN31: A network that models sentiment-relatedness from sentence-level representations and predicts missing modalities based on available modalities. MMIN31: A network that models sentiment-relatedness from sentence-level representations and predicts missing modalities based on available modalities. TATE_C35: A model that assists joint representation learning by covering missing cases using tag encoding technology. TATE_C35: A model that assists joint representation learning by covering missing cases using tag encoding technology. MTMSA11: A model that translates visual and audio modalities into text modalities through modality translation, aiming to improve the quality of feature fusion and thus enhance the robustness of the model. MTMSA11: A model that translates visual and audio modalities into text modalities through modality translation, aiming to improve the quality of feature fusion and thus enhance the robustness of the model. TATE_J50: This model adds different weights to different modalities based on TATE_C to fully utilize each modality. TATE_J50: This model adds different weights to different modalities based on TATE_C to fully utilize each modality. SMCMSA13: A model that uses a pre-constructed pool of similar samples and cosine similarity to identify similar modalities and complete missing modalities. SMCMSA13: A model that uses a pre-constructed pool of similar samples and cosine similarity to identify similar modalities and complete missing modalities. Performance of STMSA in less classification We conducted three-classification experiments on the CMU-MOSI dataset and two-classification experiments on the IEMOCAP dataset. We then compared our proposed model with ten benchmark models. The model results of MTMSA, ICDN, and MRAN were selected from Liu et al.11, while the other models’ results were selected from Zeng et al.35,50. To verify the STMSA model’s performance under complex missing modalities conditions, we divided the experiments on each dataset into two parts. The first part involved conducting experiments under single-modality missing conditions, while the second part involved conducting experiments under multimodal missing conditions. Additionally, we set the missing rate from 0 to 0.5 and used M-F1 and ACC as performance evaluation metrics. The results are shown in Tables 2 and 3.Table 2Experimental results under a single-modality missing.DatasetsModels00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCCMU-MOSIAE56.7879.6954.0779.1753.4078.1351.2872.5350.7573.4844.9969.32CRA56.8579.7354.3779.3853.5778.2451.6772.8451.0273.7945.3869.45MCTN57.3279.7555.4879.8753.9977.4952.3171.5951.6473.8145.7668.11TransM57.8480.2157.5379.6955.2178.4252.8772.9252.4972.4045.8668.23ICDN55.7182.3054.8581.2554.5480.7353.5179.6946.0968.2341.0060.42MRAN56.7983.8556.2182.2155.2082.2954.5281.2554.1580.7353.9978.65MMIN60.4182.2957.7581.8655.3880.2053.6579.2452.5576.3348.9570.76TATE_C58.3284.9058.2184.4655.4681.2555.1180.7354.1180.2151.7174.04MTMSA58.1284.8957.4384.8957.4883.8555.9181.7754.8781.2554.3179.16TATE_J60.1886.5458.4985.8556.5282.6556.0881.7754.6780.5352.6676.04Ours58.6586.4658.5285.9457.8783.8556.8183.8554.4380.2154.3679.69IEMOCAPAE76.1582.0975.2480.2675.0278.0173.9277.4370.1976.0167.2776.43CRA77.0582.1375.9580.9775.1378.0974.0278.1170.6976.1267.7576.49MCTN78.5782.2777.7481.0275.3778.2774.6978.5271.7576.2968.1776.63TransM79.5782.6478.0381.8676.3380.4375.8378.6472.0177.2768.5776.65ICDN77.3782.8176.4681.3474.1380.5665.0078.0473.2675.1760.5073.35MRAN81.2185.9881.0684.8880.6184.3879.9983.5178.6382.9075.8281.33MMIN80.8383.4378.8582.5877.0981.2776.6380.4372.8178.4370.5877.45TATE_C81.1585.3979.9985.0979.1084.0778.4583.2576.7482.7574.4382.43MTMSA81.3686.1481.8185.2481.4784.4680.2084.2879.5382.9475.8482.55TATE_J81.7686.4681.2585.2479.5784.8078.0683.7677.8482.9775.7682.51Ours83.7487.3383.2187.1682.8586.3181.0385.7280.0885.1379.0584.79Significant values are in bold. Table 2Experimental results under a single-modality missing.DatasetsModels00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCCMU-MOSIAE56.7879.6954.0779.1753.4078.1351.2872.5350.7573.4844.9969.32CRA56.8579.7354.3779.3853.5778.2451.6772.8451.0273.7945.3869.45MCTN57.3279.7555.4879.8753.9977.4952.3171.5951.6473.8145.7668.11TransM57.8480.2157.5379.6955.2178.4252.8772.9252.4972.4045.8668.23ICDN55.7182.3054.8581.2554.5480.7353.5179.6946.0968.2341.0060.42MRAN56.7983.8556.2182.2155.2082.2954.5281.2554.1580.7353.9978.65MMIN60.4182.2957.7581.8655.3880.2053.6579.2452.5576.3348.9570.76TATE_C58.3284.9058.2184.4655.4681.2555.1180.7354.1180.2151.7174.04MTMSA58.1284.8957.4384.8957.4883.8555.9181.7754.8781.2554.3179.16TATE_J60.1886.5458.4985.8556.5282.6556.0881.7754.6780.5352.6676.04Ours58.6586.4658.5285.9457.8783.8556.8183.8554.4380.2154.3679.69IEMOCAPAE76.1582.0975.2480.2675.0278.0173.9277.4370.1976.0167.2776.43CRA77.0582.1375.9580.9775.1378.0974.0278.1170.6976.1267.7576.49MCTN78.5782.2777.7481.0275.3778.2774.6978.5271.7576.2968.1776.63TransM79.5782.6478.0381.8676.3380.4375.8378.6472.0177.2768.5776.65ICDN77.3782.8176.4681.3474.1380.5665.0078.0473.2675.1760.5073.35MRAN81.2185.9881.0684.8880.6184.3879.9983.5178.6382.9075.8281.33MMIN80.8383.4378.8582.5877.0981.2776.6380.4372.8178.4370.5877.45TATE_C81.1585.3979.9985.0979.1084.0778.4583.2576.7482.7574.4382.43MTMSA81.3686.1481.8185.2481.4784.4680.2084.2879.5382.9475.8482.55TATE_J81.7686.4681.2585.2479.5784.8078.0683.7677.8482.9775.7682.51Ours83.7487.3383.2187.1682.8586.3181.0385.7280.0885.1379.0584.79Significant values are in bold. Experimental results under a single-modality missing. Significant values are in bold. Experiment under single modality missing. In this experiment, the missing rate was set from 0 to 0.5, and the experimental results are shown in Table 2. On the CMU-MOSI dataset, when the missing rate is set to 0, the M-F1 score of STMSA is 1.76% lower than that of MMIN, and the ACC value is 0.08% lower than that of TATE_J. When the missing rate is set to 0.4, the M-F1 score and ACC value of STMSA are 0.44% and 1.04% lower than those of MTMSA. Notably, when the missing rate is set to 0.1, 0.2, 0.3, and 0.5, the M-F1 score and ACC value of STMSA are superior to those of other models. Specifically, when the missing rate is 0.5, STMSA improves the M-F1 score by 0.05% to 13.36% and the ACC value by 0.53% to 19.27% compared to other models. On the IEMOCAP dataset, when the missing rate is set to 0, 0.1, 0.2, 0.3, 0.4, and 0.5, the M-F1 score and ACC value of STMSA are completely superior to other benchmark models. Experiments under multiple modalities missing. In this experiment, the missing rate was set from 0 to 0.5, and the experimental results are shown in Table 3. On the CMU-MOSI dataset, when the missing rate is set to 0, the M-F1 score of STMSA decreases by 1.76% compared to MMIN, and the ACC value decreases by 0.08% compared to TATE_J. When the missing rate is set to 0.5, the M-F1 value and ACC value decrease by 0.29% and 0.56% compared to TATE_J. Notably, when the missing rate is set to 0.1, 0.2, 0.3, and 0.4, STMSA’s M-F1 score and ACC value outperform the other 10 baseline models. Specifically, when the missing rate is 0.3, STMSA improves the M-F1 value by 1.77% to 9.19% and the ACC value by 3.65% to 12.59% compared to other models. On the IEMOCAP dataset, when the missing rate is set to 0, 0.1, 0.2, 0.3, 0.4, and 0.5, the M-F1 score and ACC value of STMSA are completely superior to other benchmark models.Table 3Experimental results under multiple -modality missing.DatasetsModels00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCCMU-MOSIAE56.7879.6952.8075.6550.8474.1846.2369.1844.4069.0540.2966.01CRA56.8579.7352.8575.6851.0274.7346.8769.2345.1769.4841.7766.82MCTN57.3279.7552.9775.8951.7574.1646.9869.2945.7369.5542.9867.02TransM57.8480.2153.4977.0851.9774.2448.2370.5147.0270.3843.2867.74ICDN55.7182.3054.5580.2153.3478.1347.0469.2742.4162.5037.8157.29MRAN56.7983.8555.7483.3354.0980.7351.6077.0849.5373.9649.1772.40MMIN60.4182.2955.4980.1252.7976.2648.9773.2747.3974.2844.6368.92TATE_C58.3284.9056.3881.7754.8781.0752.1277.6051.1976.5651.1573.23MTMSA58.1284.8957.4384.3755.7181.2552.8278.1251.3276.0451.7273.43TATE_J60.1886.5457.3782.8155.6281.6753.6578.1253.2877.9851.9276.60Ours58.6586.4657.4784.8956.4883.3355.4281.7754.7777.9851.6376.04IEMOCAPAE76.1582.0975.0779.8474.2076.9171.5576.0769.7375.1667.1575.22CRA77.0582.1375.2179.9574.2277.0371.8676.4170.1375.2967.3175.42MCTN78.5782.2776.8380.5674.7777.8972.2777.0371.0275.8467.5175.88TransM79.5782.6477.2181.1375.8779.0172.3678.1571.3876.8868.0276.04ICDN77.3782.8172.5679.2571.7378.9969.9477.1769.5974.6568.9873.26MRAN81.2185.9880.2285.0779.8683.6079.1482.8975.8081.2568.6178.30MMIN80.8383.4378.0282.2376.3879.5373.0579.0271.2277.2769.3977.01TATE_C81.1585.3978.3783.6377.5582.3376.1482.2174.0981.9472.4980.57MTMSA81.3686.1480.2885.1780.3984.1279.3083.8576.0783.0774.8082.03TATE_J81.7686.4680.6785.3079.1283.7778.9984.6478.4482.7576.9782.25Ours83.7487.3381.9185.8980.9284.8879.5585.5579.0184.1277.0483.19Significant values are in bold. Table 3Experimental results under multiple -modality missing.DatasetsModels00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCCMU-MOSIAE56.7879.6952.8075.6550.8474.1846.2369.1844.4069.0540.2966.01CRA56.8579.7352.8575.6851.0274.7346.8769.2345.1769.4841.7766.82MCTN57.3279.7552.9775.8951.7574.1646.9869.2945.7369.5542.9867.02TransM57.8480.2153.4977.0851.9774.2448.2370.5147.0270.3843.2867.74ICDN55.7182.3054.5580.2153.3478.1347.0469.2742.4162.5037.8157.29MRAN56.7983.8555.7483.3354.0980.7351.6077.0849.5373.9649.1772.40MMIN60.4182.2955.4980.1252.7976.2648.9773.2747.3974.2844.6368.92TATE_C58.3284.9056.3881.7754.8781.0752.1277.6051.1976.5651.1573.23MTMSA58.1284.8957.4384.3755.7181.2552.8278.1251.3276.0451.7273.43TATE_J60.1886.5457.3782.8155.6281.6753.6578.1253.2877.9851.9276.60Ours58.6586.4657.4784.8956.4883.3355.4281.7754.7777.9851.6376.04IEMOCAPAE76.1582.0975.0779.8474.2076.9171.5576.0769.7375.1667.1575.22CRA77.0582.1375.2179.9574.2277.0371.8676.4170.1375.2967.3175.42MCTN78.5782.2776.8380.5674.7777.8972.2777.0371.0275.8467.5175.88TransM79.5782.6477.2181.1375.8779.0172.3678.1571.3876.8868.0276.04ICDN77.3782.8172.5679.2571.7378.9969.9477.1769.5974.6568.9873.26MRAN81.2185.9880.2285.0779.8683.6079.1482.8975.8081.2568.6178.30MMIN80.8383.4378.0282.2376.3879.5373.0579.0271.2277.2769.3977.01TATE_C81.1585.3978.3783.6377.5582.3376.1482.2174.0981.9472.4980.57MTMSA81.3686.1480.2885.1780.3984.1279.3083.8576.0783.0774.8082.03TATE_J81.7686.4680.6785.3079.1283.7778.9984.6478.4482.7576.9782.25Ours83.7487.3381.9185.8980.9284.8879.5585.5579.0184.1277.0483.19Significant values are in bold. Experimental results under multiple -modality missing. Significant values are in bold. Theoretical analysis: According to Tables 2 and 3, the AE and CRA models perform poorly among the selected benchmark models. In contrast, the MCTN and TransM models perform better than the AE and CRA models. Therefore, it can be concluded that, when dealing with missing modalities, modality translation-based methods are more advantageous than autoencoder-based methods. Additionally, we find that CRA only uses simple cascade operations in the feature fusion process, while TransM uses a paralllel end-to-end translation structure that enables the feature fusion vector to contain more intermodal correlation features. For the ICDN model, we observe that, when the missing rate exceeds 0.4, the M-F1 score and ACC value drop significantly. This is because ICDN achieves feature fusion by establishing projections between modalities. However, when a modality is missing, the mapping process cannot learn effective complementary features. Consequently, ICDN struggles to address this issue, resulting in a significant decline in performance. Meanwhile, the experimental results show that the performance of the MSA model decreases as the missing rate increases. However, Table 3 shows that when the missing rate is 0.3 on the IEMOCAP dataset, the M-F1 score and ACC value of our proposed model are higher than when the missing rate is 0.2. We analyzed that the variation in sample composition caused by random missing data partitioning is the primary reason for performance fluctuations. Since datasets under different missing rates are generated through random partitioning, this random strategy may lead to accidental differences in the distribution of actual missing modal types and the difficulty distribution of the samples themselves across varying missing rates. For instance, at a 0.3 missing rate, a higher proportion of discarded samples may feature clearly expressed sentiments or contain text modalities, resulting in overall better performance compared to sample combinations at a 0.2 missing rate. Furthermore, STMSA’s text-centric design makes it sensitive to the presence of text-modal samples. Consequently, when text-modal samples constitute a higher proportion of available samples at a given missing rate, the model is more likely to obtain accurate semantic guidance, leading to improved performance. Although performance at a dropout rate of 0.3 is marginally better than at 0.2, the overall results for IEMOCAP in Table 3 show a consistent trend of declining model performance as the dropout rate increases (from 0 to 0.5, M-F1 decreases from 83.74% to 77.04%). Minor fluctuations do not alter this monotonicity and are normal phenomena in randomized experiments. On the CMU-MOSI dataset, STMSA achieves M-F1 score and ACC value second only to those of MMIN and TATE_J when the missing rate is 0. However, STMSA achieves optimal performance when the missing rate is 0.1, 0.2 and 0.3, demonstrating its ability to handle missing modalities. The reason is that TATE_J uses label assistance to enable the encoder to cover all missing cases, but lacks an effective method to further integrate complementary features between modalities. On the other hand, STMSA enhances the feature fusion process by establishing semantic mapping relationships between the three modalities through a sequential translation method. Additionally, STMSA does not set specific generation and completion strategies, suggesting that it is a unified model architecture capable of handling missing data. Qualitative Analysis: The comprehensive experimental results lead to the conclusion that STMSA generally outperforms baseline models in randomly modally missing environments. However, the model’s performance degrades under certain missing rates and continues to decline as the missing rate increases. For instance, on the CMU-MOSI dataset, STMSA achieves an accuracy of 80.21% when the missing rate is 0.4. We conducted an analysis of this phenomenon. First, we randomly partitioned the CMU-MOSI and IEMOCAP datasets with varying missing rates, then trained the model on the partitioned datasets to develop its capability to handle missing modalities. Furthermore, for human-annotated data scenarios within the datasets, our method demonstrates superior controllability and scalability. This is because we can automatically generate missing data while precisely controlling both the missing rate and the missing pattern (e.g., single-modal and multi-modal missing). However, in our experiments, single-modal and multi-modal missing assume an entire modality is entirely absent. This random missing may omit critical information. For instance, when only the visual modality contains key features and is randomly selected as the missing modality in a sample, the absence of these key features inevitably impacts STMSA’s performance. Additionally, while manual annotation methods cannot automatically generate large numbers of missing samples, it can effectively reduce biases caused by random omissions. For instance, in the case of a shy, face-covering smile, although key facial details are obscured by the hand in the visual modality, manual annotation can assign a credible “happy” label to this visual modality feature to aid prediction. Therefore, our training method may fail when encountering such samples compared to manual annotation. Furthermore, manual annotation relies on multimodal information for detailed subjective-objective distinctions, whereas models lacking certain modalities may struggle to accurately interpret context, emotional shifts, or expressive boundaries. To address this, we conducted in-depth analysis within the dataset. For instance, on CMU-MOSI, consider the monologue “[I love The Shawshank Redemption] [and I love Transformers]” about movies. A model relying solely on text might treat the entire sentence as a coherent subjective statement, overlooking potential changes in facial expressions, tone, or pauses when mentioning different films. However, human annotations would identify “I love” as the boundary, recognizing it as two distinct subjective statements based on accompanying facial expressions or intonation changes. Models lacking visual or auditory modalities cannot capture this subtle segmentation criterion. Finally, we selected random missing segmentation to simulate complex missing environments while enabling reproducible missing patterns throughout experiments for systematic research. Performance verification in multi-classification We conducted four-classification and seven-classification experiments on the IEMOCAP dataset to verify STMSA’s performance in multi-classification tasks and robustness under missing modalities. Table 4 shows the classification labels and distribution information of the training and test sets for the two experiments. In both experiments, we set the missing rate from 0 to 0.5 and selected the following comparison models: TATE_C, TATE_J, MRAN, ICDN, MTMSA, and SMCMSA. The results of the experiments are shown in Figs. 5 and 6. The results for the comparison models are from Zeng et al.10. Figure 5 shows the M-F1 scores and ACC values of the TATE_J, MRAN, ICDN, MTMSA and SMCMSA models in the four-classification experiment. The results of the experiment indicate that STMSA outperforms the comparison models when the missing rate ranges from 0 to 0.5. Regarding M-F1 scores, STMSA achieves values that are 1.16% and 0.76% higher than SMCMSA’s when the missing rate is 0.4 and 0.5, respectively. Regarding the ACC values, STMSA achieves the largest improvement compared to SMCMSA when the missing rate is 0.2 and 0.3, with increases of 1.53% and 1.3%, respectively. Figure 6 shows the M-F1 scores and ACC values of the TATE_C, TATE_J, MRAN, ICDN and MTMSA models in the seven-classification experiment. As shown in Fig. 6, when the missing rate is 0.1, the M-F1 score of STMSA is 4.59% higher than that of MTMSA. When the missing rate is 0.5, STMSA’s M-F1 value is 2.86% higher than MTMSA’s. As shown in Fig. 6, when the missing rate is 0, 0.1, or 0.2, STMSA’s ACC values are highest compared to MTMSA’s, increasing by 2.14%, 1.77%, and 2.62%, respectively. Qualitative analysis: Based on the above experimental results, our proposed STMSA model has advantages in multi-classification tasks and is robust under missing modalities. Additionally, multi-classification tasks place higher demands on the feature fusion and decision modules of MSA models. As the number of classification labels increases, the model’s performance decreases. As shown in Fig. 6, MRAN performs poorly in the seven-classification task. We analyze that MRAN only models dependencies between modalities via a mapping method, lacking the ability to extract rich emotional features from multiple dimensions. By contrast, all of the remaining models use multi-head attention mechanisms to capture potential contextual features in sequences from multiple dimensions, allowing the fusion vectors to contain richer emotional features and adapt to multi-classification tasks. Multi-classification tasks impose stringent robustness requirements on models, particularly when expanding from 4 to 7 categories. The semantic boundaries between sentiment categories become increasingly blurred (e.g., “Happy” with “Excited, “Sad” with “Depressed” in the IEMOCAP dataset), and sparse samples in certain categories further amplify the impact of modal deficiencies on model performance. The experimental results in Fig. 6 demonstrate that STMSA maintains a significant performance advantage in the 7-classification task. The core reason lies in its text-modal-dominated tri-modal complementary feature fusion mechanism, which precisely matches the demands of multi-classification scenarios. For categories with pronounced semantic differences, the text modality provides explicit emotional polarity. Combined with audio features (high energy during anger, slow speech during sadness) and visual facial features (furrowed brows during anger, downturned mouth corners during sadness), this forms a triple complementary framework. Even with missing modalities, the strong semantic signals from a single modality can still support accurate classification. Furthermore, for semantically similar categories, STMSA’s sequential translation module captures finer cross-modal distinctions. This is because “excitement” typically involves faster speech rate (audio feature) and exaggerated facial movements (visual feature). These fine-grained features, calibrated by high-quality textual modalities, effectively distinguish “joy” with its gentle cadence and natural smile. This enables the model to maintain high category discrimination even when a single modality is missing. In summary, STMSA’s experimental results in multi-classification tasks demonstrate its capability to handle missing modalities. By dynamically fusing features through text-modal chain translation, STMSA resolves challenges of category discrimination and imbalanced sample distribution in multi-classification tasks, offering a novel solution for multi-classification under missing modality scenarios.Fig. 5Results under the four-classification experiment.Fig. 6Results under the seven-classification experiment.Table 4Distributions of labels in the IEMOCAP dataset.CategoryHap.Ang.Sad.Neu.Fru.Exc.Sur.Four-classesTrain3676556611016–––Test228448423692–––Seven-classesTrain3546706361013110960857Test24144344869574043350 Results under the four-classification experiment. Results under the seven-classification experiment. Table 4Distributions of labels in the IEMOCAP dataset.CategoryHap.Ang.Sad.Neu.Fru.Exc.Sur.Four-classesTrain3676556611016–––Test228448423692–––Seven-classesTrain3546706361013110960857Test24144344869574043350 Distributions of labels in the IEMOCAP dataset. Ablation experiment To explore the performance of the STMSA model under different translation strategies and the effectiveness of each model module, we perform modality and module ablation experiments on the CMU-MOSI dataset. We use “V”, “A”, and “T” to represent the visual, audio, and text modalities, respectively. Additionally, we use words like “VtoT” to represent the translation processes. The specific experimental settings and results are as follows. Ablation experiments on modules Three model variants were generated based on the original model to verify the effectiveness of each STMSA module. The specific model variants are as follows: (1) STMSA-Pre: STMSA directly analyzed and processed multimodal data without pretraining the network. (2) STMSA-STM: The sequential translation module was removed from the model, and the self-attention-encoded features of each modality were fused directly. (3) STMSA-CSM: The common space module was removed, and the modality features after sequential translation were used directly for output and classification. The results of the module ablation experiments are shown in Table 5. First, for the STMSA-Pre variant, when the missing rate is 0, the M-F1 score and ACC value decrease by 0.48% and 0.52%, respectively, compared to STMSA. When the missing rate is 0.3, the M-F1 score and ACC value decrease significantly by 7.13% and 10.42%, respectively. This occurs because the pre-trained task uses complete modalities, so when a modality is missing, the model learns its features from the pre-trained outputs. The above experimental results demonstrate that the pre-trained module effectively enhances STMSA’s ability to handle missing modalities. Compared to STMSA, the STMSA-STM variant exhibits lower M-F1 and ACC values. The largest decreases in these values occur when the missing rate is 0.3: reductions of 2.65% and 3.64% in M-F1 and ACC, respectively. When the missing rate is 0.2, the M-F1 and ACC values decrease by 2.45% and 3.12%, respectively, proving that the sequential translation module can fully mine and utilize the available intermodal features in missing modalities environments, thereby enhancing the robustness of STMSA. For the STMSA-CSM variant, the M-F1 score and ACC value decrease by 3.07% and 4.17%, respectively, when the missing rate is 0. When the missing rate is 0.2, these values decrease by an additional 2.43% and 4.25%, respectively. When the missing rate is 0.4, the M-F1 and ACC values decrease by 3.78% and 5.21%, respectively. These results demonstrate the significant contribution of the public projection module to STMSA performance.Table 5Results of module ablation experiment.Models00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCSTMSA-STM58.1585.9456.8183.8554.4280.7354.1680.2152.7778.6452.6577.60STMSA-CSM55.5882.2956.0982.8152.7978.1252.0877.0851.6376.0453.5379.17STMSA-Pre58.1785.9457.8785.4253.3378.6449.6873.4353.0578.6452.1377.08STMSA58.6586.4658.5285.9456.8483.8556.8183.8554.4380.2154.0879.69 Table 5Results of module ablation experiment.Models00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCSTMSA-STM58.1585.9456.8183.8554.4280.7354.1680.2152.7778.6452.6577.60STMSA-CSM55.5882.2956.0982.8152.7978.1252.0877.0851.6376.0453.5379.17STMSA-Pre58.1785.9457.8785.4253.3378.6449.6873.4353.0578.6452.1377.08STMSA58.6586.4658.5285.9456.8483.8556.8183.8554.4380.2154.0879.69 Results of module ablation experiment. Performance validation of STMSA across different translation sequences and structures To explore the effectiveness of STMSA under different translation orders and structures, we designed the following sequential translation tasks and conducted experiments. These tasks are as follows: Task A (VtoTtoA): In this case, a third modality is introduced to enhance the translation process, but there is no bidirectional sequential translation structure. Task B (AtoTtoV): In this case, a third modality is introduced to enhance the translation process, but there is no bidirectional sequential translation structure. Task C (VtoA, AtoV): In this case, a third modality was not introduced to improve the translation process. Task D (VtoT, AtoT): In this case, a third modality was not introduced to improve the translation process. Instead, text modality was used to guide the translation of modalities. Task E (VtoAtoT, AtoVtoT): In this case, a third modality was introduced to improve the translation process by converting the mixed visual and audio features into text. Task F (TtoVtoA, TtoAtoV): In this case, a third modality was introduced to improve the translation process. The text modality was translated into the two lower-quality modalities. Each task was performed using the CMU-MOSI dataset and setting the missing rate between 0 and 0.5. The results of the experiments are shown in Table 6. The experimental results show that the overall performance of the six tasks is lower than that of the STMSA. The performance of Task A and Task B is significantly lower than that of other translation tasks, indicating that the bidirectional translation structure plays a crucial role in integrating complementary information from the three modalities. Furthermore, with increasing missing rates, Task A and Task B exhibit a steeper decline in performance. This occurs because when modalities are missing, the model lacks a bidirectional translation structure to reconstruct missing modal information when the translation chain breaks. Consequently, Task A and Task B perform poorly under high missing rate conditions. Additionally, When the missing rate is set to 0.3, the M-F1 score and ACC value of Task C are only 52.60% and 77.60%, respectively. This indicates that different translation orders produce different results in environments where modalities are missing. Poor results affect the effectiveness of features fusion, thereby reducing classification accuracy. We analyze that in Task A, high-quality text modality is not introduced into the fusion process, resulting in overall performance lower than STMSA. In Task B, although text modality is introduced to enhance the fusion process, the translation process lacks the introduction of a third modality, thereby limiting the translation performance. Task D involves translating text into visual and audio modalities. Experimental results show that its performance is lower than that of STMSA. In STMSA, the translation outputs establish direct interactions between text modality, visual modality, and audio modality. According to Liu et al.11, text modality exhibits high-quality emotion recognition capabilities. Therefore, the STMSA, which has two modalities that interact directly with text, exhibits the best performance. Additionally, as observed in the experimental results of Table 6, the performance of Tasks E and F falls below that of STMSA. This indicates that when the text modality occupies either the starting or ending position in bidirectional translation, it fails to fully leverage the semantic guidance provided by the text. Consequently, visual and audio features cannot effectively undergo semantic calibration with the text. This fully validates the effectiveness of the design principle where the text modality serves as the dominant factor.Table 6Experimental results of STMSA under different translation orders and translation structures.Models00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCTask A47.4072.3943.2870.7742.4169.5441.8670.1936.4865.1236.4965.57Task B50.8775.0048.3673.2550.9575.4440.6769.0139.4568.7641.5969.34Task C57.0584.3755.2981.7754.3680.2154.6980.7354.0279.6853.6479.16Task D56.5783.8557.0784.3753.1478.6454.8581.2552.2777.0852.9778.12Task E55.7182.2955.2781.7752.7578.1252.6077.6052.4377.6052.3377.08Task F56.8683.8554.2180.2155.0681.2556.3883.3353.0578.6453.3878.64STMSA58.6586.4658.5285.9456.8483.8556.8183.8554.4380.2154.0879.69 Table 6Experimental results of STMSA under different translation orders and translation structures.Models00.10.20.30.40.5M-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCM-F1ACCTask A47.4072.3943.2870.7742.4169.5441.8670.1936.4865.1236.4965.57Task B50.8775.0048.3673.2550.9575.4440.6769.0139.4568.7641.5969.34Task C57.0584.3755.2981.7754.3680.2154.6980.7354.0279.6853.6479.16Task D56.5783.8557.0784.3753.1478.6454.8581.2552.2777.0852.9778.12Task E55.7182.2955.2781.7752.7578.1252.6077.6052.4377.6052.3377.08Task F56.8683.8554.2180.2155.0681.2556.3883.3353.0578.6453.3878.64STMSA58.6586.4658.5285.9456.8483.8556.8183.8554.4380.2154.0879.69 Experimental results of STMSA under different translation orders and translation structures. Analysis of the dynamic changes in training loss Our pre-training strategy is conducted on a full-modal basis, aiming to enable the model to learn the complete modal feature distribution and enhance its ability to handle scenarios with missing modalities. The training process of STMSA can be divided into two stages. In the first stage, STMSA is trained directly on the full modality to obtain a model with relatively good training performance. In the second stage, the pre-training strategy is introduced to continue training on the full modality to obtain the final model. The loss function is one of the key factors guiding the model to converge toward the target value. Therefore, we conducted tests on the CMU-MOSI dataset, obtaining a dynamic curve of STMSA’s loss values as training steps increased, as shown in Fig. 7. Figure 7 reveals that the loss curve exhibits an overall downward trend. The initial total loss value is 26.1797, indicating a high loss value. This occurs because the training parameters initially learned from the dataset are not yet optimized for the corresponding data distribution. As training progresses, the loss value gradually decreases, ultimately reaching 5.16331. The initial value of the pre-training loss is 4.2614, which eventually reaches 0.82263 during training. These experimental results demonstrate that STMSA can converge to a stable optimal state.Fig. 7The change of the loss values in training of STMSA. The change of the loss values in training of STMSA. Qualitative analysis: Although STMSA employs multiple loss terms (e.g., pre-training loss, translation loss, decoder loss, classification loss, etc.), its loss value steadily decreases throughout training, demonstrating STMSA’s effective handling of these loss components. Furthermore, the absence of abrupt peaks or troughs in the loss curve indicates that STMSA avoids issues like gradient explosion or vanishing gradients. Overall, the loss curve’s gradual and steady decline reflects well-maintained gradients, indicating STMSA’s stable learning capability. However, some fluctuations in loss values are unavoidable during certain training steps in total loss–such as steps 0–60, 180–240, 420–480, and 660–720. This occurs because each batch of data is randomly shuffled, potentially leading to variations in data distribution and subsequent loss fluctuations. In fact, these fluctuations do not compromise STMSA’s overall performance. Ultimately, the continuous decline and eventual convergence of STMSA’s loss value demonstrate its stability and effectiveness. Moreover, minor fluctuations in the loss value can sometimes help STMSA escape local optima to learn global target features, thereby achieving high-precision classification. Additional analysis On the CMU-MOSI dataset, we conducted a binary classification experiment targeting positive, neutral, and negative sentiment categories. In addition to the M-F1 and ACC metrics, we introduced two new evaluation measures–the Matthews Correlation Coefficient (MCC) and Mean Absolute Error (MAE)–to comprehensively assess the model’s performance across different sentiment categories. The experimental results are shown in Table 7. The results indicate that the model exhibits relatively weaker recognition performance for the neutral sentiment category. We analyze that this may stem from the inherently limited emotional polarity information within neutral samples. This scarcity hinders the model’s ability to extract sufficiently discriminative signals during feature learning and decision boundary delineation, consequently impairing classification accuracy. This observation also suggests that future work could focus on designing more discriminative feature representations or incorporating external knowledge to enhance the classification of samples with ambiguous emotional expressions or weak polarity.Table 7Experimental results of the model on the CMU-MOSI dataset under positive, neutral, and negative conditions.DatasetPer-classM-F1ACCCorrMAECMU-MOSINeutral81.1081.250.6310.187Negative82.7382.810.6680.172Positive83.8483.850.6310.161Table 8Experimental results on model computational complexity.DatasetTesting timeTesting memoryTraining timeTraining memoryFLOPsCMU-MOSI0.06730 s31.42 MB47.03 s14238.3 MB31.72 billion Table 7Experimental results of the model on the CMU-MOSI dataset under positive, neutral, and negative conditions.DatasetPer-classM-F1ACCCorrMAECMU-MOSINeutral81.1081.250.6310.187Negative82.7382.810.6680.172Positive83.8483.850.6310.161 Experimental results of the model on the CMU-MOSI dataset under positive, neutral, and negative conditions. Table 8Experimental results on model computational complexity.DatasetTesting timeTesting memoryTraining timeTraining memoryFLOPsCMU-MOSI0.06730 s31.42 MB47.03 s14238.3 MB31.72 billion Experimental results on model computational complexity. Additionally, we tested the computational complexity of the model, with experimental results shown in Table 8. Our model was trained on a computer with the following configuration: Windows 10 operating system, Intel®Core\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{TM}$$\end{document} i9-10900K CPU, NVIDIA RTX 3090 GPU, and 96 GB of memory. The model was implemented using TensorFlow version 1.14.0 and Python 3.7, and it was run on the CPU rather than the GPU to ensure its adaptability to a broader range of real-world application scenarios. In Table 8, Testing Time represents the testing runtime for a single sample, Testing Memory denotes the memory usage during testing a single sample, Training Time indicates the time consumed per training epoch, Training Memory reflects the memory usage per training epoch, and FLOPs signifies floating-point operations per second. Experimental results show that the model’s memory usage per training epoch reached 14,238.3 MB. Although this memory consumption is relatively high, it remains within acceptable limits for practical deployment. Comprehensive analysis indicates that the current memory and time overhead primarily stem from the sequential translation structure and multimodal fusion mechanism designed to handle uncertain modal missing data. Through this design, STMSA achieves unified handling of diverse missing scenarios without requiring specific completion strategies, and its overall complexity remains competitive in practical application scenarios. Conclusion The study of uncertain missing modalities has become one of the key challenges in the field of multimodal sentiment analysis. To address this issue, we propose a sequential translation-based multimodal sentiment analysis model (STMSA). Our model facilitates interactions between three modalities through sequential translation and uses a common space to obtain complete fusion features. These features are then used for final sentiment classification. The aim of sequential translation methods is to extend the interactive processes between two original modalities to three. Additionally, when there are modalities missing, translation methods involving only two modalities become inefficient due to dependencies between modalities. Sequential translation methods can improve this dependency problem by introducing a third modality. Based on this foundation, we employ a bidirectional translation strategy to establish information interactions between text modality and the other two modalities. Additionally, we utilize pre-trained models to guide the joint representation of common space to approximate the joint distribution of all modalities. STMSA has no specific missing modalities completion strategy, as it aims to reduce model complexity and explore a unified fusion architecture for dealing with missing data. We conducted extensive experiments to demonstrate the performance of STMSA under different modal missing rates. The experimental results indicate that STMSA has certain advantages. Therefore, STMSA offers a new approach to addressing uncertain missing modalities. However, the model still has several limitations. First, it exhibits weak recognition capabilities for neutral sentiment categories, stemming from the ambiguous sentiment polarity and insufficient feature distinctiveness of such samples. Second, its high computational resource consumption limits deployment in resource-constrained environments such as mobile devices. Future work will focus on model lightweighting and neutral sentiment enhancement to advance this model toward practical applications.