Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs
Yijie Jin, Junjie Peng, Xuanchao Lin, Haochen Yuan, Lan Wang, Cangzhi Zheng {jyj2431567, jjie.peng, linxuanchao, yuanhc, Wanglan1997, cangzhizheng}@shu.edu.cn · 2025
Read Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs by Yijie Jin, Junjie Peng, Xuanchao Lin, Haochen Yuan, Lan Wang, Cangzhi Zheng {jyj2431567, jjie.peng, linxuanchao, yuanhc, Wanglan1997, cangzhizheng}@shu.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01068.txt.
Introduction
Yijie Jin and co-authors present a 2025 result on how multimodal transformers work, with an eye to speed. Such models fuse text, sound, and image traits to judge sentiment, but they cost many parameters. The team proves the models match a kind of layered graph in structure.
From that match they build a masked design that shares weights with no loss of order in the data. The new model fuses all modes at once with a third of the parameters of the plain form. Tests on sentiment sets show higher scores, and gains held when the design was added to top models.
Catalog record
| CREATOR | Yijie Jin, Junjie Peng, Xuanchao Lin, Haochen Yuan, Lan Wang, Cangzhi Zheng {jyj2431567, jjie.peng, linxuanchao, yuanhc, Wanglan1997, cangzhizheng}@shu.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01068
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs Yijie Jin, Junjie Peng, Xuanchao Lin, Haochen Yuan, Lan Wang, Cangzhi Zheng {jyj2431567, jjie.peng, linxuanchao, yuanhc, Wanglan1997, cangzhizheng}@shu.edu.cn Abstract Multimodal Sentiment Analysis (MSA) is a rapidly developing field that integrates multimodal information to recognize sentiments, and existing models have made significant progress in this area. The central challenge in MSA is multimodal fusion, which is predominantly addressed by Multimodal Transformers (MulTs). Although act as the paradigm, MulTs suffer from efficiency concerns. In this work, from the perspective of efficiency optimization, we propose and prove that MulTs are hierarchical modal-wise heterogeneous graphs (HMHGs), and we introduce the graph-structured representation pattern of MulTs. Based on this pattern, we propose an Interlaced Mask (IM) mechanism to design the Graph-Structured and Interlaced-Masked Multimodal Transformer (GsiT). It is formally equivalent to MulTs which achieves an efficient weight-sharing mechanism without information disorder through IM, enabling All-Modal-In-One fusion with only 1/3 of the parameters of pure MulTs. A Triton kernel called Decomposition is implemented to ensure avoiding additional computational overhead. Moreover, it achieves significantly higher performance than traditional MulTs. To further validate the effectiveness of GsiT itself and the HMHG concept, we integrate them into multiple state-of-the-art models and demonstrate notable performance improvements and parameter reduction on widely used MSA datasets. 1 Introduction With the growing ubiquity of diverse social media platforms such as YouTube and TikTok, users now express sentiments through various forms of information, including text, video, and audio. To achieve more natural human-computer interactions, multimodal sentiment analysis (MSA) has become a popular research area ( Gandhi et al. 2023 ) . MSA is briefly exemplified in Figure 1 . Figure 1: An example of multimodal sentiment analysis: end-to-end discriminative task pipeline. Figure 2: Graph structure comparison. A: Naive graph structure constructed by concatenated multimodal sequences. B: Forest structure of MulTs, constructed by decoupled bi-modality combinations. C: Tree structure of GsiT, constructed by concatenated multimodal sequences and IM machanism. The main challenge of MSA is to integrate heterogeneous data containing different sentiment information, thus achieving effective sentiment analysis. The practical manifestations of these challenges primarily lie in the performance of multimodal fusion methods, the representation capacity of multimodal features, and the robustness of the model. To address these challenges, methods of MSA involve designing effective multimodal fusion methods ( Zadeh et al. 2017 ; Tsai et al. 2019a ; Zhang et al. 2023 ; Zheng et al. 2024 ) to fully integrate heterogeneous data, and developing representation learning-based methods ( Yu et al. 2021 ; Yang et al. 2023 ; Lin and Hu 2024 ) to enhance unimodal information and model robustness. Among these, multimodal fusion is the core issue of MSA and also the focus of this paper. In the realm of multimodal fusion, Multimodal Transformer (MulT) ( Tsai et al. 2019a ) and its enhanced successors ( Zhang et al. 2021 ; Zhang et al. 2023 ; Zong et al. 2023 ; Wang et al. 2024 ; Zheng et al. 2024 ; Wu et al. 2024 ) , collectively known as MulTs, have shown significant effectiveness in MSA. Despite their status as the prevailing paradigm, the extensive use of Cross-Modal Attention (CMA) and Multi-Head Self-Attention (MHSA) mechanisms leads to inefficiencies in MulTs. Since MSA is an end-to-end discriminative task, it is imperative to reduce system overhead and improve model performance for the practical implementation of future MSA systems. Thus, the main objective of this work is to introduce a more efficient paradigm for MSA. Additional related works can be found in Appendix G . In this work, from the perspective of efficiency optimization, we discover and prove the theoretical equivalence between CMA/MHSA and Graph Attention Networks (GAT) ( Velickovic et al. 2018 ) , where GAT uses multi-head attention as the aggregation function. Specifically, CMA is equivalent to a unidirectional complete bipartite graph of bi-modality combinations, while MHSA is equivalent to a directed complete graph of uni-modality. Based on this, MulTs can be defined as a forest composed of three independent trees. Each tree is constructed from three subgraphs, with hierarchical relationships constrained by a complex system of multiple functions. This mathematical representation formally defines the theorem that MulTs are hierarchical modal-wise heterogeneous graphs (HMHGs), as shown in Figure 2 . Based on the above theorem, we identify the redundancy in MulTs’ model parameters and their potential for compression while preserving theoretical equivalence. Leveraging this discovery, we propose the Graph-Structured Interlaced-Masked Multimodal Transformer (GsiT) by compressing a forest composed of three independent trees into a single shared tree. GsiT introduces a novel Interlaced Mask (IM) mechanism for multimodal weight sharing, enabling All-Modal-In-One fusion without information disorder. Furthermore, we implement a Triton kernel named Decomposition to maintain efficiency. With only 1/3 of the parameters of traditional MulTs, GsiT maintains theoretical consistency with the MulTs’ paradigm. Comprehensive experimental analysis reveals that GsiT outperforms traditional MulTs significantly in the same experimental setup, boasting a substantial edge in efficiency. To validate the effectiveness and transferability of GsiT, we conducted comprehensive evaluations on the most widely used multimodal sentiment analysis datasets, including CMU-MOSI ( Zadeh et al. 2016 ) , CMU-MOSEI ( Bagher Zadeh et al. 2018 ) , CH-SIMS ( Yu et al. 2020 ) (for multilingual adaptability), and MIntRec ( Zhang et al. 2022 ) (for broader multimodal domains). Our findings show that GsiT not only outperforms as a backbone-level model but also that baseline models incorporating the HMHG concept achieve significant improvements in overall performance. 2 Insights MulTs facilitate multimodal fusion by breaking down multimodal data into pairs of modalities for processing. By creating various combinations of these bi-modality units, MulTs ensure a comprehensive integration of heterogeneous data. This approach can be recognized as a hierarchical and graph-structured fusion method. To better illustrate, we first formally define hierarchical relationships. Definition 1 . Let S and T be two sets, belonging to the domain \mathbb{X} and the range \mathbb{Y} , respectively, i.e., S\subseteq\mathbb{X} and T\subseteq\mathbb{Y} . If there exists a mapping f:\mathbb{X}\to\mathbb{Y} such that for any element s_{i}\in S , its corresponding mapped value f(s_{i})\in T depends on some subset S_{i}\subseteq S , and these dependency relationships can be constructed recursively, then the dependency relationship between S and T is called a hierarchical relationship . Furthermore, this hierarchical relationship can be represented by a directed tree structure , where vertices represent elements in the sets, and edges represent dependency relationships. To better define this type of model, we propose the following theorem: Theorem 1 . Multimodal Transformers are hierarchical modal-wise heterogeneous graphs. The formal theorem and its corresponding proof can be found in Section 3 . Since MSA task systems are end-to-end discriminative systems, we give the following insight. Insight 1 . For MSA systems, the resource savings achieved by designing low-cost, high-performance models, which lead to overall performance improvements, are more significant in some aspects than the accuracy improvements brought by using large models with high representation capacity. 3 Multimodal Transformers as Graphs We first define modality text, vision, and audio as t,v,a , while multimodality as m . Assuming multimodal sequences \mathcal{V}_{u_{1}}\in\mathbb{R}^{T_{u_{1}}\times d_{u_{1}}} , where u_{1}\in{\{m,t,v,a\}} , T_{u_{1}} denotes the temporal dimension (number of vertices), d_{u_{1}} denotes feature dimension. Those sequences are then concatenated into a single