An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis
Phuong Q. Dao, Mark Roantree, Vuong M. Ngo · 2025
Read An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis by Phuong Q. Dao, Mark Roantree, Vuong M. Ngo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23617.txt.
Introduction
Phuong Q. Dao, Mark Roantree, and Vuong M. Ngo work on multimodal sentiment analysis, reading human emotion from text and images together rather than either alone. Two signals usually say more than one, and transformer models give strong tools for each side: BERT for text and the Vision Transformer for pictures. The authors first fuse the two early, so the streams shape each other from the start.
Their extended model, the Dual Transformer Contrastive Network, adds another transformer layer to sharpen the text before fusion and uses contrastive learning to pull matching text and image features closer. On the TumEmo benchmark it reached the best accuracy, about seventy-eight percent, with a matching F1 score. On MVSA-Single it stayed competitive at roughly seventy-seven percent accuracy. The results back early fusion plus deeper context as a sound recipe for emotion reading.
Catalog record
| CREATOR | Phuong Q. Dao, Mark Roantree, Vuong M. Ngo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis Phuong Q. Dao, Mark Roantree, Vuong M. Ngo Abstract Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by jointly analyzing data from multiple modalities—typically text and images—offering a richer and more accurate interpretation than unimodal approaches. In this paper, we first propose BERT-ViT-EF, a novel model that combines powerful Transformer-based encoders—BERT for textual input and ViT for visual input—through an early fusion strategy. This approach facilitates deeper cross-modal interactions and more effective joint representation learning. To further enhance the model’s capability, we propose an extension called the Dual Transformer Contrastive Network (DTCN), which builds upon BERT-ViT-EF. DTCN incorporates an additional Transformer encoder layer after BERT to refine textual context (before fusion) and employs contrastive learning to align text and image representations, fostering robust multimodal feature learning. Empirical results on two widely used MSA benchmarks—MVSA-Single and TumEmo—demonstrate the effectiveness of our approach. DTCN achieves best accuracy (78.4%) and F1-score (78.3%) on TumEmo, and delivers competitive performance on MVSA-Single, with 76.6% accuracy and 75.9% F1-score. These improvements highlight the benefits of early fusion and deeper contextual modeling in Transformer-based multimodal sentiment analysis. 1 Introduction In the age of social media and ubiquitous user-generated content, understanding human sentiment has become increasingly important. As a result, Multimodal Sentiment Analysis (MSA) has emerged as a critical area in natural language processing and computer vision. Unlike traditional text-only approaches, MSA leverages complementary signals across modalities to better capture subtle emotional expressions, sarcasm, or visual context that may not be apparent in language alone. This capability is particularly vital in domains such as social media monitoring, customer feedback analysis, and affective computing, where emotions are inherently multimodal. Consequently, developing effective fusion strategies and modality-aware representations is essential for advancing sentiment understanding in real-world, noisy, and richly contextual environments. Unlike traditional sentiment analysis, which usually relies on textual information alone, MSA incorporates multiple sources of information, including text and images, to provide a richer and more robust understanding of emotional content. Early approaches to MSA were often based on shallow feature engineering or classical machine learning techniques, treating each modality in isolation [ 21 ] . While these methods offered initial insights, they lacked the capacity to capture the deep interdependencies between modalities. With the advent of deep learning, especially Transformer-based architectures, the field has witnessed a significant leap forward [ 28 , 31 ] . Models such as BERT [ 8 , 14 ] have revolutionized textual understanding by capturing contextual nuances at the word and sentence levels. Similarly, the introduction of the Vision Transformer (ViT) [ 12 ] has made it possible to learn hierarchical visual features without relying on convolutional operations. However, integrating these powerful unimodal encoders into a unified multimodal framework remains a challenge. Most existing works adopt a late fusion strategy, where modality-specific predictions are combined at the decision level. Although straightforward, this approach often fails to fully exploit cross-modal interactions [ 13 ] . To address this limitation, recent studies have explored early fusion techniques, where representations from different modalities are integrated prior to classification [ 2 ] . Early fusion has been shown to enhance performance by enabling deeper semantic alignment between text and visual content [ 11 , 23 ] . Furthermore, dual-branch or two-stream architectures have proven effective in retaining the integrity of modality-specific encoders while allowing shared representation learning [ 9 ] . In this paper, we present a Dual Transformer Contrastive Network (DTCN) model that combines BERT for text and ViT for images within an early fusion framework. By integrating the [CLS] tokens from each modality at the representation level, our model encourages interaction between modalities at a deeper level. DTCN is evaluated on two widely used benchmark datasets: MVSA-Single and TumEmo, which are standard for assessing MSA systems [ 5 , 13 , 20 , 24 , 28 , 30 , 32 , 33 ] . Our main contributions are summarized as follows: • We empirically demonstrate that early fusion outperforms late fusion on two benchmark datasets. • BERT-ViT-EF, integrating BERT and ViT using early fusion, is proposed to enable joint cross-modal learning and demonstrates strong performance on MSA tasks. BERT captures contextual textual features, while ViT provides rich visual representations. • DTCN, an enhanced architecture built upon BERT-ViT-EF, incorporates an additional Transformer encoder layer after BERT (before fusion) and employs contrastive learning. This design fosters deeper cross-modal interaction and enhances joint representation learning. • Our models are evaluated and compared against both classical and Transformer-based MSA models on the TumEmo and MVSA-Single datasets, achieving state-of-the-art performance on TumEmo and competitive results on MVSA-Single. The rest of this paper is organized as follows. Section 2 reviews relevant research in MSA, focusing on Transformer-based architectures and fusion strategies. Section 3 presents our proposed Dual Transformer Contrastive Network, detailing the architecture, early fusion mechanism, and training objectives. Section 4 describes the datasets, experimental setup, and implementation details. Section 5 reports and analyzes experimental results, including comparisons with existing baselines. Finally, Section 6 concludes the paper and discusses directions for future work. 2 Related Work 2.1 Deep learning and Fusion Strategies for MSA MSA combines textual and visual information to better capture emotional signals in user-generated content. Traditional methods often relied on handcrafted features and classical machine learning algorithms [ 21 ] , which lacked the capacity to model complex interactions between modalities. Deep learning approaches marked a significant turning point in MSA. Models based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM), such as Multi-view Attentional Network (MVAN) [ 28 ] and Sentiment Knowledge Enhanced Attention Fusion Network (SKEAFN) [ 31 ] , incorporated attention mechanisms to better align textual and visual information. However, these models primarily used static embeddings (e.g., GloVe) and ResNet for image features, limiting their contextual understanding. Fusion strategy remains a critical design factor in MSA [ 7 ] . While many systems adopt late fusion for simplicity, studies like CLMLF [ 13 ] show that joint representation learning via carefully designed loss functions can yield better results. However, CLMLF still combines modalities at a late stage, limiting early interaction. In contrast, early fusion enables joint learning from the start. Intermediate fusion offers a middle ground by integrating modalities at multiple stages after initial unimodal encoding, aiming to balance specialization and interaction. 2.2 Classical MSA Models HSAN-M [ 25 ] is a hierarchical semantic attentional network for MSA. It leverages image captions as semantic information and uses attention mechanisms to extract and combine features from different modalities. It aims to capture complementary information by focusing on salient parts of each modality. MultiSentiNet-M [ 26 ] is a deep semantic network that extracts deep semantic features from images using salient detectors and employs a visual feature-guided attention LSTM model to extract important words for sentiment analysis. This model highlights the importance of both visual and textual content for understanding human sentiments. It also leverages scene and object features. Co-Memory-M [ 27 ] utilizes co-memory networks to iteratively model mutual influences between image and text data for MSA. It explores the interrelations between image and text to achieve a more comprehensive understanding of sentiment. Se-MLNN [ 4 ] is a model that integrates