MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection Journal: Information Fusion

Yeganeh Abdollahinejad, Ahmad Mousavi, Naeemul Hassan, Kai Shu, Nathalie Japkowicz, Shahriar Khosravi, Amir Karami · 2026

Read MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection Journal: Information Fusion by Yeganeh Abdollahinejad, Ahmad Mousavi, Naeemul Hassan, Kai Shu, Nathalie Japkowicz, Shahriar Khosravi, Amir Karami (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16172.txt.

Introduction

Abdollahinejad and six colleagues present MOMENTA, a system for spotting misinformation in social-media posts that mix text, images, and changing narratives over time. It uses expert modules for each modality, aligns text and images with two-way attention, and flags disagreements between them, while tracking how stories evolve. The design aims for steadier performance across varied datasets, domains, and narrative settings than static cross-modal methods.

Catalog record

CREATORYeganeh Abdollahinejad, Ahmad Mousavi, Naeemul Hassan, Kai Shu, Nathalie Japkowicz, Shahriar Khosravi, Amir Karami
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58393
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection Journal: Information Fusion Yeganeh Abdollahinejad, Ahmad Mousavi, Naeemul Hassan, Kai Shu, Nathalie Japkowicz, Shahriar Khosravi, Amir Karami Abstract The widespread dissemination of multimodal content on social media has made misinformation detection increasingly challenging, as misleading narratives often arise not only from textual or visual content alone, but also from semantic inconsistencies between modalities and their evolution over time. Existing multimodal misinformation detection methods typically model cross-modal interactions statically and often show limited robustness across heterogeneous datasets, domains, and narrative settings. To address these challenges, we propose MOMENTA, a unified framework for multimodal misinformation detection that captures modality heterogeneity, cross-modal inconsistency, temporal dynamics, and cross-domain generalization within a single architecture. MOMENTA employs modality-specific mixture-of-experts modules to model diverse misinformation patterns, bidirectional co-attention to align textual and visual representations in a shared semantic space, and a discrepancy-aware branch to explicitly capture semantic disagreement between modalities. To model narrative evolution, we introduce an attention-based temporal aggregation mechanism with drift and momentum encoding over overlapping time windows, enabling the framework to capture both short-term fluctuations and longer-term trends in misinformation propagation. In addition, domain-adversarial learning and a prototype memory bank improve domain invariance and stabilize representation learning across datasets. The model is trained using a multi-objective optimization strategy that jointly enforces classification performance, cross-modal alignment, contrastive learning, temporal consistency, and domain robustness. Experiments on four benchmark datasets, Fakeddit , MMCoVaR , Weibo , and XFacta , show that MOMENTA achieves strong and consistent performance across multiple evaluation metrics, including accuracy, F1-score, AUC, and MCC. These findings suggest that integrating expert specialization, discrepancy-aware fusion, temporal reasoning, and domain-invariant representation learning provides an effective direction for real-world multimodal misinformation detection. 1 Introduction The widespread adoption of social media has significantly increased the generation, transmission, and consumption of information. This has created a trade-off between knowledge aggregation and the spread of misinformation 1 . These platforms facilitate the transmission of veridical information, but also accelerate the spread of false or misleading content 8 . The extensive spread of misinformation via these large-scale technologies leads to various moral, social, and epistemic harms, for example, by undermining trust in public institutions. In the health domain, exposure to misleading social media content has been associated with reduced trust in healthcare institutions, lower confidence in professional medical guidance, and weaker adherence to recommended health behaviors 35 , 10 , 3 . In social and political contexts, misinformation has also contributed to distrust in media, government information, and political institutions, further amplifying its spread 16 , 15 . As these examples suggest, addressing misinformation is far from a merely technical issue; it is a social, political, and public health priority. Misinformation detection becomes even more difficult when posts combine textual and visual content. In such cases, effective systems must not only understand each modality independently, but also reason over their semantic interactions 25 , 44 , 28 . Content that appears veridical in isolation may become misleading when paired with unrelated, repurposed, or manipulated visual content. Prior work has shown that misleadingness can emerge from the mismatch between a headline and its accompanying video, underscoring that deceptive intent often arises from cross-modal inconsistency rather than from either modality alone 36 . Earlier approaches frequently processed posts independently or relied on shallow fusion strategies such as simple feature concatenation, limiting their ability to capture semantic disagreement and richer multimodal dependencies. Recent work has increasingly explored foundation-model-based architectures for multimodal misinformation detection. Large pretrained encoders such as XLM-RoBERTa and CLIP provide strong cross-lingual and cross-modal representations for heterogeneous social media data 6 , 27 , 43 . At the same time, Mixture-of-Experts (MoE) architectures have emerged as an effective way to model the heterogeneity of misinformation by routing inputs through specialized expert networks. Frameworks such as MIMoE-FND dynamically model unimodal agreement and cross-modal semantic alignment, while MisD-MoE uses modality-specific experts and adaptive feature selection to reduce redundancy and improve classification robustness 21 , 19 . Despite these advances, most existing approaches still treat cross-modal relationships as largely static and do not explicitly model how misinformation narratives evolve over time. Another major challenge is domain shift. Models trained on one platform or dataset often generalize poorly to new domains with different linguistic styles, visual distributions, topic structures, or cultural contexts. Recent event-centric methods such as E-CaTCH have shown that temporal information, including semantic drift and narrative progression, can substantially improve misinformation detection robustness 25 . However, jointly integrating temporal reasoning, cross-modal alignment, and domain-invariant representation learning remains an underexplored problem. In this work, we propose MOMENTA , a Mixture-of-Experts over Multimodal Embeddings with Neural Temporal Aggregation framework for multimodal misinformation detection. MOMENTA employs modality-specific MoE layers with input-dependent gating to capture diverse misinformation patterns without introducing excessive architectural complexity. Bidirectional co-attention aligns textual and visual representations in a shared semantic space, while an auxiliary discrepancy branch explicitly models semantic inconsistencies between modalities. To represent narrative evolution, we aggregate posts through overlapping temporal windows and encode both drift and momentum. We further incorporate domain-adversarial learning 9 and a prototype memory bank to encourage domain-invariant representations across datasets. The model is trained using a multi-objective optimization strategy that jointly supports classification, cross-modal alignment, contrastive learning, temporal consistency, and domain robustness. Through experiments on Fakeddit, MMCoVaR, Weibo, and XFacta, we show that MOMENTA achieves strong and consistent performance across diverse multimodal misinformation benchmarks. Our main contributions are summarized as follows: 1. We propose MOMENTA, a unified multimodal misinformation detection framework that combines modality-specific expert specialization, cross-modal alignment, discrepancy-aware fusion, temporal aggregation, and domain generalization within a single architecture. 2. We introduce a temporal aggregation mechanism based on overlapping windows with drift and momentum encoding to model the evolution of misinformation narratives over time. 3. We incorporate domain-adversarial learning and a prototype memory bank to encourage domain-invariant representations and improve robustness across heterogeneous datasets. 4. We evaluate MOMENTA on four benchmark datasets and show strong and consistent performance across multiple multimodal misinformation detection settings. 2 Related Work 2.1 Multimodal Misinformation Detection The increasing sophistication of misinformation in online ecosystems has rendered unimodal detection approaches insufficient. Early methods focused on either textual linguistic features or visual artifacts independently, failing to capture the complex interactions between modalities that adversaries exploit. As a result, recent research has shifted toward multimodal frameworks that jointly analyze textual and visual signals to improve detection robustness and generalization 30 , 32 , 41 . Modern multimodal systems typically rely on large-scale pretrained models to encode heterogeneous inputs into semantically meaningful representations. Transformer-based language models such as XLM-RoBERTa 6 and vision-language models such as CLIP 27 have become foundational due to their strong cross-lingual and cross-modal generalization capabilities. These embeddings are subsequently projected into a shared latent space to facilitate interaction across modalities. This projection step is critical, as it enables semantic alignment between heterogeneous inputs rather than serving as a simple dimensionality reduction process 41 , 20 . Despite these advances, multimodal detection remains challenging due to the inherent semantic gap between modalities and the adversarial nature of misinformation, motivating the development of more sophisticated fusion and reasoning mechanisms 25 , 39 . 2.2 Cross-Modal Interaction and Semantic Consistency An important characteristic of many

Read the complete text