FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter
Junxi Wang, Yaxiong Wang, Lechao Cheng, Zhun ZhongHefei University of Technology, China [email protected]@hfut.edu.cn , [email protected]@gmail.com · 2025
Read FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter by Junxi Wang, Yaxiong Wang, Lechao Cheng, Zhun ZhongHefei University of Technology, China [email protected]@hfut.edu.cn , [email protected]@gmail.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19639.txt.
Introduction
Junxi Wang and co-authors adapt in this 2025 paper a large vision-language model into a fake news hunter for short videos. False clips endanger public trust, and older detectors lack the world knowledge to check claims. Their system sorts clips into four cases by whether the footage and the text are real or fake, and fields expert modules for each.
One adapter runs detection experts first, then attribution experts for a fuller verdict. A second module hunts for clashes between what is seen and what is said. On two benchmark sets the system beat prior art by three to five points, setting fresh marks. Platforms could run the check before clips spread.
Catalog record
| CREATOR | Junxi Wang, Yaxiong Wang, Lechao Cheng, Zhun ZhongHefei University of Technology, China [email protected]@hfut.edu.cn , [email protected]@gmail.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter Junxi Wang, Yaxiong Wang, Lechao Cheng, Zhun ZhongHefei University of Technology, China [email protected]@hfut.edu.cn , [email protected]@gmail.com Abstract We present FakeSV-VLM in this paper, a new VLM-based framework for detecting fake news on short video platforms. Despite significant efforts to combat this issue due to the severe threat that fake news videos pose to public information security, existing methods still fall short in detection accuracy, often due to lack of knowledge to verify the news is real or not. However, large Vision Language Models (VLMs) have absorbed extensive real-world knowledge from massive multimodal datasets. Motivated by this, we adapt advanced VLMs for fake news detection in short videos. Upon close examination of news samples, we observe that short video samples can be categorized into four distinct scenarios: both video and text are real (for real samples), or both are fake, or either the video or text is fake (for fake samples). Inspired by this insight, we design four experts tailored to handle each scenario and integrate them into VLM via Mixture of Experts. Specifically, we develop the Progressive MoE Adapter (PMOE) module where detection experts first provide an initial analysis, followed by attribution experts for a comprehensive diagnosis, leading to a robust decision. Additionally, we also note the fake news videos often show inconsistency between two modalities. Consequently, we further design the Alignment-driven Event Checking (ADEC) module, which perceives the fake news by capturing the inconsistency between different modalities. Extensive experiments on two benchmark datasets, FakeSV and FakeTT, verify the superiority of our model. It significantly outperforms current state-of-the-art models by +3.32% and +5.02%, establishing a new benchmark in the field. Our code is available at https://github.com/Celina-love-sweet/FakeSV-VLM. 1 Introduction With the rapid development of short video platforms, the spread of fake news videos on these platforms has posed significant risks to society, covering key areas such as politics Wittenberg et al. 2021 , economics Buntić et al. 2020 , and more. According to relevant statistics, over 500 hours of videos are uploaded to YouTube every minute Venkatagiri et al. 2023 . Relying entirely on manual verification of the authenticity of news videos would consume a huge amount of human resources. To address this challenge, some methods for detecting fake news videos have emerged in the early stages Zampoglou et al. 2019 ; Sitara and Mehtre 2017 ; Abbasi Aghamaleki and Behrad 2017 . However, with the continuous evolution of video editing and synthesis technologies, manipulating and fabricating news videos has become increasingly easier Niu et al. 2023 . Moreover, the ongoing development of large language models further enhances the ability to fabricate fake news Vykopal et al. 2023 ; Wu et al. 2024 , making detection even more difficult and posing an unprecedented threat to public information security. To address this challenge, several methods have been proposed. FakingRecipe Bu et al. 2024 predicts video authenticity by fusing visual, textual, and audio features, while NEED Qi et al. 2023b leverages graph neural networks to analyze relationships among videos from the same event. Although effective, these approaches often rely heavily on cross-modal fusion and are constrained by limited training data, lacking the ability to fully utilize real-world knowledge. Some methods incorporate knowledge from Vision Language Models (VLMs)—for instance, CA-FVD Wang et al. 2025 detects modality alignment using VLMs, and ExMRD Hong et al. 2025 distills knowledge through designed reasoning chains—but they typically depend on VLMs to pre-generate text, limiting scalability and practical deployment. To address the aforementioned issues, we propose directly using VLMs to predict fake news videos. Pre-trained VLMs contain real-world knowledge, which can be fully utilized during the detection process to infer the authenticity of news videos. However, since VLMs are not specifically designed for the task of predicting fake news videos, applying them directly for detection often does not yield ideal results. Based on our observations, news videos can be classified into two broad categories: real and fake. At a more fine-grained level, fake news videos can be subdivided further into content forgery, description forgery, and full forgery of both. To enhance the capability of VLMs in modeling hierarchical forgery patterns in fake news videos, we propose the Progressive Mixture-Of-Experts Adapter (PMOE) module. Specifically, a set of learnable Artifact Tokens is introduced to aggregate potential manipulation cues, which are concatenated with multimodal features and fed into the early layers of the LLM. In the intermediate layers, the contextualized Artifact Tokens are extracted for two-stage reasoning: the Detection MoE estimates the overall authenticity of the video, and the Attribution MoE infers the specific manipulation type, such as visual tampering, textual misrepresentation, or cross-modal forgery. This enables fine-grained identification of diverse forgery strategies. Given the frequent semantic inconsistencies between visual content and textual descriptions in fake news videos, we propose the Alignment-driven Event Checking (ADEC) module to enhance cross-modal event-level consistency verification. ADEC first extracts intermediate textual representations from the LLM and applies average pooling to obtain a compact event-centric semantic embedding. This embedding is then compared with the visual representation using a contrastive learning objective. Supervised by ground-truth labels, the model is encouraged to align cross-modal semantics and detect event-level discrepancies indicative of manipulation. Based on the above design, we propose the FakeSV-VLM framework to advance multimodal fake news videos detection. The main contributions of this work are summarized as follows: \bullet We integrate VLMs into the training process and directly apply them to fake news videos detection tasks, driving the development of this field. \bullet We design the PMOE and ADEC modules to enable robust manipulation reasoning and event-level cross-modal consistency checking, thereby effectively enhancing the VLM’s capacity for fake news videos detection. \bullet The FakeSV-VLM framework is end-to-end, highly generalizable, and achieves remarkable results, far surpassing the existing SOTA methods. 2 Methodology Figure 1: Illustration of the proposed FakeSV-VLM framework, which is built upon a Vision Language Model (VLM). The model is trained with the classification loss, together with objectives from PMOE and ADEC modules. 2.1 Overview Our FakeSV-VLM architecture is shown in Figure 1 . During training, the model takes as input keyframes and textual descriptions, which are encoded by the visual encoder and tokenizer of the VLM. These multimodal features are concatenated with learnable Artifact Tokens and passed into the early layers of the LLM. The updated Artifact Tokens are then fed into the Progressive MoE Adapter (PMOE) for two-stage reasoning: overall authenticity prediction and manipulation type attribution. Meanwhile, the Alignment-driven Event Checking (ADEC) module aligns semantic information across modalities via contrastive learning. Ultimately, the entire model is optimized using the final classification loss, together with the losses from the PMOE and ADEC modules. 2.2 Event-Prompting Template Design Prompt. To incorporate role-specific and contextual information into the Vision Language Model (VLM), we begin by designing a prompt grounded in prior work. Specifically, we refer to the prompting format used in the comparative experiments of Bu et al. 2024 , which serves as a foundation for our design. Additional details are provided in Appendix B . The initial system prompt is as follows: You are an experienced news video fact-checking assistant, maintaining a neutral and objective stance at all times. You are capable of handling various types