Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities
Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen ✉, Zhiyong Wang, Tat-Seng Chua · 2026
Read Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities by Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen ✉, Zhiyong Wang, Tat-Seng Chua (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07763.txt.
Introduction
Dou and five colleagues reframe multimodal deepfake detection around shared forgery knowledge instead of surface clues. Current forensic tools lean on superficial artifacts tied to single modes and collapse when facing unseen dark modes. Their modality-agnostic framework strips away style quirks tied to one mode to extract the essential cross-mode latent forgery signal. The shift moves the field from feature fusion toward mode generalization. Tests show the approach holds up where artifact-chasing detectors degrade sharply.
Catalog record
| CREATOR | Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen ✉, Zhiyong Wang, Tat-Seng Chua |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121181 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07763
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen ✉, Zhiyong Wang, Tat-Seng Chua Abstract As generative artificial intelligence evolves, deepfake attacks have escalated from single-modality manipulations to complex, multimodal threats. Existing forensic techniques face a severe generalization bottleneck: by relying excessively on superficial, modality-specific artifacts, they neglect the shared latent forgery knowledge hidden beneath variable physical appearances. Consequently, these models suffer catastrophic performance degradation when confronted with unseen ”dark modalities.” To break this limitation, this paper introduces a paradigm shift that redefines multimodal forensics from conventional ”feature fusion” to ”modality generalization.” We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge. Furthermore, we define two progressive dimensions to quantify model generalization: transferability toward semantically correlated modalities (Weak MAF), and robustness against completely isolated signals of ”dark modality” (Strong MAF). To rigorously assess these generalization limits, we introduce the DeepModal-Bench benchmark, which integrates diverse multimodal forgery detection algorithms and adapts state-of-the-art generalized learning methods. This study not only empirically proves the existence of universal forgery traces but also achieves significant performance breakthroughs on unknown modalities via the MAF framework, offering a pioneering technical pathway for universal multimodal defense. I Introduction With the explosive evolution of generative AI, deepfake attacks have transitioned from single-modality [ 11 , 41 , 51 , 36 ] visual manipulation to highly coordinated, multimodal deceptions [ 59 , 22 , 25 , 16 ] . Modern forgery techniques can now synthesize highly realistic video, audio, and associated text simultaneously. This cross-modal consistency not only enhances deceptiveness but also poses a severe challenge to traditional forensic technologies. To counter this, mainstream multimodal defenses have relied heavily on feature fusion strategies. However, as illustrated in Fig. 1 , these traditional paradigms operate under a strict ”closed-modality” mindset, assuming test modalities must be seen during the training phase. Consequently, they suffer from a severe ”modality-binding” bottleneck. When confronted with the continuous emergence of novel media formats in the real world, such as infrared signals, depth streams, or heterogeneous unstructured data, these models exhibit catastrophic performance degradation because they cannot detach from the specific physical representations they have memorized. This vulnerability exposes a fundamental limitation: existing methods [ 26 , 67 , 63 ] merely overfit to the surface-level artifacts of specific known modalities, failing entirely to capture the underlying essence of generative algorithms. To break this impasse, we argue that the strategic focus of multimodal forgery detection must undergo a paradigm shift, from modality-binding feature fusion to modality generalization. We propose a bold hypothesis: although the physical manifestations of forgery vary drastically across different media, there exists a shared latent forgery knowledge region that transcends these modal boundaries. Capturing this inherent statistical bias, which is left behind by generative models regardless of the output medium, is the key to achieving a universal, ”train once, generalize across all modalities” defense mechanism. Fig. 1: From modality-binding to modality-agnostic forgery (MAF). Unlike existing tasks that overfit to specific physical representations seen during training, our proposed MAF task focuses on extracting shared latent forgery knowledge. This modality-agnostic feature learning enables robust source-free domain generalization against unknown modalities (e.g., unseen audio), systematically tackling both Weak and Strong MAF challenges. Based on this hypothesis, we formally introduce the modality-agnostic forgery (MAF) task and detection framework. As depicted in Fig. 1 , MAF is designed to decouple modality-specific style noise, precisely extracting the essential, cross-modal latent forgery features required to maintain robust discriminative power on entirely unseen test modalities. To comprehensively assess this capability under varying real-world perceptual constraints, we establish two progressive evaluation paradigms. Weak MAF evaluates generalization toward unseen modalities that can still be mapped into a unified semantic space via existing pre-trained perceptors (e.g., ImageBind [ 15 ] or LanguageBind [ 68 ] ), verifying whether a model can achieve ”forensic fingerprint” alignment beneath macro-semantic alignment. In contrast, Strong MAF tackles the rigorous ”perception island” challenge: defending against completely isolated ”dark modalities” that are neither seen during training nor semantically alignable by existing models. This strict setting forces the framework to capture the inherent statistical flaws of generative AI purely from the underlying logic level, achieving higher-dimensional robustness without the prerequisite of shared representations. To systematically evaluate and drive forward this modality-agnostic research trajectory, we construct DeepModal-Bench, the first comprehensive benchmark specifically tailored for modality generalization in forgery detection. Integrating diverse state-of-the-art multimodal algorithms and adapting advanced domain generalization (DG) strategies, our extensive experiments across multiple large-scale public datasets provide compelling empirical evidence of universal forgery traces that transcend physical representations. In summary, the main contributions of this work are threefold: • We redefine the objective of multimodal forensics, shifting the paradigm from modality-binding feature fusion to modality generalization by proposing the concept of shared latent forgery knowledge. • We introduce the MAF framework and formulate two rigorous evaluation settings (Weak MAF and Strong MAF) to systematically tackle both semantically correlated unseen modalities and physically isolated ”dark modalities.” • We release DeepModal-Bench, the first comprehensive benchmark for this task. Our extensive evaluations not only validate the objective existence of universal forgery traces but also illuminate the critical pathway from semantic alignment to true forensic alignment, offering a pioneering technical route for universal multimodal defense. II Related Work Modality Binding. Multimodal learning has rapidly evolved from simple pairwise alignment toward unified modality binding. While CLIP [ 42 ] laid the foundational paradigm, advanced models such as ImageBind [ 15 ] and its successors [ 68 , 48 , 53 , 17 , 35 ] have successfully anchored diverse modalities within a shared semantic embedding space. To further enhance binding quality and system robustness, researchers have actively explored optimization pathways ranging from feature decoupling and geometric constraints [ 46 , 62 , 28 , 40 , 19 , 12 , 34 , 56 ] to dynamic modality selection. However, despite these architectural leaps, existing binding mechanisms remain fundamentally tethered to the physical representations seen during training. When encountering entirely unseen or isolated ”dark modalities,” their performance degrades catastrophically. This persistent ”modality-binding” bottleneck serves as the theoretical motivation for our modality-agnostic forgery (MAF) framework and the DeepModal-Bench benchmark, driving the shift toward truly modality-agnostic detection. Forgery Detection Driven by the surge of multimodal generative technologies, forgery detection is undergoing a crucial paradigm shift from shallow ”perceptual classification” to ”deep semantic understanding.” This evolution is heavily supported by diverse large-scale datasets [ 65 , 45 , 8 , 20 , 6 , 24 ] and advanced frameworks like LAV-DF [ 7 ] and HAMMER [ 45 ] , which utilize multi-task prompt learning for precise manipulation localization. Furthermore, leveraging large multimodal models (LMMs), next-generation architectures construct deep forgery knowledge spaces to defend against unknown diffusion models [ 47 , 20 ] , while others integrate LLMs [ 59 , 58 , 16 ] for interpretable reasoning. However, these advanced methodologies rely excessively on explicit ”modality alignment”, inadvertently creating a severe modality-binding problem. Confronted with unseen ”dark modalities” (e.g., infrared or depth streams), these models fail to decouple the essence of forgery from physical representations. To break this impasse, we introduce the MAF framework, shifting the