CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation Note: This paper was submitted to the 2025 ACM SIGSAC Conference on Computer and Communications Security (CCS’25).

Mazal Bethany, Nishant Vishwamitra, Cho-Yu Jason Chiang, Peyman Najafirad · 2025

Read CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation Note: This paper was submitted to the 2025 ACM SIGSAC Conference on Computer and Communications Security (CCS’25). by Mazal Bethany, Nishant Vishwamitra, Cho-Yu Jason Chiang, Peyman Najafirad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01900.txt.

Introduction

Mazal Bethany, Nishant Vishwamitra, Cho-Yu Jason Chiang, and Peyman Najafirad attack the machines that fact-check claims. Evidence-based misinformation detectors retrieve proof, then compare claim against evidence. Token-level adversarial tricks, built for single classifiers, cannot fool such a two-stage pipeline. Their system, CAMOUFLAGE, rewrites whole claims instead.

Two language model agents cooperate. An attacker rewrites the claim, preserving meaning while bending retrieval or comparison off course. A prompt optimizer studies the failed attempts and sharpens the attacker's instructions for the next round. Only the detector's binary verdict guides the loop, a true black-box setting. Structural and stylistic transformations replace synonym swaps, with the dose adapted to past outcomes. The study exposes how fragile automated fact-checking remains. Meaning-preserving rewrites, it turns out, can walk past the guards.

Catalog record

CREATORMazal Bethany, Nishant Vishwamitra, Cho-Yu Jason Chiang, Peyman Najafirad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation Note: This paper was submitted to the 2025 ACM SIGSAC Conference on Computer and Communications Security (CCS’25). Mazal Bethany, Nishant Vishwamitra, Cho-Yu Jason Chiang, Peyman Najafirad Abstract . Automated evidence-based misinformation detection systems, which evaluate the veracity of short claims against evidence, lack comprehensive analysis of their adversarial vulnerabilities. Existing black-box text-based adversarial attacks are ill-suited for evidence-based misinformation detection systems, as these attacks primarily focus on token-level substitutions involving gradient or logit-based optimization strategies, which are incapable of fooling the multi-component nature of these detection systems. These systems incorporate both retrieval and claim-evidence comparison modules, which requires attacks to break the retrieval of evidence and/or the comparison module so that it draws incorrect inferences. We present CAMOUFLAGE, an iterative, LLM-driven approach that employs a two-agent system, a Prompt Optimization Agent and an Attacker Agent, to create adversarial claim rewritings that manipulate evidence retrieval and mislead claim-evidence comparison, effectively bypassing the system without altering the apparent meaning of the claim. The Attacker Agent produces semantically equivalent rewrites that attempt to mislead detectors, while the Prompt Optimization Agent analyzes failed attack attempts and refines the system prompt of the Attacker to guide subsequent rewrites. This enables larger structural and stylistic transformations of the text rather than token-level substitutions, adapting the magnitude of changes based on previous outcomes. Unlike existing approaches, CAMOUFLAGE optimizes its attack solely based on binary model decisions (true or false) to guide its rewriting process, eliminating the need for classifier logits or extensive querying. We evaluate CAMOUFLAGE on four evidence-based misinformation detection systems, including two recent academic systems and two real-world APIs. Our approach achieves an average attack success rate of 46.92% while preserving textual coherence and semantic equivalence to the original claims, as validated through human evaluation studies. DOI: XXXXXXX.XXXXXXX Conference: 2025 ACM SIGSAC Conference on Computer and Communications Security; October 13–17, 2025; Taipei, Taiwan ISBN: 978-1-4503-XXXX-X/2018/06 884 CCS: Security and privacy Social network security and privacy CCS: Computing methodologies Natural language generation 1. Introduction Misinformation presents a significant threat to information integrity and social discourse in today’s digital landscape ( van der Linden, 2024 ; Vosoughi et al., 2018 ) . The rapid spread of false information online has prompted the development of automated misinformation detection systems ( Shahid et al., 2022 ) . Among these, evidence-based misinformation detection systems have emerged as particularly promising, as they evaluate the veracity of short claims against established facts from trusted knowledge bases or databases, providing more interpretable and trustworthy decisions ( Wu et al., 2022 ) . These systems typically consist of three key components: a retrieval mechanism to gather relevant evidence, an evidence database serving as a repository of verified information, and an evidence comparison mechanism that determines whether the evidence supports or contradicts the claim ( Popat et al., 2018 ; Du et al., 2022 ) . Despite their importance in combating misinformation, the robustness of evidence-based misinformation detection systems has not been comprehensively evaluated against sophisticated adversarial attacks. Existing black-box adversarial attacks on text classifiers face a significant challenge when targeting these systems. These attacks predominantly rely on token-level substitutions ( e.g. , with synonyms ( Jin et al., 2020 ) or character-level perturbations ( Gao et al., 2018 ) ), which struggle to fool evidence-based detection systems due to their multi-component nature, comprising both retrieval and comparison modules which creates additional complexity for attackers. Since the traditional gradient or logit-based optimization strategies used by existing black-box attacks do not directly compromise either the retrieval of evidence by the retrieval module, or the claim-evidence comparisons drawn by the comparison module, they fail to compromise evidence-based detection systems. This architectural complexity means attackers need to manipulate how evidence is retrieved and/or interpreted across multiple system components, rather than simply fool a single classifier. We propose CAMOUFLAGE , ( C laim A lteration for M isleading O utput U sing F eedback from L anguage A gent G uideanc E ), an iterative, LLM-driven approach that employs a two-agent system: an Attacker Agent and a Prompt Optimization Agent to create adversarial claim rewritings. Both agents are powered by LLMs, leveraging their advanced natural language understanding and generation capabilities. The Attacker Agent produces semantically equivalent rewrites that mislead detectors by implementing comprehensive structural and stylistic transformations rather than simple word substitutions, by strategically reformulating sentence structures and introducing stylistic variations that may confuse the components of the evidence-based misinformation detection system. Meanwhile, the Prompt Optimization Agent continuously analyzes the outcomes of previous attack attempts, identifies patterns of success and failure by tracking transformations that successfully fool the detection system while staying within semantic preservation constraints, and uses these insights to dynamically refine the instructions for the Attacker Agent’s subsequent rewriting attempts. By generating claims that are both semantically equivalent to the original claim and strategically misaligned with the detection system’s retrieval and comparison heuristics, CAMOUFLAGE breaks the retrieval of evidence and the comparison process so that it draws incorrect inferences. Unlike existing approaches, CAMOUFLAGE optimizes its attack solely based on binary model decisions to guide its rewriting process, eliminating the need for classifier logits or extensive querying. By implementing larger semantic transformations in each iteration rather than token-by-token modifications, our approach dramatically reduces the number of queries required to find successful adversarial examples. Our experiments demonstrate that CAMOUFLAGE achieves an average attack success rate of 46.92% against four evidence-based misinformation detection systems, including two recent academic misinformation detection systems ( Singhal et al., 2024 ; Tian et al., 2024 ) and two real-world APIs ( IDIR Lab, 2017 ; Perplexity Team, 2024 ) . Furthermore, CAMOUFLAGE significantly outperforms existing black-box attack methods, which achieve at most 26.56% attack success rate on the same systems. Our approach maintains semantic equivalence and textual coherence as verified through human evaluation, while requiring only 10 queries to compromise the victim model. We also demonstrate that a simple text simplification defense can reduce CAMOUFLAGE’s effectiveness by up to 65.18%, highlighting a potential pathway to defend against these type of attacks. Our contributions are as follows: • New Findings. We study the vulnerability of evidence-based misinformation detection systems to adversarial attacks, revealing significant robustness gaps that could be exploited to spread misinformation. • New Attack. We propose a new adversarial attack on evidence-based misinformation detection systems that preserves the semantics and coherence of the original misinformation claim while only requiring detector labels and minimal queries to the victim model. • Comprehensive Examination. We explore why our proposed attack fools evidence-based misinformation detection systems, conduct a comprehensive evaluation of our approach, and propose a defense against our proposed attack that reduces the complexity of the perturbed text, informed by observations that the perturbed text was more challenging to read. 2. Related Work 2.1. Misinformation Detection Misinformation refers to false or misleading information that is spread, regardless of intent. In today’s online ecosystem, misinformation, including “fake news”, travels rapidly through social media, often outpacing the truth ( Nakov et al., 2021 ) . Traditional fact-checking by journalists and dedicated organizations (e.g. PolitiFact, Snopes) is meticulous but slow, where a single claim might take professionals hours or days to verify. As the volume and velocity of online

Read the complete text