Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems Conference: Workshop on Artificial Intelligence and Security; November 2026; Salt Lake City, UT, USA CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Information systems Retrieval models and ranking

Scott Thornton · 2026

Read Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems Conference: Workshop on Artificial Intelligence and Security; November 2026; Salt Lake City, UT, USA CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Information systems Retrieval models and ranking by Scott Thornton (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18034.txt.

Introduction

Thornton studies corpus poisoning attacks on retrieval-augmented generation systems and retrieval-layer defenses. Attackers plant a sleeper and a trigger document, tuned with greedy coordinate gradient steps. On a 67,941-document Security Stack Exchange corpus, poisoning reached a 38% co-retrieval rate. Hybrid retrieval with BM25 plus vector search cut that rate to zero across 50 attacks. Joint attacks on both signals still fail more often than with vector search alone.

Catalog record

CREATORScott Thornton
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41571
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18034

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems Conference: Workshop on Artificial Intelligence and Security; November 2026; Salt Lake City, UT, USA CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Information systems Retrieval models and ranking Scott Thornton Abstract . Retrieval-Augmented Generation (RAG) systems enhance LLMs with external knowledge but introduce poisoning attack surfaces through the retrieval mechanism. We show that a simple hybrid BM25 + vector retriever provides an effective architectural defense against gradient-guided RAG poisoning, and present exploratory evidence that corpus composition affects both attack feasibility and detection difficulty. Using dual-document (sleeper–trigger) poisoning optimized via Greedy Coordinate Gradient (GCG), our large-scale evaluation ( n = 50 attacks on Security Stack Exchange, 67,941 docs) shows that gradient-guided poisoning achieves 38.0 % co-retrieval on pure vector retrieval. Across all 50 attacks, hybrid BM25 + vector retrieval reduced gradient-guided attack success from 38 % to 0 % , demonstrating that a simple architectural change at the retrieval layer can eliminate this attack class without modifying the LLM. When attackers jointly optimize for both sparse and dense retrieval channels, hybrid retrieval is partially circumvented (20–44 % success), but still significantly raises the attack bar compared to pure vector retrieval. Across five LLM families (GPT-5.3, GPT-4o, Claude Sonnet 4.6, Llama 4, GPT-4o-mini), attack success varies dramatically from 46.7 % (GPT-5.3) to 93.3 % (Llama 4), demonstrating that model-level safety training significantly affects resilience to RAG poisoning but does not eliminate the threat. Cross-corpus evaluation at n = 25 on FEVER Wikipedia confirms corpus-dependent effects (0 % attack success across all retrieval configurations). For practitioners, we recommend hybrid retrieval with α ≤ 0.5 \alpha\leq 0.5 as an immediately deployable first-line defense, supplemented by model-level safety evaluation and QPD monitoring. 1. Introduction Retrieval-Augmented Generation (RAG) is the dominant architecture for deploying LLMs with external knowledge ( 5 ) . By grounding generation in retrieved documents, RAG reduces hallucination and enables dynamic knowledge updates—but the retrieval mechanism itself is an attack surface. Adversaries who can inject documents into the knowledge base can manipulate model outputs without touching model weights or system prompts ( 14 ; 1 ) . This attack vector is particularly insidious because poisoned documents persist across all future queries matching retrieval criteria, must avoid detection by monitoring systems, and activate only for specific target queries while remaining dormant for benign ones. The OWASP LLM Top 10 (2025) recognizes vector and embedding weaknesses as a top-10 risk ( 7 ) , yet systematic study of how corpus composition affects the attack–defense balance remains absent. Prior work demonstrates that poisoning attacks are feasible on standard QA datasets ( 14 ; 11 ; 2 ) and that various defenses offer partial mitigation ( 6 ; 12 ) . However, a fundamental question remains unaddressed: how does the composition of the knowledge base affect both attack effectiveness and defense difficulty? Organizations deploy RAG across diverse domains—security documentation, encyclopedic knowledge, medical records, legal texts—and defensive strategies that work on one corpus may fail on another. We answer this question through four contributions: (1) Hybrid retrieval as architectural defense (primary contribution). A simple BM25 + vector hybrid retriever reduced gradient-guided poisoning from 38 % to 0 % co-retrieval across all tested mixing weights ( \alpha = 0.3–0.7) in our n = 50 evaluation on Security Stack Exchange. Adaptive keyword injection (25 configurations) also failed. Joint sparse + dense optimization partially circumvents the defense (20–44 % success), establishing that hybrid retrieval raises the attack bar significantly but is not impenetrable (§ 6.2.2 ). (2) Multi-model end-to-end LLM evaluation. We evaluate the full attack chain across five LLMs from four providers. Attack success ranges from 46.7 % (GPT-5.3) to 93.3 % (Llama 4), with safety violation rates from 6.7 % (Claude Sonnet 4.6) to 93.3 % (Llama 4). This demonstrates both that model-level safety training is insufficient without retrieval-level defenses, and that safety training maturity varies dramatically across model families. (3) Corpus-dependent attack–defense characterization (exploratory). Cross-corpus evaluation at n = 9 (pilot) and n = 25 (FEVER confirmation) shows that technical corpora (Security Stack Exchange) and general corpora (FEVER Wikipedia) exhibit qualitatively different security profiles: the technical corpus enables 66.7 % attack stealth but only 11.1 % overall success, while FEVER achieves 100 % co-retrieval but 0 % stealth across all 25 trials and all retrieval configurations. Detection difficulty varies 13–62 \times between corpora. (4) Production-scale case study. A preliminary case study on 156K documents from a major network security vendor provides initial evidence that attacks are non-transferable across corpora (0 % naive success) while corpus-adaptive attacks achieve 100 % retrieval with #1–2 ranking. 2. Related Work 2.1. RAG Poisoning Attacks PoisonedRAG ( 14 ) formulates knowledge corruption as an optimization problem with retrieval and generation conditions, achieving 90 % attack success on standard QA benchmarks. RAG Security Bench ( 11 ) provides the most comprehensive evaluation to date, testing 13 poisoning methods across 5 datasets with 7 defense mechanisms and finding that hybrid defenses outperform single-mechanism approaches but still leave residual risk. RAG Safety ( 2 ) extends poisoning evaluation to multimodal and advanced RAG architectures, demonstrating broad susceptibility. Traceback analysis ( 12 ) reveals that current defense techniques provide limited robustness against sophisticated poisoning. RAGuard ( 6 ) expands retrieval scope and applies fragment-wise perplexity filtering but shows limited effectiveness on technical corpora. Our work complements these broad multi-dataset evaluations by treating corpus domain as the primary security variable. While prior work evaluates attacks and defenses across multiple datasets, they do not explicitly quantify corpus-dependent detection difficulty or frame corpus domain as a first-class security parameter. 2.2. Hybrid Retrieval Systems Hybrid retrieval combining sparse (BM25) and dense (vector) search has emerged as state-of-the-art for information retrieval quality ( 8 ; 9 ) . Reciprocal Rank Fusion provides effective score combination ( 3 ) . While hybrid retrieval is widely deployed for performance and recent work evaluates complex hybrid defense pipelines ( 11 ) , there is little explicit analysis of simple BM25 + vector mixing as a primary architectural defense against poisoning. We provide the first systematic evaluation showing that such a hybrid retriever blocked all gradient-guided dual-document attacks in our experiments. 2.3. Adversarial ML Foundations Data poisoning is a well-established threat to machine learning systems ( 1 ; 4 ) . Universal adversarial triggers attack NLP models ( 10 ) , and GCG optimization produces universal jailbreak suffixes ( 13 ) . RAG poisoning differs fundamentally by targeting the retrieval mechanism rather than model parameters: attackers manipulate which documents reach the LLM context window without requiring access to model weights or training processes. 3. Threat Model and Attack Design 3.1. Threat Model Attacker capabilities. The attacker can inject documents into the RAG corpus ( e.g. via user-generated content, API submissions, or compromised data pipelines), understands target query patterns and the embedding model (or a proxy), but cannot modify LLM parameters, system prompts, or retrieval algorithms. Attack class scope. We evaluate two attack classes: (1) gradient-guided semantic optimization —GCG-style coordinate search maximizing cosine similarity in embedding space, and (2) joint sparse + dense optimization —simultaneous targeting of both vector similarity and BM25 keyword overlap to circumvent hybrid retrieval. We do not evaluate black-box search methods

Read the complete text