Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks

Jinyan Su, Jin peng Zhou, Zhengxin Zhang, Preslav Nakov, Claire Cardie · 2024

Read Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks by Jinyan Su, Jin peng Zhou, Zhengxin Zhang, Preslav Nakov, Claire Cardie (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16708.txt.

Introduction

Jinyan Su and co-authors stress-test retrieval-augmented generation systems against poisoning attacks on their knowledge sources. These systems fetch passages to ground their answers. Planted false passages can hijack the outputs badly.

The study builds a careful taxonomy of contexts. Adversarial, untouched, and guiding passages are mixed deliberately. Retrievers are graded on how often they surface poison. Generators are graded on how much poison sways them. Skeptical prompting woke the models' own reasoning defenses. That defense worked only in models that reason well. Combined fixes on both stages helped the most. The work maps where RAG pipelines need armor the most, in plain, clear terms.

Catalog record

CREATORJinyan Su, Jin peng Zhou, Zhengxin Zhang, Preslav Nakov, Claire Cardie
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks Jinyan Su, Jin peng Zhou, Zhengxin Zhang, Preslav Nakov, Claire Cardie Abstract Retrieval-Augmented Generation (RAG) systems have emerged as a promising solution to mitigate LLM hallucinations and enhance their performance in knowledge-intensive domains. However, these systems are vulnerable to adversarial poisoning attacks, where malicious passages injected into the retrieval corpus can mislead models into producing factually incorrect outputs. In this paper, we present a rigorously controlled empirical study of how RAG systems behave under such attacks and how their robustness can be improved. On the generation side, we introduce a structured taxonomy of context types—adversarial, untouched, and guiding—and systematically analyze their individual and combined effects on model outputs. On the retrieval side, we evaluate several retrievers to measure how easily they expose LLMs to adversarial contexts. Our findings also reveal that “skeptical prompting” can activate LLMs’ internal reasoning, enabling partial self-defense against adversarial passages, though its effectiveness depends strongly on the model’s reasoning capacity. Together, our experiments 1 1 1 Code is available at https://github.com/JinyanSu1/eval_PoisonRaG . and analysis provide actionable insights for designing safer and more resilient RAG systems, paving the way for more reliable real-world deployments. 1 Introduction Large language models (LLMs), though powerful, are inherently limited to the knowledge from their training data and are prone to hallucinations Pal et al. (2023) ; Huang et al. (2024) ; Ahmad et al. (2023) . Retrieval augmented generation (RAG) Mao et al. (2020) ; Lewis et al. (2020) ; Guu et al. (2020) ; Chen et al. (2024) , a framework that combines LLMs with external knowledge, has been introduced as a promising approach to reduce hallucinations and improve the overall model accuracy in knowledge-intensive domains Wu et al. (2024) ; Li et al. (2024) ; Kirchenbauer and Barns (2024) ; Xiong et al. (2024) . Despite these benefits, a growing body of research highlights that RAG systems are vulnerable to various adversarial poisoning attacks Pan et al. (2023) ; Zou et al. (2024) ; RoyChowdhury et al. (2024) ; Tan et al. (2024) ; Cheng et al. (2024) ; Deng et al. (2024) : by injecting adversarial data into the underlying RAG database, attackers can manipulate the retriever to return factually incorrect information, thereby reducing model accuracy and propagating disinformation. Mitigating the impact of such attacks requires addressing both the retrieval and the generation aspects of the system. On the retrieval side, this involves improving the quality of the retrieved contexts by minimizing the inclusion of maliciously crafted content. It is crucial to examine how likely adversarial contexts are to be retrieved and how the contexts interact with each other and influence the downstream generation process. From a generation perspective, as LLMs continue to advance and demonstrate improved critical thinking and knowledge capabilities, we wonder if these enhancements inherently strengthen the robustness of RAG systems against adversarial attacks? On one hand, as LLMs acquire more knowledge, they may answer more questions correctly without relying on external contexts. On the other hand, their improved critical thinking skills may enable them to discern when to trust their internal knowledge versus when to rely on external sources, particularly in the presence of inconsistencies. In this paper,we design rigorous, controlled experiments to analyze how context quality affects RAG systems, both individually and in combination-We explore both the retrieval and the generation aspects of RAG systems to understand how to make them more robust against adversarial poisoning attacks. Specifically, we make the following key contributions: • We introduce a structured taxonomy of context types—adversarial, untouched, and guiding—and systematically evaluate both their individual and collective influence on RAG robustness. In contrast, prior work primarily focuses on coarse-grained issues such as imperfect retrieval or knowledge conflict, which lack the granularity needed to conduct controlled experiments that isolate and analyze the effects of specific context types and their interactions. • We demonstrate that skeptical prompting activates LLMs’ internal reasoning capabilities, enabling them to critically assess retrieved content and defend against adversarial contexts without requiring external correction mechanisms. However, While it can serve as one practical mitigation strategy, our analysis further reveals that the effectiveness of this self-defense behavior is closely tied to the LLM’s reasoning and critical thinking capacity. • Through extensive experiments, we uncover several findings that raise new research questions and directions. For instance, we observe that DPR tends to embed incorrect passages farther from queries, making it less likely to retrieve adversarial contexts. We also find that skeptical prompting preserves performance on clean contexts while significantly enhancing robustness in adversarial settings. These findings not only inform practical system design but also open up compelling avenues for future research, such as developing retriever-aware attack strategies or designing adaptive prompting mechanisms. 2 Related Work Retrieval-Augmented Generation (RAG). RAG is a widely adopted paradigm that effectively combines the strengths of information retrieval and text generation Lewis et al. (2020) . A typical RAG system operates in two phases: (1) Retrieval Phase: In this phase, the system retrieves the k most relevant passages from the corpus \mathcal{C} based on the input query q . The corpus \mathcal{C} can be easily updated with new information, ensuring the system remains adaptable to changing knowledge and specific domains. (2) Generation Phase: In this phase, the retrieved passages are combined with the original query to form the input to a large language model (LLM). By integrating its pre-trained knowledge with the retrieved passages, the LLM generates a more grounded and accurate response. This approach reduces hallucinations commonly observed in standalone LLMs and allows LLM response to adapt more effectively to evolving knowledge. Adversarial Poisoning on RAG. Pan et al. (2023) warned of the dangers of disinformation pollution with LLMs, studied several attacks and proposed simple defenses based on subsamplimng of the retrieved passages and skeptical prompting. Zhong et al. (2023) ; Su et al. (2024) focused only on the retrieval phase and investigated using gradient information to create adversarial contexts that are likely to be retrieved by many queries. Long et al. (2024) conducted a targeted query attack, generating adversarial passage only for the target queries while not considering the influence of retrieved adversarial passages on the LLM’s outputs, while Zou et al. (2024) focused on both the retrieval phase and the generation phase by crafting adversarial passages for specific queries and directly influencing the LLM’s generation by inducing the model to output a target answer for specific questions. More discussions on related work can be find in Appendix A.1 . RAG with Irrelevant or Incorrect Contexts. Beyond intentionally injected adversarial contexts, RAG systems inevitably introduce irrelevant or incorrect information due to the limitations of their retrievers Yin et al. (2023) . Many works Asai et al. (2023) ; Yan et al. (2024) ; Chen et al. (2024) ; Wang et al. (2024) showed that irrelevant or incorrect contexts could detrimentally affect model performance. Huang et al. () showed that when provided with both correct and incorrect contexts, models tend to overly rely on external information, regardless of its factual accuracy. More discussion on these related works can be find in Appendix A.1 . 3 Peering Into Adversarial Poisoning Attacks In RAG systems, LLMs retrieve information from external knowledge bases to

Read the complete text