SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model
Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang · 2025
Read SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model by Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18636.txt.
Introduction
Xun Liang and co-authors build SafeRAG, a benchmark for attacks on retrieval augmented language systems, in a 2025 paper. Such systems fetch outside text to ground their answers, but fetched text can be poisoned. Attackers can plant noise, clashing claims, soft ads, or blocking floods.
The team sorted attacks into four families and built test data for each, mostly by hand. Fourteen system parts, from retrievers to filters to the models themselves, then faced the suite. Every attack family hurt service quality, and even crude attacks slipped past defenses. The authors offer the set as a standing security check for builders of retrieval systems.
Catalog record
| CREATOR | Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang Abstract The indexing-retrieval-generation paradigm of retrieval-augmented generation (RAG) has been highly successful in solving knowledge-intensive tasks by integrating external knowledge into large language models (LLMs). However, the incorporation of external and unverified knowledge increases the vulnerability of LLMs because attackers can perform attack tasks by manipulating knowledge. In this paper, we introduce a benchmark named SafeRAG designed to evaluate the RAG security. First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service. Next, we construct RAG security evaluation dataset (i.e., SafeRAG dataset) primarily manually for each task. We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter. Experiments conducted on 14 representative RAG components demonstrate that RAG exhibits significant vulnerability to all attack tasks and even the most apparent attack task can easily bypass existing retrievers, filters, or advanced LLMs, resulting in the degradation of RAG service quality. Code is available at: https://github.com/IAAR-Shanghai/SafeRAG . 1 Introduction Retrieval-augmented generation (RAG) provides an efficient solution for expanding the knowledge boundaries of large language models (LLMs) Zhao et al. 2024 ; Gupta et al. 2024 ; Fan et al. 2024 ; Wang et al. 2024 . Many advanced LLMs, such as ChatGPT OpenAI et al. 2024 , Gemini Team et al. 2024 , and Perplexy.ai 1 1 1 https://www.perplexity.ai/ , have incorporated external retrieval modules within their web platforms. However, in the RAG pipeline, query-relevant texts are processed sequentially through the retriever , the filter before being synthesized into a response by the generator , introducing potential security risks, as attackers can manipulate texts at any stage of the pipeline. Figure 1: Motivation. The attack tasks considered by the existing RAG benchmarks fail to bypass the RAG components, which hindering accurate RAG security evaluation. Our SafeRAG introduces enhanced attack tasks to evaluate the potential vulnerabilities of RAG. Current attacks targeting RAG can be divided into four tasks: • Noise : Due to the limitation in retrieval accuracy, the retrieved contexts often contain large quantities of noisy texts that are at most merely similar to the query but do not actually contain the answer. Attackers can exploit this retrieval limitation to dilute useful knowledge by deliberately injecting extensive noisy texts Chen et al. 2024a ; Fang et al. 2024 . • Conflict : Knowledge from different sources may conflict with one another, creating opportunities for attackers to manipulate. Simply injecting conflict texts could prevent LLMs from determining which piece of knowledge is more reliable, resulting in vague or even incorrect responses. Wu et al. 2024a ; Liu et al. 2023 ; Zou et al. 2024 . • Toxicity : The internet often contains toxic texts published by attackers. Such malicious texts are highly likely to be incorporated into the RAG pipeline, inducing LLMs to generate toxic responses Deshpande et al. 2023 ; Perez and Ribeiro 2022 . • Denial-of-Service (DoS) : The target of DoS is to cause LLMs to refuse to answer, even when evidence is available Chaudhari et al. 2024 ; Shafran et al. 2024 . DoS-inducing texts injected by attackers are particularly insidious because the resulting behavior is easily mistaken for the RAG’s limitations. However, most of existing attack tasks often fail to bypass the safety RAG components, making the attacks no longer suitable for RAG security evaluation. There are four main reasons. R -1 : Simple safety filters can effectively defend against noise attack Li et al. 2024 , as existing noise is often concentrated in superficially relevant contexts, which may actually belong to either similar-topic irrelevant contexts or relevant contexts that do not contain answers. (Fig. 1 -⑨). R -2 : Existing conflict primarily focuses on questions that LLMs can directly answer but contain factual inaccuracies in the related documents Xu et al. 2024 . Current adaptive retrievers Tan et al. 2024 have been able to effectively mitigate such context-memory conflict. R -3 : Advanced generators demonstrate strong capabilities in detecting and avoiding explicit and implicit toxicity, such as bias, discrimination, metaphor, and sarcasm Sun et al. 2023 ; Wen et al. 2023 . R -4 : Traditional DoS attack mainly involves maliciously inserting explicit (Fig. 1 -⑤⑥) or implicit (Fig. 1 -⑦⑧) refusal signals into the RAG pipeline. Fortunately, such signals are often filtered out as they inherently do not support answering the question, or they are ignored by generators due to being mixed into evidences Shafran et al. 2024 . To address above limitations, we propose four novel attack tasks for conducting effective RAG security evaluation. Firstly, we define silver noise (Fig. 1 -②), which refers to evidence that partially contains the answer. Such noise can circumvent most safety filters , thereby undermining the RAG diversity ( R -1 ). Secondly, unlike the widely studied context-memory conflict, we explore a more hazardous inter-context conflict (Fig. 1 -④). Since LLMs lack sufficient parametric knowledge to handle external conflicts, they are more susceptible to being misled by tampered texts ( R -2 ). Thirdly, we reveal the vulnerability of RAG under the soft ad attack (Fig. 1 -③). As a special type of implicit toxicity, the soft ad can evade LLMs and ultimately be inserted into the response of generators ( R -3 ). Finally, to enable refusal signals to bypass filters or generators , we propose a white DoS (Fig. 1 -①) attack. Under the guise of a safety warning, such attack falsely accuses the evidence of containing a large number of distorted facts, thereby achieving its purpose of refusal ( R -4 ). Existing benchmarks mainly focus on applying a certain attack task at specific stages of the RAG pipeline and observing the impact of the selected attack on the retriever or generator . In this paper, we introduce the RAG security evaluation benchmark, SafeRAG , which systematically evaluates the potential security risks in the retriever and generator by performing four improved attack tasks across different stages of the RAG pipeline. Our main contributions are: • We reveal four attack tasks capable of bypassing the retriever , filter , and generator . For each attack task, we develop a lightweight RAG security evaluation dataset, primarily constructed by humans with LLM assistance. • We propose an economical, efficient, and accurate RAG security evaluation framework that incorporates attack-specific metrics, which are highly consistent with human judgment. • We introduce the first Chinese RAG security benchmark, SafeRAG , which analyzes the risks posed to the retriever and generator by the injection of noise , conflict , toxicity , and DoS at various stages of the RAG pipeline. Table 1: Related works. Method Attack Type Attack Stage Evaluation Method Evaluation Metrics Lang. Evaluation Task RGB Chen et al. 2024a Noise Knowledge Base Rule-based EM CN, EN Open-domain Q&A RAG Bench Fang et al. 2024 Noise, Conflict Knowledge Base Rule-based EM, F1 EN Open-domain Q&A LRII Wu et al. 2024b Noise, Conflict Filtered Context Model-based Misleading Ratio, Uncertainty Ratio EN Open-domain Q&A, Simple Fact Q&A RECALL Liu et al. 2023 Conflict Filtered Context Model-based, Rule-based Accuracy, BLEU, ROUGE-L, Misleading