ControlNet : A Firewall for RAG-based LLM System

Hongwei Yao 1,2 , Haoran Shi 2 , Yidou Chen 2 , Yixin Jiang 3 , Cong Wang 1 🖂, Zhan Qin 2 🖂 · 2025

Read ControlNet : A Firewall for RAG-based LLM System by Hongwei Yao 1,2 , Haoran Shi 2 , Yidou Chen 2 , Yixin Jiang 3 , Cong Wang 1 🖂, Zhan Qin 2 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09593.txt.

Introduction

Hongwei Yao and co-authors build a firewall for AI systems that fetch facts from outside sources before they answer. Such systems cut made-up answers, but they open new risks: private data can leak out, and poisoned data can be fed in through the fetch step.

Their firewall, ControlNET, watches the flow of queries in and out. It spots hostile queries by a shift in the model's inner signals, and it blunts their effect by a spread of meaning. Tests on four data sets with three open models showed strong scores for spotting and stopping threats, while the system stayed harmless in normal use.

Catalog record

CREATORHongwei Yao 1,2 , Haoran Shi 2 , Yidou Chen 2 , Yixin Jiang 3 , Cong Wang 1 🖂, Zhan Qin 2 🖂
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-09593

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ControlNet : A Firewall for RAG-based LLM System Hongwei Yao 1,2 , Haoran Shi 2 , Yidou Chen 2 , Yixin Jiang 3 , Cong Wang 1 🖂, Zhan Qin 2 🖂 Abstract . Retrieval-Augmented Generation (RAG) has significantly enhanced the factual accuracy and domain adaptability of Large Language Models (LLMs). This advancement has enabled their widespread deployment across sensitive domains such as healthcare, finance, and enterprise applications. RAG mitigates hallucinations by integrating external knowledge, yet it introduces privacy risk and security risk, notably data breaching risk and data poisoning risk . While recent studies have explored prompt injection and poisoning attacks, there remains a significant gap in comprehensive research on controlling inbound/outbound query flows to mitigate these threats. In this paper, we propose an AI firewall, ControlNet , designed to safeguard RAG-based LLM systems from these vulnerabilities. ControlNet controls query flows by leveraging activation shift phenomena to detect malicious queries and mitigate their impact through semantic divergence. We conduct comprehensive experiments using state-of-the-art open-source LLMs—Llama3, Vicuna, and Mistral—across four benchmark datasets (MS MARCO, HotpotQA, FinQA, and MedicalSys). Our empirical results demonstrate that ControlNet is not only effective but also harmless. It achieves an AUROC exceeding 0.909 for risk detection, with minimal degradation in Precision and Recall , both of which show reductions of less than 0.03 and 0.09, respectively, for risk mitigation. Overall, ControlNet offers an effective, harmless, robust defense mechanism, marking a significant advancement toward the secure deployment of RAG-based LLM systems. 1. Introduction In recent years, Large Language Models (LLMs) such as ChatGPT, Claude and DeepSeek ( Guo et al. 2025 ) have demonstrated remarkable proficiency in domain-specific and knowledge-intensive tasks, including personal assistant, medical assistant, financial advisor, and legal research. This advancement has been significantly driven by the integration of Retrieval-Augmented Generation (RAG) techniques. By incorporating external knowledge retrieval, RAG mitigates the hallucination problem inherent in standalone generative models ( Ye et al. 2024 ; Gunjal et al. 2024 ) , improving their ability to respond to queries requiring real-time or out-of-distribution information ( Fan et al. 2024 ) . A typical RAG-based LLM system operates by processing user queries through a structured retrieval mechanism. When a client submits a query, the server encodes it into a vector representation, which is then used to retrieve relevant documents from a vector database (denoted as VectorDB ). These documents, stored in vector form, are subsequently integrated with the original query and processed by the LLM to generate a response. The knowledge provider ensures that the VectorDB remains updated with accurate and relevant documents. Finally, the generated response is delivered to the client (as shown in Figure 1 (a)). This retrieval-based approach has proven highly effective in reducing factual inaccuracies, contributing to the widespread adoption of RAG as a foundational technology for enhancing the practical applicability of LLMs. Figure 1. Illustration of the data flow in a RAG-based LLM system. (a) Without the firewall, the doctor gains unauthorized access to financial data. (b) With the firewall ControlNet , role-based access control ensures the doctor can only retrieve patient information. Despite these advancements, the incorporation of external knowledge documents introduces critical vulnerabilities related to privacy and security . These risks manifest in two primary forms, data breaching risk 1 1 1 https://securityintelligence.com/articles/chatgpt-confirms-data-breach and data poisoning risk 2 2 2 https://www.theguardian.com/technology/2024/dec/24/chatgpt-search-tool-vulnerable-to-manipulation-and-deception-tests-show . First, data breaching risks arise from malicious clients conducting reconnaissance to extract system prompts or exploit RAG system environments. In multi-client systems where users have differentiated access rights (e.g., executives, financial officers, general staff), such vulnerabilities increase the likelihood of data exfiltration and unauthorized document access(as shown in Figure 1 (b)). Data breaching can lead to the exposure of system environments and business-sensitive information, posing severe consequences for organizations. Second, from the corpus collection perspective, data poisoning presents another security challenge. Malicious knowledge providers may introduce compromised data into the VectorDB , thereby injecting misleading information or hijacking clients’ conversation. Data poisoning can mislead clients, degrade trust in the system, and compromise decision-making processes. Given these escalating risks, there is an urgent need for effective, robust defense mechanisms to ensure the secure and trustworthy deployment of RAG-based LLM systems. Ensuring the integrity and security of RAG-based LLM systems requires the development of effective mechanisms for detecting malicious queries, filtering poisoned documents, and implementing effective query flow control. While recent studies have explored prompt injection ( Liu et al. 2024a ; Liu et al. 2023 ; Shi et al. 2024 ; Zhu et al. 2024 ) and poisoning attacks ( Yao et al. 2024b ; Yang et al. 2024 ; Zhan et al. 2024 ; Yu et al. 2023 ; Toyer et al. 2024 ) , and guardrails ( Inan et al. 2023 ; Meta 2024 ; Rebedea et al. 2023 ) in LLMs, there remains a significant gap in comprehensive research on controlling inbound/outbound query flows to mitigate these threats. Although guardrails contribute to safety and alignment but often fail to manage complex, multi-role system flows effectively. Addressing these risks in RAG-based systems presents several key challenges. First, both incoming queries and retrieved documents exist as unstructured textual data, rendering traditional regular expression-based matching ineffective. Instead, effective flow control should leverage deeper semantic features, such as neuron activation patterns within the model. Second, establishing a principled connection between neuron activation patterns and client-based access control in RAG systems remains an open problem. Third, once an malicious query or poisoned document is detected, mitigating their impact through query sanitization presents a further challenge. To address these challenges, future research must prioritize the design of robust query sanitization techniques and adversarial impact mitigation strategies. Our Work. In this paper, we introduce a comprehensive RAG security framework that considers three distinct entities: the LLM server, the clients, and the corpus collector. We systematically investigate privacy and security risks in RAG-based LLM systems, identifying five major attacks: reconnaissance, data exfiltration, unauthorized access, knowledge poisoning, and conversation hijacking . To address these threats, we propose ControlNet , a novel AI firewall for RAG-based LLM systems. ControlNet controls query flow by leveraging activation shift phenomena, detecting malicious queries based on distinct activation vector patterns indicative of semantic divergence. Upon detection, ControlNet mitigates risks by steering the LLM’s behavior away from harmful responses, thereby ensuring secure and privacy-preserving interactions. We conduct extensive experiments using three state-of-the-art (SoTA) open-source LLMs—Llama3, Vicuna, and Mistral—across four benchmark datasets: MS MARCO, HotpotQA, FinQA, and MedicalSys. These datasets cover diverse application scenarios, including personal assistant, digital enterprise, financial planning, and healthcare. Our empirical results substantiate that ControlNet is not only effective but also harmless and robust. It achieves an AUROC exceeding 0.909 for risk detection, with minimal degradation in Precision and Recall , both of which show reductions of less than 0.03 and 0.09, respectively, for risk mitigation. The contributions of this work are summarized as follows: • Systematic Risk Analysis : We present a comprehensive taxonomy of privacy and security vulnerabilities in RAG-based systems, categorizing five attacks under two primary risk domains. This framework provides a foundational structure for future research and development. • Benchmark Dataset : We collect and release an open-access, multi-role database for the medical system with local hospitals, comprising over 20,000 data samples across 4 roles.

Read the complete text