BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel
Hao Li, Jiajun He, Guangshuo Wang, Dengguo Feng, Zheng Li, Min Zhang · 2025
Read BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel by Hao Li, Jiajun He, Guangshuo Wang, Dengguo Feng, Zheng Li, Min Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12043.txt.
Introduction
Hao Li and five co-authors uncover a new privacy risk in retrieval-augmented generation, or RAG, systems that answer questions using private text collections. Attackers can try to tell whether a given text was in that collection. Older attacks work poorly here because the system is a black box and gives weak signals.
The team names the leak BudgetLeak. It varies the generation budget, the cap on how long a reply can be, and watches how answer quality changes. Texts in the collection gain quality faster as the budget grows. Tests on four datasets beat prior baselines, which points to a need for new defenses.
Catalog record
| CREATOR | Hao Li, Jiajun He, Guangshuo Wang, Dengguo Feng, Zheng Li, Min Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12043
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel Hao Li, Jiajun He, Guangshuo Wang, Dengguo Feng, Zheng Li, Min Zhang Abstract Retrieval-Augmented Generation (RAG) enhances large language models by integrating external knowledge, but reliance on proprietary or sensitive corpora poses various data risks, including privacy leakage and unauthorized data usage. Membership inference attacks (MIAs) are a common technique to assess such risks, yet existing approaches underperform in RAG due to black-box constraints and the absence of strong membership signals. In this paper, we identify a previously unexplored side channel in RAG systems: the generation budget, which controls the maximum number of tokens allowed in a generated response. Varying this budget reveals observable behavioral patterns between member and non-member queries, as members gain quality more rapidly with larger budgets. Building on this insight, we propose BudgetLeak, a novel membership inference attack that probes responses under different budgets and analyzes metric evolution via sequence modeling or clustering. Extensive experiments across four datasets, three LLM generators, and two retrievers demonstrate that BudgetLeak consistently outperforms existing baselines, while maintaining high efficiency and practical viability. Our findings reveal a previously overlooked data risk in RAG systems and highlight the need for new defenses. † † footnotetext: Corresponding authors. Introduction With the rapid expansion of training corpora and model sizes, large language models (LLMs) such as GPT-4 [ 4 ] , LLaMA [ 52 ] , Mistral [ 24 ] , and ChatGLM [ 67 ] have demonstrated remarkable generative capabilities, significantly impacting both professional domains and everyday interactions. Despite these advances, LLMs continue to face fundamental challenges, including hallucinations [ 49 , 63 ] and outdated knowledge [ 15 , 26 ] , which raise serious concerns about their reliability in real-world applications. A widely adopted approach to address these issues is Retrieval-Augmented Generation (RAG) [ 28 , 15 , 14 ] , which improves response quality by incorporating up-to-date and contextually relevant external information into the generation process. At the same time, the dependence of RAG systems on external databases, particularly those containing sensitive information, introduces significant privacy and security risks. In the healthcare domain, systems such as MedPlan may enrich with patient-specific information retrieved from external sources, enhancing the quality of clinical assessments and personalized recommendations. Such integration, however, raises concerns about unauthorized access to highly sensitive patient data [ 21 ] . A similar issue arises in the financial sector, where RAG systems employed for financial intent detection like BAI-Fintent often rely on key transaction records, which—if improperly secured—could expose confidential financial information [ 51 , 55 ] . In the cryptography domain, tools like CRYPTOSCOPE search for vulnerability cases stored in an external database to support cryptographic logic vulnerability detection. This practice increases the likelihood of security preferences being exposed [ 34 ] . These examples underscore the importance of carefully examining how sensitive data is ingested and protected in RAG-based applications. (a) Similarity Scores (b) Response Accuracy Figure 1 : Distribution of similarity scores and response accuracy on RAG with LLaMA over HealthCareMagic-100k. To address this challenge, we investigate membership inference attacks (MIAs), a well-established class of techniques for evaluating privacy risks and unauthorized data exposure. MIAs assess whether a given sample was included in a model’s training set by analyzing the model’s prediction behavior. In RAG systems, however, the goal shifts to identifying whether a data sample resides in the system’s knowledge base. Several methods have been proposed for this purpose. Li et al. [ 32 ] divided a target sample into two segments, submitted the first segment to the RAG system, and measured the similarity between the generated continuation and the second segment as a membership indicator. Liu et al. [ 36 ] masked key tokens in the target sample and queried the RAG system to fill in the blanks, using fill-in accuracy as the membership score. Naseh et al. [ 39 ] converted the target sample into a set of Yes/No questions and evaluated the RAG system’s responses, with accuracy serving as the membership signal. These approaches compensate for the lack of conventional numerical signals (e.g., loss, confidence, entropy) in RAG outputs. In general, member samples yield higher similarity, fill-in accuracy, and response accuracy, reflecting better overall output quality. Unfortunately, these signals are insufficient for reliable membership inference. As shown in Figure 1 (see more in Appendix Figure 12 ), members and non-members exhibit substantial overlap across these metrics. This ambiguity stems from the fact that RAG output quality is influenced by multiple factors beyond membership. First, responses quality depends not only on whether the input appears in the external knowledge base, but also on whether the internal language model encodes semantically related information. Bi et al. [ 9 ] show that the balance between internal and external knowledge can be tuned during generation, but such control is infeasible in real-world black-box scenarios. Second, response quality varies with sample diversity, a well-known challenge in membership inference [ 58 , 10 , 44 , 19 ] . Wang et al. [ 54 ] address this by assuming that the adversary has access to all target samples and apply a one-time calibration using shared in/out RAGs. While effective, this reduces the MIA task to ranking samples within a fixed dataset, which is unrealistic in practice. Therefore, extracting reliable membership signals from RAG systems without such assumptions remains technically challenging. To fill this gap, we introduce a previously unexplored side-channel for membership inference: the generation budget , defined as the maximum token limit for generated responses. To our knowledge, this is the first study to show that this parameter systematically leaks membership information in RAG systems. When the generation budget is small, responses are compressed, often omitting details and reducing similarity to the ground-truth answer. With a large budget, the system produces more informative and specific content, yielding outputs that better align semantically with the ground truth. This effect differs between member and non-member queries. For member queries, where the ground-truth answer is typically included in the retrieved context, larger budgets enable the generator to reconstruct detailed responses more effectively, leading to sharper improvements in similarity or fidelity metrics. Non-member queries show weaker gains under the same conditions. This divergence in response patterns constitutes a novel and practical side channel for membership inference attacks. Based on this observation, we present BudgetLeak, a novel side-channel membership inference attack that exploits the generation budget as a leakage signal in RAG systems. Specifically, BudgetLeak leverages generation budget sensitivity to distinguish members from non-members. The adversary repeatedly queries the RAG system with a target sample under varying the generation budget (e.g., 10, 20, 30 tokens) until it exceeds the ground-truth answer length. For each response, the adversary computes its semantic similarity or other fidelity metrics relative to the ground-truth answer. These metric values are then ordered by the generation budget to form sequences, which are aggregated into a unified multi-metric representation. This representation captures how metrics evolve with increasing budgets, including growth patterns, fluctuations, and cross-metric correlations. The adversary inputs this representation into a sequence attack model, such as a Recurrent Neural Network (RNN), to learn the temporal and relational patterns that distinguish members from non-members. We also propose