GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards

Yule Liu Heyi Zhang Jingyi Zheng Zhen Sun Zifan Peng, Tianshuo Cong Yilong Yang Xinlei He Zhuo Ma · 2025

Read GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards by Yule Liu Heyi Zhang Jingyi Zheng Zhen Sun Zifan Peng, Tianshuo Cong Yilong Yang Xinlei He Zhuo Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14045.txt.

Introduction

Yule Liu, Heyi Zhang, and seven co-authors audit whether private training prompts leaked into a language model's reinforcement learning stage. Modern models are tuned with verifiable rewards on valuable, non-public prompt sets, raising fears of unauthorized data use. Standard membership tests look for fitting to one fixed answer string, which does not fit training that rewards the model's own successful replies.

Their framework, DIBA, compares a tuned model against its earlier checkpoint on each query. It combines reward-side evidence, changes in task success, with policy-side evidence, drift in behavior for the same prompt. Averaging over repeated sampled replies gives a stable signal that beat strong likelihood-based baselines in white-box tests.

Catalog record

CREATORYule Liu Heyi Zhang Jingyi Zheng Zhen Sun Zifan Peng, Tianshuo Cong Yilong Yang Xinlei He Zhuo Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14045

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards Yule Liu Heyi Zhang Jingyi Zheng Zhen Sun Zifan Peng, Tianshuo Cong Yilong Yang Xinlei He Zhuo Ma Abstract Membership inference attacks (MIAs) on large language models (LLMs) pose significant privacy risks across various stages of model training. Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have brought a profound paradigm shift in LLM training, particularly for complex reasoning tasks. However, the on-policy nature of RLVR introduces a unique privacy leakage pattern: since training relies on self-generated responses without fixed ground-truth outputs, membership inference must now determine whether a given prompt (independent of any specific response) is used during fine-tuning. This creates a threat where leakage arises not from answer memorization. To audit this novel privacy risk, we propose D ivergence- i n- B ehavior A ttack (DIBA), the first membership inference attack specifically designed for RLVR. DIBA shifts the focus from memorization to behavioral change, leveraging measurable shifts in model behavior across two axes: advantage-side improvement (e.g., correctness gain) and logit-side divergence (e.g., policy drift). Through comprehensive evaluations, we demonstrate that DIBA significantly outperforms existing baselines, achieving around 0.8 AUC and an order-of-magnitude higher [email protected]%FPR. We validate DIBA’s superiority across multiple settings—including in-distribution, cross-dataset, cross-algorithm, black-box scenarios, and extensions to vision-language models. Furthermore, our attack remains robust under moderate defensive measures. To the best of our knowledge, this is the first work to systematically analyze privacy vulnerabilities in RLVR, revealing that even in the absence of explicit supervision, training data exposure can be inferred through behavioral traces. † † footnotetext: † Corresponding authors: Yilong Yang ( [email protected] ) and Xinlei He ( [email protected] ). Introduction Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks [ 32 , 3 ] , including academic writing [ 20 ] , code generation [ 16 ] , and question answering [ 1 ] . Modern LLMs are typically trained through a three-stage paradigm [ 25 , 11 ] : pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL). Specifically, Reinforcement Learning with Verifiable Rewards (RLVR) [ 32 , 38 , 45 , 35 ] , a recently developed new family of RL algorithms, has brought a significantly profound paradigm shift in the game of LLMs. Almost all mainstream LLMs [ 32 , 3 , 36 ] are employing RLVR as their last step in the pipeline. During the training procedure, models ingest trillions of tokens, which are often drawn from public but privacy-sensitive sources, to enable broad linguistic understanding [ 42 , 1 ] . Given this extensive data exposure, ensuring that trained models do not inadvertently leak information about their training data has become a critical concern. Recent work has shown that each stage of the training pipeline introduces distinct privacy risks, prompting the development of numerous membership inference attack (MIA) techniques to audit data exposure [ 40 , 46 , 13 , 14 , 28 ] . Due to fundamental differences in training objectives and data structures, the adversary’s goal varies significantly across stages. We compare the difference across MIA in different training paradigms in Figure 1 , with each having a different inference target (detailed in Section 2 ). Figure 1 : MIAs in different LLM training paradigms. However, RLVR’s new training diagram [ 32 , 45 ] introduces unique privacy risks. RLVR does not rely on human-provided preferences or exact reference answers. Instead, it generates multiple responses to a given prompt and uses simple, automated rules (e.g., checking whether a math answer is correct) to assign a binary reward (0 or 1) to each response. The model is then updated to favor responses that receive a reward of 1. This approach requires only a verifiable final answer—not a step-by-step “ground truth”—and avoids explicit token-by-token memorization. As a result, traditional attacks that exploit memorization of the ground-truth response become ineffective (typically achieving AUC < 0.6, see Table 1 ). In this challenging RLVR scenario, privacy leakage no longer manifests through memorization of specific outputs. Instead, it emerges as subtle behavioral changes, i.e., slight differences in how the model handles certain inputs. This behavioral divergence between the base and fine-tuned models constitutes a new threat surface and provides a more generalizable dimension for membership inference attacks. To the best of our knowledge, we are the first work to study membership inference in this challenging and realistic RLVR scenario. To address the unique privacy challenges posed by RLVR, we propose D ivergence- i n- B ehavior A ttack (DIBA), a novel membership inference framework that shifts focus from memorization to behavioral change. Instead of relying on memorization features, such as likelihood or output fluency, DIBA quantifies the divergence in behavioral changes along two complementary axes: • The advantage side, which captures improvements in correctness (i.e., reward gain), contributing strongly to overall discriminative power (AUC); • The logit side, which measures changes in token-level probabilities, providing calibrated, high-precision signals crucial for low-FPR detection. Compared to existing baselines, DIBA achieves an order-of-magnitude improvement in [email protected]%FPR while maintaining strong AUC (up to 0.84), demonstrating its superior effectiveness in high-precision, privacy-sensitive settings. The attack generalizes well across datasets, transferring robustly from harder (e.g., paraphrased) to easier ones, indicating resilience to surface-form variations. Additionally, we find that transferring across different algorithms has a larger influence on attack performance than model sizes. When extended to vision-language models (VLMs), DIBA remains effective with an AUC of approximately 0.7, confirming the presence of behavioral signals in multimodal settings, while [email protected]%FPR is notably lower. Ablation studies reveal that sufficient response sampling and moderate temperature are crucial for reliable feature estimation. Moreover, using responses sampled from the fine-tuned model yields stronger logit-side signals than those from the base model, further emphasizing the importance of targeting learned behaviors. We find that membership leakage is strongest on prompts that exhibit measurable behavioral change during training, and weakest on consistently correct or incorrect prompts. This underscores that behavioral divergence, not static correctness, is the primary driver of inference success. Our analysis shows that prompts with pre-existing high correctness can lead to elevated false positives, reflecting how RLVR amplifies inherent model capabilities. Additionally, we evaluate three practical defenses. DIBA exhibits decent robustness against these potential defenses, and only the paraphrase attack can effectively reduce the performance, which is at a steep computation cost. In summary, we make the following contributions: • We are the first to reveal the privacy risk in RLVR, tackling a challenging threat model where only prompts, not responses, are targeted. • We propose DIBA, a new paradigm for MIA that captures the behavioral divergence instead of the memorization after training, to audit the privacy leakage within RLVR. • We conduct an extensive evaluation of DIBA across in-distribution, cross-dataset, cross-algorithm, and extension to VLMs settings, and demonstrate both the effectiveness of behavioral divergence as a leakage signal. Related Work Reinforcement Learning for LLM. Extensive studies have shown the superior ability of LLM across various domains and tasks, such as academic writing [ 20 , 1 ] , code generation [ 16 , 36 ] , and math reasoning [ 32 ] , which mainly stem from the well-known three-stage training process [ 25 ] , i.e.,

Read the complete text