Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Vincent Wang, Chris Lee · 2026
Read Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs by Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Vincent Wang, Chris Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11061.txt.
Introduction
Lecheng Yan and co-authors investigate a paradox in reinforcement learning with verifiable rewards. The method, RLVR, boosts language model reasoning by rewarding correct answers. Yet models like Qwen 2.5 gain even when rewards are spurious or wrong. The authors dig into the mechanism behind that puzzle. They name what they find the Perplexity Paradox.
Under spurious rewards, answer-token perplexity drops while the prompt side's coherence degrades. That split suggests the model bypasses reasoning in favor of memorization shortcuts. If gains can come from shortcuts, benchmark jumps may overstate true reasoning growth. The work warns trainers to audit what RLVR actually teaches, not just what scores say.
Catalog record
| CREATOR | Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Vincent Wang, Chris Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Vincent Wang, Chris Lee Abstract Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a ‘‘Perplexity Paradox’’: spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18–20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering—artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models 1 1 1 Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts . . 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has established itself as a powerful paradigm for advancing the reasoning capabilities of Large Language Models (LLMs), by treating the final-answer correctness as the reward signal, showing particular promise in deterministic domains such as mathematics and coding ( 8 ; 11 ) . The standard interpretation of RLVR’s success relies on the assumption that performance improvements derive from models optimizing robust reasoning policies through ground-truth supervision. However, recent empirical studies have complicated this narrative: the Qwen2.5-Math model family achieved significant performance gains on standard benchmarks (e.g., MATH-500, AIME) even when trained with spurious rewards—signals that were random, purely format-based, or even incorrect ( 24 ) . Building on these observations, another work ( 27 ) argued that such gains are less likely the result of genuine reasoning generalization and more attributable to data contamination, identifying that Qwen2.5 had likely memorized the test sets during pre-training. Synthesizing these observations raises a critical question: if the base model already contained the contaminated data, why was its initial accuracy limited, and how could training with incorrect rewards paradoxically unlock this performance? We hypothesize that spurious RLVR activates the model’s latent capacity for memorization, overriding general reasoning pathways in favor of retrieving stored answers . To validate this hypothesis, it is essential to situate these findings within the reasoning-memorization in LLMs. Recent interpretability research has sought to distinguish these behaviors, with studies identifying specific neuron activation patterns that signal memorization ( 25 ) or isolating linear directions in the residual stream that mediate the switch between reasoning and memory modes ( 9 ) . Notably, the phenomenon of “over-memorization” has been linked to counterintuitive loss landscapes, where models maintain high test accuracy despite exhibiting high test perplexity ( 23 ) : a finding that strongly corroborates the behavioral anomalies we observe in this work. However, while these studies characterize memorization in general pre-training or fine-tuning contexts, they do not explain the specific dynamics of spurious RLVR. A critical gap remains: distinct from standard training, how exactly does spurious RLVR trigger this “mode switch”? The internal mechanisms that allow Qwen2.5 to abandon general reasoning for stubborn memorization under such conditions remain largely opaque. To address this questions, we investigate the learning dynamics of LLMs subjected to spurious RLVR training. We define spurious RLVR as reinforcement learning with verifiable rewards where the reward signal is decoupled from genuine reasoning quality, including random rewards, format-only rewards, or intentionally incorrect rewards. We uncover a phenomenon that mirrors the “over-memorization” signatures identified in prior work ( 23 ) : as spurious rewards typically degrade language modeling globally, the contaminated model exhibits a unique divergence where perplexity on answer tokens decreases significantly, even as prompt perplexity rises . This divergence provides compelling evidence that the Qwen model is not acquiring genuine reasoning capabilities ( 30 ) ; instead, the optimization trajectory results in degraded prompt-side coherence as a side effect of shortcut formation, a direct mapping from specific prompts to memorized answers. To uncover the physical location of this shortcut, we employ a comprehensive mechanistic analysis combining Path Patching ( 19 ) , Logit Lens ( 20 ) , and Jensen-Shannon Divergence (JSD) metrics. Our investigation reveals a crucial decoupling between function and structure within the Qwen model’s layers. Specifically, we identify the middle layers (L18–L20) as the Functional Anchor , where the critical decision to retrieve a memorized answer is causally determined and a high-probability trigger token is injected. Conversely, the subsequent layers (L21+) act as Structural Adapters , exhibiting the most substantial weight changes (high JSD) not to store knowledge, but to perform a representational transformation that reorganizes the internal state to accommodate the abrupt signal from the functional anchor layers. We further formalize this dynamic process by modeling the layer-wise evolution of hidden states as a continuous trajectory ( 14 ; 21 ; 18 ) using Neural Differential Equations (NDEs) ( 3 ) . By quantifying the “separation force” acting between generalization and leakage data samples in the latent space, our NDE analysis mathematically confirms that the trajectory bifurcation, the physical divergence point where the model abandons standard processing pathways to engage the specialized memorization circuit, is causally determined at the functional anchor layers. Crucially, we move beyond passive observation to active control. We identify specific multilayer perceptron (MLP) neurons within these layers that mediate the shortcut retrieval. By performing a mechanistic intervention, specifically scaling the keys of these MLP neurons, we demonstrate the ability to causally steer the Qwen model’s behavior. We show that adjusting these scaling factors can either amplify the model’s reliance on contaminated data or suppress the shortcut to reveal the model’s underlying baseline performance. This provides definitive causal evidence for the Anchor-Adapter circuit and suggests a new methodology for mitigating the effects of data contamination in RLVR. To summarize, our work has three main contributions: 1. Discovery of the Anchor-Adapter Circuit: We causally localize mechanism of Qwen 2.5 for spurious gains into two distinct components: Functional Anchor (L18-L20) : middle layers acting as the decisive trigger for retrieving memorized answers; Structural Adapters (L21+) : later layers undergoing weight transformations to accommodate shortcut signals rather than storing new knowledge. 2. Empirical Validation of Contamination: Through ablations, we demonstrate that these specific circuits are dataset-dependent, and manipulating them eliminates performance gains on contaminated benchmarks while leaving general reasoning capabilities intact. 3. Causal Steering via MLP Intervention : We demonstrate a precise mechanistic intervention by scaling specific MLP keys within identified layers. This allows us to bidirectionally steer the model’s performance, artificially inflating or suppressing contamination effect, confirming causal role of these neurons in shortcut activation. 2 Related Work Spurious Rewards and Shortcut Learning in RLVR. Recent work has begun to analyze the black box of RLVR performance gains, questioning whether improved benchmark scores reflect enhanced reasoning or sophisticated pattern matching. 24 first demonstrated that models like Qwen2.5 Math can achieve substantial accuracy improvements even when supervised by random or incorrect signals, framing these as spurious rewards that challenge the necessity of ground-truth feedback. This skepticism